Source: Ignite 2024: Announcing the Azure AI Foundry SDK | Microsoft Community Hub

Exploring the new Azure AI Foundry

From the AI Foundry Portal to the SDK

At the last Microsoft Ignite, a wave of announcements across Microsoft’s vast ecosystem was showcased, with AI and Agentic workloads taking center stage.

Among these exciting developments, a special mention goes to the newly announced Azure AI Foundry.

Source: Ignite 2024: Announcing the Azure AI Foundry SDK | Microsoft Community Hub

What is Azure AI Foundry?

Azure AI Foundry is Microsoft’s all-in-one platform designed to simplify the development, deployment, and management of AI applications. It consists of two main components:

• A GUI portal, formerly known as Azure AI Studio
• An Azure AI Foundry SDK which enables developers to work with robust pre-built components and leverage pre-built app templates (we will see it more in detail later on in the article).

By integrating various AI services and models into a single environment, it provides developers with the tools they need to efficiently build and implement AI solutions.

• Extensive Model Catalog: Features a rich library of foundational models from providers like OpenAI, Mistral, Meta and others, all optimized for seamless use within the platform. Plus, you can leverage the fine-tuning option to create tailored, purpose-specific LLMs for your unique use cases.
• Interactive Playgrounds: Includes various playgrounds for hands-on experimentation with AI models and services, such as Azure AI Speech and Azure AI Language. These playgrounds allow users to explore features like real-time speech-to-text and translation, facilitating the integration of these capabilities into AI applications.
• Development SDK: Offers a software development kit that enhances the coding experience, allowing developers to work within familiar environments such as GitHub, Visual Studio, and Copilot Studio.

Note: Recent announcements showcased how Copilot Studio and AI Foundry are getting more integrated, allowing custom AI components developen in AI Foundry (like AI Search Index or custom models) to be integrated into low-code Copilot Studio workflow. To learn more, you can refer to this session.

• Evaluation Tools: Offers built-in evaluation capabilities to assess the performance and safety of AI models and applications. Users can create evaluation runs against models, test datasets, or flows, utilizing both built-in and custom evaluation metrics. This feature provides comprehensive insights into the application’s capabilities and limitations, ensuring high-quality AI solutions.
• Monitoring and Tracing: Provides robust monitoring and tracing functionalities to observe and analyze AI applications in real-time. By integrating with Azure Monitor Application Insights, users can collect telemetry data, set up alerts, and visualize performance metrics. This ensures that applications run smoothly and any issues are promptly identified and addressed.
• Safety + Security: Incorporates Azure AI Content Safety to detect and filter harmful content in both user-generated and AI-generated inputs and outputs. This feature helps maintain ethical standards and compliance by identifying and mitigating risks associated with offensive or inappropriate material.

By offering these integrated tools and services, Azure AI Foundry aims at providing a robust toolchain to develop, test, deploy and scale into production powerful AI apps.

Let’s have an overview of the main features and components.

Tour of the main components and new features

To start working on Azure AI Foundry, you will need to initialize a project in the context of a Hub. Let’s break down this two core components:

• AI Hub: This central resource offers features such as data upload, artifact storage, connections to Azure services like Azure OpenAI and Azure AI Services, base model endpoints, compute resources, and robust security and governance measures.
• AI Projects: Within the AI Hub, projects are created as child resources, inheriting connections and compute resources. They facilitate the organization of components like datasets, models, and indexes within isolated data containers. Projects also support open-source model deployments from catalogs and fine-tuned model endpoints, enabling seamless deployment and management of models.

You can create your new Hub and Project directly from the AI Foundry Management Center.

Once created your project, let’s navigate through it and start exploring all the components.

Model catalog and benchmarking

The Azure AI Foundry Model Catalog serves as a centralized hub for discovering and utilizing a diverse array of AI models, facilitating the development of AI applications.

Note that the catalog is not limited to Large Language Models or GenAI models in general; in fact, you will be able to find more standard AI architectures, like YoloX for multi-object tracking or pre-built AI models like the Azure-AI-Speech. As such, AI Foundy model catalog can be your one-stop destination for any AI-related task, including — but not limited to — GenAI tasks.

Another interesting feature that you can leverage from the model catalog is the model’s description and benchmark results. For example, in the following example, we are seeing the Mistral-large-2407 model card, including the LLM quality-related metrics, such as the Quality Index; but we can also have access to other metrics like cost, throughput and latency.

Furthermore, you canc compare your model to other models on specific benchmarks, so that you can pick that one that best suit your business scenario.

This dashboard allows you to quickly examine your tradeoffs when it comes prioritizing different variables, like cost, time to respond, quality and so on.

Playgrounds

The Playground is that place when you can start interacting and testing your models of choice. In addition to the well-known chat Playground — where you can test your LLMs — you have now access to different playgrounds for different AI purposes, including Speech, Images and industry-specific scenarios (like Healthcare).

Note: Another Playground experience — AI Agents — has been announced during Ignite in Private Preview. This will be an additional feature within the Playground and available with different models. You can apply for the private preview here.

When it comes to the Chat Playground, there are a couple of new features that are worth mentioning.

• Generate Prompt. When interacting with the model, you have the option of customizing the system message of your AI Assistant. Now, you can leverage an AI-powered feature to generate it on your behalf.

This will guarantee a well-written system message which incorporates state-of-the-art prompt engineering techniques. For example, let’s say I want to configure an AI Assistant for my restaurant and let’s leverage this new feature to quickly generate the system message on my behalf:

Full system message:

Provide assistance and support for managing and promoting your Italian restaurant. This includes offering customer service, operational advice, marketing strategies, menu suggestions, and answering queries related to Italian cuisine.

# Features

- Customer Interaction: Assist with greeting customers, handling reservations, and addressing frequently asked questions.
- Menu Guidance: Provide suggestions for authentic Italian dishes, ingredient sourcing, and dietary accommodations.
- Marketing Tips: Offer strategies for local advertising, social media engagement, and promotions.
- Operational Advice: Recommend best practices for restaurant management, staffing, and inventory control.

# Output Format

Responses should be concise and actionable, formatted as short paragraphs. Use bullet points for lists where needed.

# Examples

**Example 1:**

*Input:* What are some popular Italian desserts we can add to our menu?

*Output:* 
Consider adding these classics:
- Tiramisu: A layered dessert with coffee-soaked ladyfingers and mascarpone cream.
- Panna Cotta: A creamy pudding served with fresh berries or caramel sauce.
- Cannoli: Crisp pastry shells filled with sweetened ricotta cheese.

**Example 2:**

*Input:* How can we attract more customers during the midweek?

*Output:*
- Offer a midweek lunch special with discounts on select dishes.
- Host a "Taste of Italy" event, featuring a special tasting menu.
- Collaborate with local businesses for cross-promotional deals.

# Notes

- Focus on maintaining authentic Italian cuisine while considering local preferences and dietary trends.
- Tailor marketing strategies according to the restaurant's location and target demographic.

As you can see, even if my request is poorly written, this AI-powered features was able to generate an extremely well-written system message, which I can always adjust depending on my need.

• Manual Evaluation. The second feature I want to highlight is the possibility of starting evaluating your model directly wihtin the Playground, just with manual inputs.

Let’s see an example.

Starting from the manual evaluation UI, you can add your input and expected output (or upload a prepared dataset for that), and evaluate the model response. For example, let’s evaluate the model’s response to user’s greeting:

Since the model’s response was in line with my expectations, I provided a thumb up. You can add many inputs and feedbacks and eventually export or save the results. In this second case, you will be able to see the results under the Evaluation tab:

Under the evaluation tab, there is another interesting feature that is worth exploring.

Evaluation

Evaluation is a critical process that assesses the performance, quality, and safety of generative AI models and applications.

Azure AI Foundry includes a set of evaluators, tools that measure the quality, performance, and safety of AI models and applications. Many of them are AI-powered, meaning that they are nothing but an LLM with a specific system message that instructs it to evaluate a specific aspect of your AI app, like coherence or groundedness.

Source: Evaluation and monitoring metrics for generative AI — Azure AI Foundry | Microsoft Learn

From the evaluator library, you can already access pre-built evaluators for specific scenarios.

From a technical perspective, they are Prompt Flows and, if we inspect their structure (for example, with the Fluency Evaluator) we can see that they follows the typical structure of a Flow, including the Prompty file with the system message (as mentioned in fact, evaluators are LLM with a specific set of instructions).

You can learn more about Prompt Flow here.

You also have the option of creating your own evaluator by clicking on “New Evaluator”. You will be able to define your own system message so that your evaluator model will specifically look for that metric that you need.

Once you are happy with your configuration, you can create your evaluator and you will find it in your evaluation library.

Satefy and Security

When it comes developing AI applications, it is paramaount to have solid responsible AI practices and tools in place to prevent harmful concequences.

• Map: Identify and prioritize AI content risks through analysis and testing.
• Measure: Evaluate risks using clear metrics and iterative testing.
• Mitigate: Address risks with tools like prompt engineering and content filters, then reassess effectiveness.
• Operate: Create and implement a deployment and readiness plan.

With Azure AI foundry, these steps are made easier with Azure AI Content Safety, a service designed to help organizations detect and moderate harmful or inappropriate content- both in input and output — when working with AI-based applications. It leverages Azure OpenAI models in the backend to flag offensive, harmful, or illegal content, as well as performing more complex and sophisticated analysis to detect suspicious patterns.

All the Content Safety features can be consumed via APIs (we will see a demonstration in the following section).

The first and original set of capabilities of Azure AI Content Safety is that of flagging content (in the form of text and/or images) into pre-defined harm categories:

• Hate and Fairness
• Sexual
• Violence
• Self-Harm

Along with the category, the Content Safety also labels the flag with an associated severity risk, from 0 (lowest level of risk) to 7 (highest level of risk).

By enabling this feature of content safety on top of your AI app, you can prevent it from generating harmful content. Furthermore, you can customize the level of risk you want to be triggered before blocking the workflow, by creating your own filter and customize it at both input and output level:

Note: In addition to the aforementioned 4 categories, you will find additional features listed in the filter, including Prompt Shield for jailbreak and indirect attacks (for input filter) and Protected material for text and code and groundedness (for output filter).

Note that, in the portal, you have the option of testing different configurations of your Content Safety filter before deploying it, so that you can find the perfect combination of threshold levels:

Lastly, you have the option of creating Blocklists with specific terms you want your AI app to filter, which might not fall in the prebuilt categories. You can create your Blocklist from here:

And then enable it directly within your content filter.

AI Foundry SDK

At the beginning of the article, we mentioned that AI Foundry comes with a GUI and with an SDK. Let’s now have a look at the SDK. To start using the SDK, you will need to install the following libraries:

pip install azure-ai-projects azure-identity azure-ai-inference

and then retrieve your Project connection string — this will be the only information needed for your SDK to work, since it will be able to retrieve all your assets and artifacts from your AI foundry project.

You can now initialize your project in your code as follows:

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project_connection_string="your-connection-string"

project = AIProjectClient.from_connection_string(
  conn_str=project_connection_string,
  credential=DefaultAzureCredential())

Then, you can start invoking your models. And here it is the “magic”: regardless of your model, you can leverage the same, templatized component that will let you communicate with any LLM with the same format, meaning that I can use, for example, a Phi-3.5-mini-instruct and then change to the GPT-4o simply changing the “model” parameter.

from azure.ai.inference.tracing import AIInferenceInstrumentor
chat = project.inference.get_chat_completions_client()
response = chat.complete(
    model="Phi-3.5-mini-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful writing assistant"},
        {"role": "user", "content": "Tell me a short joke."},
    ]
)


print(response.choices[0].message.content)

Sure, here's a light-hearted joke for you:

Why don't scientists trust atoms?

Because they make up everything, even the jokes!

Remember, humor can be subjective, but I hope this little pun brings a smile to your face!

Finally, I can enable the trace for my application and see the results directly into my AI Foundry project:

# get an chat inferencing client using the project's default model inferencing endpoint
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
from azure.core.settings import settings 

settings.tracing_implementation = "opentelemetry"

from azure.ai.inference.tracing import AIInferenceInstrumentor 

# Instrument AI Inference API 

AIInferenceInstrumentor().instrument()

chat = project.inference.get_chat_completions_client()

# Enable instrumentation of AI packages (inference, agents, openai, langchain)
project.telemetry.enable()

# Log traces to the project's application insights resource
application_insights_connection_string = project.telemetry.get_connection_string()
if application_insights_connection_string:
    configure_azure_monitor(connection_string=application_insights_connection_string)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("example"):
    response = chat.complete(
        model="Phi-3.5-mini-instruct",
        messages=[
            {"role": "system", "content": "You are a helpful writing assistant"},
            {"role": "user", "content": "Tell me a short joke."},
        ]
    )


print(response.choices[0].message.content)

For a comprehensive view of the SDK capability, you can refer to the official MS documentation here.

Conclusions

Azure AI Foundry represents a significant step forward in empowering developers and organizations to harness the power of artificial intelligence. By providing a unified platform with robust tools for model development, evaluation, deployment, and monitoring, it simplifies the complexities of building AI solutions while ensuring safety, efficiency, and scalability.

Plus, the recent announcements about the further integrations with the Microsoft 365 Agents SDK from one side, and Copilot Studio from the other side, makes this platform the perfect destination for AI developers.

I’m looking forward to learn more about AI Foundry updates!

References

• Ignite 2024: Announcing the Azure AI Foundry SDK | Microsoft Community Hub
• https://techcommunity.microsoft.com/blog/azure-ai-services-blog/introducing-azure-ai-agent-service/4298357
• Evaluation and monitoring metrics for generative AI — Azure AI Foundry | Microsoft Learn
• Prompt flow in Azure AI Foundry portal — Azure AI Foundry | Microsoft Learn
• Content Safety in Azure AI Foundry portal overview — Azure AI Foundry | Microsoft Learn
• How to get started with Azure AI Foundry SDK — Azure AI Foundry | Microsoft Learn