Semantic Kernel 101

Part 1: Understanding the framework and building your first Agent

10 min readDec 3, 2024

Semantic kernel is a lightweight framework which make it easier to develop AI-powered applications. It falls into the category of AI orchestrators like Llama-Index, LangChain, TaskWeaver and so on. So you migth ask yourself: how is Semantic Kernel different and what are its differentiators?

This is the type of question that we will try to cover in this series of articles, exploring the main features and capabilities of Semantic Kernel. However, as a disclaimer, my opinion is that there is no “better” or “worse” orchestrator, but rather they might serve different purposes depending on your unique use cases or, more practically, on the existing skills within your organization.

Having said that, let’s start exploring Semantic Kernel and use it to build a first AI Agent.

What is Semantic Kernel?

As an AI orchestrator, Semantic Kernel comes with a set of pre-built components that are the typical assets needed when building AI apps, including Metaprompts, Memory, VectorDBs and so on. Additionally, it has some differentiating features that needs to be introduced.

Note: to get started with SK, you can refer to the official documentation here and the main GitHub repository for hands on samples here.

Kernel

The kernel in Semantic Kernel acts like a central system that organizes and connects everything your application needs to function smoothly. Think of it as a toolbox where you place all the tools your LLM will use. When you initialize a kernel, you can configure and “bundle” it with LLMs, Plug-ins, Agents and so on.

For example, let’s say you want to build an AI agent that is capable of making restaurant bookings. The logical steps will look like following (note that this is just for showcasing the steps — we will show working code in the last section):

Initializing the Kernel:

kernel = Kernel()

2. Adding your LLM — this will be the “brain” that your kernel will leverage to take decisions:

kernel.add_service(AzureChatCompletion(model_id, endpoint, api_key))

3. Adding a Plug-in which is able to book a table in your restaurant’s website:

kernel.add_plugin(
    Booking(),
    plugin_name="BookingPlugin",
)

Note: as we will see in the next section, a plugin always comes with a natural language description — that’s how the Kernel knows whether and when to invoke it depending on user’s query.

And you can add all the additional components you need.

Plug-ins

Plug-ins — also known as Tools, Actions or Skills in the taxonomy of AI Agents — are set of functions we can provide the Kernel with so that it can perform actions. In fact, these functions can be invoked by the Kernel to accomplish the user’s request via Function Calling.

Note: function calling is a feature of the latest LLMs that enables these models to interact with external tools and APIs by generating structured data that specifies which function to call and with what parameters. This allows LLMs to perform tasks beyond text generation, such as retrieving real-time information or executing specific actions. In SK, you can enable Automatic function calling and let the kernel perform the following steps:

At their core, plug-ins are made of the following ingredients:

The name of the plugin
The names of its functions
The descriptions in natural language of the functions — so that the Kernel knows when to invoke what
The parameters of the functions
The schema of the parameters

There are 3 ways you can create a plug-in:

Native Plugins →this is the quickest way to initialize a plug-in leveraging the SK annotation. Let’s see an example with our Booking plugin. In this case, we are initializing the plugin with two functions — one for checking the current availability, the other to actually make the reservation:

class BookingPlugin:
    def __init__(self):
        # Initialize some dummy data for available tables
        self.tables = {
            "indoor": [2],  # Tables with capacities for indoor seating
            "dehor": [2, 4, 6]   # Tables with capacities for outdoor seating
        }
    @kernel_function
    def check_availability(self, num_people, preference):
        """
        Checks the availability of tables based on the number of people
        and seating preference (indoor or dehor).

        Parameters:
            num_people (int): Number of people for the booking.
            preference (str): "indoor" or "dehor".
        
        Returns:
            bool: True if a table is available, False otherwise.
        """
        if preference not in self.tables:
            return False

        available_tables = [capacity for capacity in self.tables[preference] if capacity >= num_people]
        return len(available_tables) > 0
    @kernel_function
    def make_booking(self, num_people, preference):
        """
        Makes a booking for the specified number of people and seating preference.

        Parameters:
            num_people (int): Number of people for the booking.
            preference (str): "indoor" or "dehor".

        Returns:
            str: Confirmation message or an error if no table is available.
        """
        if preference not in self.tables:
            return "Invalid seating preference. Please choose 'indoor' or 'dehor'."

        available_tables = [capacity for capacity in self.tables[preference] if capacity >= num_people]
        if not available_tables:
            return "No tables available for the selected preferences."

        # Book the first available table that fits the requirements
        table_capacity = available_tables[0]
        self.tables[preference].remove(table_capacity)
        return f"Booking confirmed for {num_people} people {preference} (Table capacity: {table_capacity})."

As you can see, before initializing the function we use the annotation @kernel_function, to make sure that they are initialized as such.

However, you migth already have a set of APIs that do this kind of job. If this is the case, you can leverage them as plugins thanks to the OpenAPI specification.

OpenAPI plugins →this way allows you to import your OpenAPI or Swagger of your APIs as it is as a plugin. For example, let’s say you already have a Swagger for the Booking plugin:

openapi: 3.0.0
info:
  title: Booking Plugin API
  version: 1.0.0
  description: API for managing table bookings, including checking availability and making reservations.
paths:
  /check-availability:
    get:
      summary: Check Table Availability
      description: Check if tables are available based on the number of people and seating preference.
      parameters:
        - name: num_people
          in: query
          description: Number of people for the booking.
          required: true
          schema:
            type: integer
            example: 4
        - name: preference
          in: query
          description: Seating preference (indoor or dehor).
          required: true
          schema:
            type: string
            enum: [indoor, dehor]
            example: indoor
      responses:
        '200':
          description: Availability status.
          content:
            application/json:
              schema:
                type: object
                properties:
                  available:
                    type: boolean
                    example: true
                  message:
                    type: string
                    example: "Tables are available for the selected preference."
        '400':
          description: Invalid parameters.
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                    example: "Invalid seating preference. Please choose 'indoor' or 'dehor'."
  /make-booking:
    post:
      summary: Make a Booking
      description: Book a table based on the number of people and seating preference.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                num_people:
                  type: integer
                  description: Number of people for the booking.
                  example: 4
                preference:
                  type: string
                  description: Seating preference (indoor or dehor).
                  enum: [indoor, dehor]
                  example: indoor
      responses:
        '200':
          description: Booking confirmation.
          content:
            application/json:
              schema:
                type: object
                properties:
                  confirmation:
                    type: string
                    example: "Booking confirmed for 4 people indoor (Table capacity: 6)."
        '400':
          description: Booking error.
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                    example: "No tables available for the selected preferences."
servers:
  - url: http://localhost:5000
    description: Local server

If this is the case, you can simply add it to your kernel as follows:

kernel.add_plugin_from_openapi(
   plugin_name="Booking",
   openapi_document_path="https://example.com/v1/swagger.json",
   execution_settings=OpenAPIFunctionExecutionParameters(
         # Determines whether payload parameter names are augmented with namespaces.
         # Namespaces prevent naming conflicts by adding the parent parameter name
         # as a prefix, separated by dots
         enable_payload_namespacing=True,
   ),
)

Logic Apps →this is a third, low-code approach to define your plugin via Azure Logic Apps. This can be very versatile in case you’ve already have Logic Apps as your enterprise tool for automation and triggering funcionalities.

Once you deployed your Logic App, you can invoke it as an URI to be added in your Kernel:

kernel.ImportPluginFromOpenApiAsync(
    pluginName: "openapi_plugin",
    uri: new Uri("https://example.azurewebsites.net/swagger.json"),
    executionParameters: new OpenApiFunctionExecutionParameters()
    {
        // Determines whether payload parameter names are augmented with namespaces.
        // Namespaces prevent naming conflicts by adding the parent parameter name
        // as a prefix, separated by dots
        EnablePayloadNamespacing = true
    }
);

Enterprise Features

In enterprise applications, Semantic Kernel incorporates filters and observability components to enhance control, security, and monitoring. Filters, such as Function Invocation Filters, Prompt Render Filters, and Auto Function Invocation Filters, allow developers to manage function execution by validating permissions, modifying prompts, and handling exceptions, thereby promoting responsible AI practices.

In addition to filters, SK also enables AI observability, which is achieved through the emission of logs, metrics, and traces adhering to the OpenTelemetry standard, enabling seamless integration with various monitoring tools. This observability framework provides insights into system behavior, facilitating effective monitoring and analysis of services built on Semantic Kernel.

Planning

Originally, this was one of the core component of SK. It consisted of a set of instructions to prompt the model to invoke its plugins with a specific plan. However, with the introduction of Function Calling, this latter became the most performing out-of-the-box planning strategy. As such, the recommendation as of today is that of relying on this strategy rather than creating hard-coded planning strategies.

Building a Restaurant Booking Assistant with SK

Now that we’ve learnt the foundations of SK, let’s build your fist AI Agent. We will stick to the restaurant’s example, building from scratch a native plugin to check table’s availability and make reservation. Note that this example is based on the agent templates available in the aforementioned github repository.

Configuring your kernel. To do so, you can leverage the ServiceSettings module from SK. Make sure to create a .env file with the following variables (in my case, I’m using the Azure OpenAI gpt-4o):

GLOBAL_LLM_SERVICE="AzureOpenAI"
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME="gpt-4o"
AZURE_OPENAI_TEXT_DEPLOYMENT_NAME=""
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME="xxx"
AZURE_OPENAI_ENDPOINT="xxx"
AZURE_OPENAI_API_KEY="xxx"
AZURE_OPENAI_API_VERSION="2024-05-01-preview"

Then, you can prepare your configuration settings as follows:

import asyncio
from typing import Annotated

from semantic_kernel.agents import ChatCompletionAgent
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
from semantic_kernel.contents.chat_history import ChatHistory
from semantic_kernel.contents.utils.author_role import AuthorRole
from semantic_kernel.functions.kernel_function_decorator import kernel_function
from semantic_kernel.kernel import Kernel

from services import Service

from service_settings import ServiceSettings

service_settings = ServiceSettings.create()

# Select a service to use for this notebook (available services: OpenAI, AzureOpenAI, HuggingFace)
selectedService = (
    Service.AzureOpenAI
    if service_settings.global_llm_service is None
    else Service(service_settings.global_llm_service.lower())
)

2. Defining the agent name and instructions:

HOST_NAME = "Host"
HOST_INSTRUCTIONS = "Help customers to investigate about tables availability and make reservations. Use emojis when talking."

3. Defining the native plugin.

class BookingPlugin:
    def __init__(self):
        # Initialize some dummy data for available tables
        self.tables = {
            "indoor": [2],  # Tables with capacities for indoor seating
            "dehor": [2, 4, 6]   # Tables with capacities for outdoor seating
        }
    @kernel_function
    def check_availability(self, num_people, preference):
        """
        Checks the availability of tables based on the number of people
        and seating preference (indoor or dehor).

        Parameters:
            num_people (int): Number of people for the booking.
            preference (str): "indoor" or "dehor".
        
        Returns:
            bool: True if a table is available, False otherwise.
        """
        if preference not in self.tables:
            return False

        available_tables = [capacity for capacity in self.tables[preference] if capacity >= num_people]
        return len(available_tables) > 0
    @kernel_function
    def make_booking(self, num_people, preference):
        """
        Makes a booking for the specified number of people and seating preference.

        Parameters:
            num_people (int): Number of people for the booking.
            preference (str): "indoor" or "dehor".

        Returns:
            str: Confirmation message or an error if no table is available.
        """
        if preference not in self.tables:
            return "Invalid seating preference. Please choose 'indoor' or 'dehor'."

        available_tables = [capacity for capacity in self.tables[preference] if capacity >= num_people]
        if not available_tables:
            return "No tables available for the selected preferences."

        # Book the first available table that fits the requirements
        table_capacity = available_tables[0]
        self.tables[preference].remove(table_capacity)
        return f"Booking confirmed for {num_people} people {preference} (Table capacity: {table_capacity})."

Note: for this demonstration, we are not going to connect to a real booking system, however if you want to see an end-to-end example of this you can refer to this repository.

4. Defining a helper function to invoke the agent:

async def invoke_agent(agent: ChatCompletionAgent, input: str, chat: ChatHistory) -> None:
    """Invoke the agent with the user input."""
    chat.add_user_message(input)

    print(f"# {AuthorRole.USER}: '{input}'")

    if streaming:
        contents = []
        content_name = ""
        async for content in agent.invoke_stream(chat):
            content_name = content.name
            contents.append(content)
        message_content = "".join([content.content for content in contents])
        print(f"# {content.role} - {content_name or '*'}: '{message_content}'")
        chat.add_assistant_message(message_content)
    else:
        async for content in agent.invoke(chat):
            print(f"# {content.role} - {content.name or '*'}: '{content.content}'")
        chat.add_message(content)

5. Finally, creating the main function where we initialize the kernel and the agent. Let’s see step by step:

Initializing the kernel and adding the Azure Chat model as described in the .env file.

async def main():
    # Create the instance of the Kernel
    kernel = Kernel()

    service_id = "agent"
    kernel.add_service(AzureChatCompletion(service_id=service_id))

Configure the function choice behavior. This setting refers to the function calling capability discussed above, and can have 3 configurations: 1) Auto if you want to allow the AI model to choose from zero or more functions from the provided set; 2) Required to force the AI model to choose one or more functions from the provided set; 3) NoneInvoke to prompt the AI model not to choose any functions. In our example, we are going to keep it to the Auto mode.

settings = kernel.get_prompt_execution_settings_from_service_id(service_id=service_id)
    # Configure the function choice behavior to auto invoke kernel functions
settings.function_choice_behavior = FunctionChoiceBehavior.Auto()

Adding the BookingPlugin:

kernel.add_plugin(BookingPlugin(), plugin_name="menu")

Creating and configuring the agent:

agent = ChatCompletionAgent(
        service_id="agent", kernel=kernel, name=HOST_NAME, instructions=HOST_INSTRUCTIONS, execution_settings=settings
    )

Defining the chat history and creating a while cycle to allow the user to interact with the Agent:

    # Define the chat history
    chat = ChatHistory()

    # Respond to user input interactively
    while True:
        user_input = input("You: ")
        if user_input.lower() in ("exit", "quit"):
            break
        await invoke_agent(agent, user_input, chat)

Great, now let’s try to run our Python app with python app.py and see the Agent behavior:

As you can see, the Agent was able to retrieve the availability and informing the user that there are no tables for 4 people indoor (as specified in the plugin).

Let’s see it in action:

Conclusion

In this article, we covered the main components of Semantic Kernel and saw a practical implementation of an AI Agent. A key feature that to aknowledge in this scenario is the way SK allows you to build plug-ins in a standardized way, rather than having different components depending on the plugin specifications. This is a great plus since it guarantees consistency and repeatability across your AI enterprise applications.

In the next chapters, we are going to cover additional capabilities and scenarios of Semantic Kernel — stay tuned!

Semantic Kernel 101

Part 1: Understanding the framework and building your first Agent

What is Semantic Kernel?

Building a Restaurant Booking Assistant with SK

Conclusion

References

Written by Valentina Alto

Responses (1)