Python: Function Calling Error when Using APIM Managed Azure OpenAI

[sophialagerkranspandey]

Discussed in #10176

^{Originally posted by awonglk January 14, 2025}
I've followed the article pointed by following issue to try to get my Semantic Kernel app working with APIM managed Azure OpenAI:
#7143

If there's no function calls involved, the responses back from LLM seems to work as normal.

But as soon as I ask a question that involves a plugin (even the core plugin time_plugin() as an example)

    kernel = Kernel()
    service_id = "function_calling"
    api_key = os.environ.get('AZURE_OPENAI_API_KEY')
    kernel.add_service(
        AzureChatCompletion(
            service_id=service_id,
            default_headers={'Ocp-Apim-SubscriptionKey': api_key}
        )
    )
    
    kernel.add_plugin(TimePlugin(), plugin_name="time")

This is what I get when asking a simple question like "What is the time?"

<class 'semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion.AzureChatCompletion'> service failed to complete the prompt\", BadRequestError('Error code: 400 - {\\'statusCode\\': 400, \\'message\\': \"Unable to parse and estimate tokens from incoming request. Please ensure incoming request does not contain any images and is of one of the following types: \\'Chat Completion\\', \\'Completion\\', \\'Embeddings\\' and works with current prompt estimation mode of \\'Auto\\'.\"}'))

Is there anything obvious that I may have missed?
Using semantic-kernel 1.16.0

[moonbox3]

A bad request means that something formed in the request body is not accepted. Can you please give us some more details about the request being made to the model?

[awonglk]

I am able to reproduce the issue with a very simple function call script below.
Fails when using APIM endpoint similar to: https://<our_apim_URL>/openai/deployments/chat/chat/completions?api-version=2024-08-01-preview
Works fine when using regular Azure OpenAI endpoint.

from semantic_kernel.kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
from semantic_kernel.connectors.ai.open_ai.prompt_execution_settings.open_ai_prompt_execution_settings import OpenAIChatPromptExecutionSettings
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.functions.kernel_arguments import KernelArguments
from semantic_kernel.contents.chat_history import ChatHistory
from semantic_kernel.core_plugins.time_plugin import TimePlugin

async def main():
    try:
        # Initialize kernel
        kernel = Kernel()
        
        # Configure Azure OpenAI service
        service_id = "function_calling"
        deployment_name = "chat"  # Your deployment name
        endpoint = "<endpoint>"  # your endpoint
        api_key = "<key>" # your key

        
        # Add Azure OpenAI service
        kernel.add_service(
            AzureChatCompletion(
                service_id=service_id,
                deployment_name=deployment_name,
                endpoint=endpoint,
                api_key=api_key,
                default_headers={'Ocp-Apim-SubscriptionKey': api_key}
            )
        )
        
        # Add TimePlugin
        kernel.add_plugin(TimePlugin(), plugin_name="time")
        
        # Create chat function with system message
        chat_function = kernel.add_function(
            prompt="""
            You are a helpful assistant. Use the available functions when appropriate.
            
            {{$chat_history}}
            Human: {{$user_input}}
            Assistant: Let me help you with that.
            """,
            plugin_name="ChatBot",
            function_name="Chat",
        )
        
        # Configure execution settings with function calling
        execution_settings = OpenAIChatPromptExecutionSettings(
            service_id=service_id,
            temperature=0.7,
            max_tokens=1000,
            function_choice_behavior=FunctionChoiceBehavior.Auto(
                filters={"included_plugins": ["time"]},
            ),
        )
        
        # Initialize chat history
        chat_history = ChatHistory()
        
        # Test message that should trigger function calling
        user_input = "What's the current time?"
        
        # Create arguments for the chat function
        arguments = KernelArguments(
            settings=execution_settings,
            user_input=user_input,
            chat_history=chat_history,
        )
        
        # Invoke the chat function
        print("Sending request to LLM...")
        response = await kernel.invoke(
            chat_function,
            return_function_results=False,
            arguments=arguments,
        )
        
        print("\nResponse:", str(response))
        
    except Exception as e:
        print(f"Error: {str(e)}")

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Let me know if there is anything else I need to provide.

[moonbox3]

Hi @awonglk, thanks for the added context. Can you please make sure you're following the follow flow when using the APIM auth:

https://jordanbeandev.com/how-to-set-up-a-python-api-to-call-an-azure-openai-instance-that-is-behind-azure-api-management-and-requires-both-a-subscription-key-an-oauth2-access-token-using-semantic-kernel/

[awonglk]

The only thing I'm not following from that link is I'm not using "OnBehalfOfCredential".
Just using plain api_key, as our APIM is setup for api key authentication.

Is using OnBehalfOfCredential auth a must?

[moonbox3]

Alright, I understand. Apologies to try and lead you down the wrong path. I'll need to figure out how to replicate your issue.

For further context, have you followed this step? Pass the API key in API requests - set-backend-service policy.

[moonbox3]

@awonglk, based on this previous comment - a user was experiencing a function calling error using APIM in SK .Net -- could it be that your model you're calling doesn't support function calling?

#9016 (comment)

[moonbox3]

Or this option as well: #8340 (comment)

AzureChatCompletion(
    api_key=`Value of Ocp-Apim-SubscriptionKey`
    endpoint=AZURE_OPENAI_ENDPOINT, # your endpoint, which doesn't include `deployments/gpt-4o/chat/completions?api-version=2024-02-15-preview`
    deployment_name=AZURE_OPENAI_CHAT_DEPLOYMENT_NAME, # gpt-4o
    api_version='2024-02-15-preview'
    default_headers={'Ocp-Apim-SubscriptionKey': <value>}
}

[awonglk]

For your 3 x comments above:

• [Pass the API key in API requests - set-backend-service policy]
  We are using managed identity for the authentication between APIM and backend LLM instead of passing the LLM's API key

• LLM Support for function calling
  We are using gpt-4o (2024-05-13), which supports parallel function calling

• Option for passing Ocp-Apim-SubscriptionKey header
  I am already passing this default_headers in the code example above. I find that whether I have it or not, doesn't make a difference. The behaviour is still the same (when no function call involved, it works. But when function calling is involved, the error occurs)

[moonbox3]

We are using gpt-4o (2024-05-13), which supports parallel function calling

Which Azure OpenAI API version are you using? For example, 2024-09-01-preview

I know APIM is a part of this, but the model is not failing with a 401/403, so it doesn't look like an auth issue.

The behaviour is still the same (when no function call involved, it works. But when function calling is involved, the error occurs

In the "it works" case, are you including tools in your request? Or are you removing tools?