Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Azure AI Chat Completion Client #4723

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

rohanthacker
Copy link
Contributor

@rohanthacker rohanthacker commented Dec 16, 2024

Related issue number

#4683 Adds initial support for Azure AI Chat Completion Client

Checks

@ekzhu
Copy link
Collaborator

ekzhu commented Dec 16, 2024

@yanivvak can you review this?

@yanivvak
Copy link

yanivvak commented Dec 17, 2024

@ekzhu @rohanthacker
Great work, I tried to deploy with 3 different options offered by Azure AI inference SDK

  1. Azure open AI - works good, I tried it with magnetic one, the websurfer got stucked, can you take a look?
  2. Serverless - it works, but I didn't got the full answer it was phi 3.5
  3. Managed compute - it didn't run for me, I assume it's an issue with the endpoint and it is not related to your code

@yanivvak
Copy link

@ekzhu ekzhu linked an issue Dec 17, 2024 that may be closed by this pull request
@ekzhu
Copy link
Collaborator

ekzhu commented Dec 18, 2024

@lspinheiro could you help reviewing this PR?


class AzureAIChatCompletionClient(ChatCompletionClient):
def __init__(self, **kwargs: Unpack[AzureAIChatCompletionClientConfig]):
if "endpoint" not in kwargs:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this part could benefit from some better separation of concerns between config validation and instantiation. e.g.

class AzureAIChatCompletionClient(ChatCompletionClient):
    def __init__(self, **kwargs: Unpack[AzureAIClientConfiguration]):
        config = self._validate_config(kwargs)
        self._client = self._create_client(config)
        self._create_args = self._prepare_create_args(config)
        # ...

    @staticmethod
    def _validate_config(config: Mapping[str, Any]) -> AzureAIClientConfiguration:
        # Validation logic here
        return config

@lspinheiro
Copy link
Collaborator

@lspinheiro could you help reviewing this PR?

Looks quite good and is consistent with the openai client. I have a minor comment about the config validation. @jackgerrits may have more options since a lot of the design decisions here are driven by his original implementation of the openai client. If anything doesn't make since in this context he would be the best person to evaluate.

@rohanthacker rohanthacker force-pushed the feature/azure-ai-inference-client branch from d53421b to daf43de Compare December 30, 2024 14:19
content = choice.message.content or ""

response = CreateResult(
finish_reason=finish_reason, # type: ignore
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use the latest update to canonicalize the finish_reason. See OpenAIChatCompletionClient

Copy link
Collaborator

@ekzhu ekzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rohanthacker Is this PR ready to review? It looks good from code perspective. @srjoglekar246 could you help to use it in an assistant agent and running some teams on it to test it out?


def __init__(self, **kwargs: Unpack[AzureAIChatCompletionClientConfig]):
config = self._validate_config(kwargs)
self._model_capabilities = config["model_capabilities"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

client = AzureAIChatCompletionClient(
endpoint="endpoint",
credential=AzureKeyCredential("api_key"),
model_capabilities={
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use model_info as model_capabilities is deprecated.

@ekzhu ekzhu marked this pull request as ready for review January 13, 2025 23:33
@ekzhu
Copy link
Collaborator

ekzhu commented Jan 13, 2025

@rohanthacker I made this PR ready for review.

@ekzhu ekzhu added this to the 0.4.1 milestone Jan 13, 2025
@ekzhu ekzhu modified the milestones: 0.4.1, 0.4.2, 0.4.x Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Adding Azure AI inference
4 participants