MLFlow metrics are not available immediately for consumption

[bastbu]

• Package Name: azure-ai-ml
• Package Version: 1.23.0
• Operating System: Ubuntu
• Python Version: 3.11

Describe the bug

The documentation on Azure ML states that when logging MLFlow metrics (which is now the recommended way to log metrics in Azure ML) synchronously, the metrics are immediately available for consumption:

Metrics, as opposed to parameters, are always numeric, and they can be logged either synchronously or asynchronously. When metrics are logged, they are immediately available for consumption upon call return.

Also, for asynchronous metric logging, the doc seems to imply that the metrics should be queryable when the job finishes:

Azure Machine Learning automatically waits when the job is about to finish, to see if there is any pending metric to be persisted. By the time a job is completed in Azure Machine Learning, all metrics are guaranteed to be persisted.

However, our tests have shown several times that the metrics are not immediately available to be queried with MLFlow when the job finishes. Is this expected, and if yes, is there a deterministic way to wait for the metrics to be available for querying?

To Reproduce

Steps to reproduce the behavior:

1. Log a metric in an Azure ML job using the MLFlow SDK
2. Query all metrics for an Azure ML job after it finished using the MLFlow SDK

Expected behavior

The reported metrics should immediately appear when querying.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.