Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submitting a Command job created using the __call__ method on another Command is missing outputs #39187

Open
gedemagt opened this issue Jan 14, 2025 · 1 comment
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@gedemagt
Copy link

  • azure-ai-ml:
  • 1.23.0:
  • Linux:
  • 3.10:

Describe the bug
When creating a Command object to be submitted as a job by firstly creating a Command with the azure.ai.ml.command and then subsequently creating a new Command object by invoking __call__ on the first Command object, the outputs are lost and it fails executing on the compute.

To Reproduce
Steps to reproduce the behavior:

  1. Firstly, create one Command
  cmd = command(
    code=".",
    command="python main.py --model-output ${{outputs.model}} --val ${{inputs.val}}",
    inputs={
        "val": "some-value",
    },
    outputs={
        "model": Output(type=AssetTypes.MLFLOW_MODEL)
    },
    environment="gm-ml-train-cuda-env@latest"
  )
  1. Secondly create a derived command from this
cmd_derived = cmd(
    val="some-value"
)
  1. Upload both to our ML workspace
ml_client.create_or_update(cmd)
ml_client.create_or_update(cmd_derived)

`

Expected behavior
I expect both jobs to be queued and start running. However only the first one will run, the second will fail with a missing output.

Screenshots

Image

@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jan 14, 2025
@xiangyan99 xiangyan99 added Machine Learning Service Attention Workflow: This issue is responsible by Azure service team. and removed needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. labels Jan 14, 2025
@github-actions github-actions bot added the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Jan 14, 2025
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

2 participants