Please pin the versions of the external python libraries - e.g. sdk/ml/azure-ai-ml/setup.py

[almmechanics]

This is the second time in the last six months that a breaking change to a library (in this case marshmallow) has caused the azure sdk to become broken.

Please pin versions for consistency !

For example the file https://github.com/Azure/azure-sdk-for-python/blob/main/shared_requirements.txt is always going to track latest - and without the known versions the validity of the sdk is perpetually in question.

Without consistency then the reliability of this codebase will always be in question, and as a result end users have to develop workaround to compensate for a perceived stable library

[aidangallagher4]

agree with this -- it also means that users can't rollback to previous versions to fix any bugs caused by bad dependencies

[xiangyan99]

Thanks for reaching out.

The file at https://github.com/Azure/azure-sdk-for-python/blob/main/shared_requirements.txt is used to prevent libraries from introducing unexpected dependencies. It specifies which libraries are permitted, but not their exact versions.

Each individual service library is responsible for defining the specific versions it relies on.

Could you let us know which library you're using?

[github-actions]

Hi @almmechanics. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[almmechanics]

@xiangyan99 Based on a local extract of the sdk/ml/azure-ai-ml/setup.py, I can identify the current packages required (not tested with the code, just what the installers use)

Package                                Version
-------------------------------------- -----------
aml_export                             1.0.0
asgiref                                3.8.1
attrs                                  25.3.0
azure-common                           1.1.28
azure-core                             1.33.0
azure-core-tracing-opentelemetry       1.0.0b12
azure-identity                         1.21.0
azure-mgmt-core                        1.5.0
azure-monitor-opentelemetry            1.6.7
azure-monitor-opentelemetry-exporter   1.0.0b36
azure-storage-blob                     12.25.1
azure-storage-file-datalake            12.20.0
azure-storage-file-share               12.21.0
certifi                                2025.1.31
cffi                                   1.17.1
charset-normalizer                     3.4.1
colorama                               0.4.6
cryptography                           44.0.2
Deprecated                             1.2.18
fixedint                               0.1.6
idna                                   3.10
importlib_metadata                     8.6.1
isodate                                0.7.2
jsonschema                             4.23.0
jsonschema-specifications              2024.10.1
marshmallow                            3.26.1
msal                                   1.32.0
msal-extensions                        1.3.1
msrest                                 0.7.1
oauthlib                               3.2.2
opentelemetry-api                      1.32.1
opentelemetry-instrumentation          0.53b1
opentelemetry-instrumentation-asgi     0.53b1
opentelemetry-instrumentation-dbapi    0.53b1
opentelemetry-instrumentation-django   0.53b1
opentelemetry-instrumentation-fastapi  0.53b1
opentelemetry-instrumentation-flask    0.53b1
opentelemetry-instrumentation-psycopg2 0.53b1
opentelemetry-instrumentation-requests 0.53b1
opentelemetry-instrumentation-urllib   0.53b1
opentelemetry-instrumentation-urllib3  0.53b1
opentelemetry-instrumentation-wsgi     0.53b1
opentelemetry-resource-detector-azure  0.1.5
opentelemetry-sdk                      1.32.1
opentelemetry-semantic-conventions     0.53b1
opentelemetry-util-http                0.53b1
packaging                              24.2
pip                                    23.0.1
psutil                                 6.1.1
pycparser                              2.22
pydash                                 8.0.5
PyJWT                                  2.10.1
python-dateutil                        2.9.0.post0
PyYAML                                 6.0.2
referencing                            0.36.2
requests                               2.32.3
requests-oauthlib                      2.0.0
rpds-py                                0.24.0
setuptools                             65.5.0
six                                    1.17.0
strictyaml                             1.7.3
tqdm                                   4.67.1
typing_extensions                      4.13.2
urllib3                                2.4.0
wrapt                                  1.17.2
zipp                                   3.21.0

These have been upper bounded into the following pyproject.toml . I have added <= to scope to the libraries for a local install on 17/4/2025 ; I have not added any >= entries to this installation, but left them as per https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ml/azure-ai-ml/setup.py

[project]
name = "aml_export"
version = "1.0.0"
description = "Export of azure ml setup.py"
authors = [{ name = "almmechanics", email = "[email protected]" }]
requires-python = ">=3.7"
classifiers=[
    "Development Status :: 5 - Production/Stable",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: 3.13",
    "License :: OSI Approved :: MIT License",
]
dependencies = [
    "pyyaml>=5.1.0,<=6.0.2",
    "msrest>=0.6.18,<=0.7.1",
    "azure-core>=1.23.0,<=1.33.0",
    "azure-mgmt-core>=1.3.0,<=1.33.0"",
    "marshmallow>=3.5,<4.0.0",
    "jsonschema>=4.0.0,<= 4.23.0",
    "tqdm<=4.67.1",
    "strictyaml<=1.7.3",
    "colorama<=0.4.6",
    "pyjwt<=2.10.1",
    "azure-storage-blob>=12.10.0,<=12.25.1",
    "azure-storage-file-share<=12.21.0",
    "azure-storage-file-datalake>=12.2.0,<=12.20.0",
    "pydash>=6.0.0,<=8.0.5",
    "isodate<=0.7.2",
    "azure-common>=1.1,<=1.1.28",
    "typing-extensions<=4.13.2",
    "azure-monitor-opentelemetry<=1.6.7",
    # TODO: remove six after using new autorest
    "six>=1.11.0,<=1.17.0",
]

[build-system]
requires = ["setuptools<72", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
packages = []

[tool.pylint.main]
ignore-paths = '^build/*'

There is quite a range of difference from the original values in sdk/ml/azure-ai-ml/setup.py

    install_requires=[
        # NOTE: To avoid breaking changes in a major version bump, all dependencies should pin an upper bound if possible.
        "pyyaml>=5.1.0",
        "msrest>=0.6.18",
        "azure-core>=1.23.0",
        "azure-mgmt-core>=1.3.0",
        "marshmallow>=3.5,<4.0.0",
        "jsonschema>=4.0.0",
        "tqdm",
        # Used for PR 825138
        "strictyaml",
        # Used for PR 718512
        "colorama",
        "pyjwt",
        "azure-storage-blob>=12.10.0",
        "azure-storage-file-share",
        "azure-storage-file-datalake>=12.2.0",
        "pydash>=6.0.0",
        "isodate",
        "azure-common>=1.1",
        "typing-extensions",
        "azure-monitor-opentelemetry",
        # TODO: remove six after using new autorest
        "six>=1.11.0",
    ],

[xiangyan99]

Thanks for the information.

From what I see, the ML library includes a check for marshmallow>=3.5,<4.0.0 to prevent major version upgrades that could introduce breaking changes. So, the recently released marshmallow 4.0.0 shouldn't be installed in your environment.

I don't see any problems.

Did I overlook anything?

[github-actions]

Hi @almmechanics. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[xiangyan99]

I think I figured out what happened.

azure-ai-ml 1.26.2 did not have the upper bound check which was broken by the marshmallow change.

We just shipped azure-ai-ml 1.26.3 which added the check.

Please use the latest one.

[github-actions]

Hi @almmechanics. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[almmechanics]

@xiangyan99 within sdk/ml/azure-ai-ml/setup.py is the following line

        # NOTE: To avoid breaking changes in a major version bump, all dependencies should pin an upper bound if possible.

Prior to singular fix of marshmallow was added today - None of the entries had upper bounds as per the above suggestion in the code - this means any of the libraries are open to breaking changes with two specific observations:

i) No upper or lower is always going to follow latest - so no chance to repeat or validate
ii) Some of them are now TWO major versions ahead of a defined minimum - so these have not been checked or tested for quite a while.

[almmechanics]

/unresolve

[prazevj]

Hi @xiangyan99 this is also affecting the azure-ai-evaluation package.

I am still unable to resolve this with the marshmallow changes. Getting Import error while using

azure-ai-evaluation==1.5.0
marshmallow==4.0.0
ImportError: cannot import name 'FieldInstanceResolutionError' from 'marshmallow.exceptions'