Azure ML Studio Local Environment — Numpy package import failure using the Azure ML Python SDK-CodePudding

I am trying to create a local environment for the ML Studio using the Python SDK, following this official cheatsheet. The result should be a conda-like environment that can be used for local testing. However, I am running into an error when importing the Numpy package with the add_conda_package() method of the CondaDependencies() class. Where I've tried not specifying, as well as specifying package versions, like: add_conda_package('numpy') or add_conda_package('numpy=1.21.2'), but it does not seem to make a difference.

Numpy's error message is extensive, and I've tried many of the suggestions, without success nonetheless. I'm grateful for any tips on what might resolve my issues!

Full code

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies


def get_env() -> Environment:
    conda = CondaDependencies()

    # add channels
    conda.add_channel('defaults')
    conda.add_channel('conda-forge')
    conda.add_channel('pytorch')

    # Python
    conda.add_conda_package('python=3.8')

    # Other conda packages
    conda.add_conda_package('cudatoolkit=11.3')
    conda.add_conda_package('pip')
    conda.add_conda_package('python-dateutil')
    conda.add_conda_package('python-dotenv')
    conda.add_conda_package('pytorch=1.10')
    conda.add_conda_package('torchaudio')
    conda.add_conda_package('torchvision')
    conda.add_conda_package('wheel')
    conda.add_conda_package('numpy=1.21.2') # <--- Error with this import 

    # create environment
    env = Environment('test_env')
    env.python.conda_dependencies = conda

    return env

Detailed error message:

User program failed with ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

The Python version is: Python3.8 from "<LOCAL_DIR>.azureml\envs\azureml_>\python.exe"
The NumPy version is: "1.19.1"

and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help.

Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.

System specifications:

Local OS: Windows 10
ML studio OS: Linux Ubuntu 18
Python version: 3.8

CodePudding user response：

I was finally able to resolve the issue by using the pip method instead of the conda method: add_pip_package('numpy') instead of add_conda_package('numpy') I can imagine this being the reason for other packages as well.

Full solution

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies


def get_env() -> Environment:
    conda = CondaDependencies()

    # add channels
    conda.add_channel('defaults')
    conda.add_channel('conda-forge')
    conda.add_channel('pytorch')

    # Python
    conda.add_conda_package('python=3.8')

    # Other conda packages
    conda.add_conda_package('cudatoolkit=11.3')
    conda.add_conda_package('pip')
    conda.add_conda_package('python-dateutil')
    conda.add_conda_package('python-dotenv')
    conda.add_conda_package('pytorch=1.10')
    conda.add_conda_package('torchaudio')
    conda.add_conda_package('torchvision')
    conda.add_conda_package('wheel')
    #conda.add_conda_package('numpy=1.21.2') # <--- Error with this import 

    # Add pip packages
    conda.add_pip_package('numpy') # <--- Fixes import error

    # create environment
    env = Environment('test_env')
    env.python.conda_dependencies = conda

    return env