AzureML registered dataset

[RafieeAshkan]

Hi,
I have been running my experiments locally using hydra with pytorch lightning and mlflow successfully. Now, I would like to run it on AzureML. When running locally, to load data I give the path to data in hydra config files as data_dir: ${paths.root_dir}/data/. However, on Azure I registered the dataset and has the folder uri as azureml:my_data:1 with the subscription group and workspace details. I was wondering if there is any chance I can use the azure uri in hydra config as the data location?
Many thanks
Ashkan

[Jasha10]

Hi @RafieeAshkan,

Hydra doesn't support azure specifically... but wouldn't it be possible to add something like data_uri: "azureml:my_data:1" to your config file?

[RafieeAshkan]

I solved the issue by first setting path as env variable in Azure job script using export DATA_DIR=${{inputs.data_dir}} and then use that in my hydra config file with data_dir: ${oc.env:DATA_DIR}. To clarify as well, in the Azure job script I have

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json

display_name: testing_rul_on_azure
experiment_name: train-rul

command: |
  export DATA_DIR=${{inputs.data_dir}}
  echo "data directory is ${DATA_DIR}"
  python train.py -m experiment=${{inputs.experiment_name}}

inputs:
  experiment_name: tft_cmapss
  data_dir:
    type: uri_folder
    path: azureml:my_data:1

environment:
  build:
    path: .
code: ../
compute: azureml:gpu-cluster