NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

Elcook · 2023-04-13T14:49:40Z

Hi,

I installed opendrift folowing instructions you provide. When using a conda environment I am not able to reach CMEMS source, no mather where I add my .dodsrc and .netrc file.

I have no problems using ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i' outside of conda environment but after doing:

conda activate opendrift
and
(opendrift) [root@opendrift-poc ~]# ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

I get the following error message:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: <!DOCTYPE^ html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <head> <title>Login</title> <link href="http://marine.copernicus.eu/favicon.ico" rel="shortcut icon" type="image/x-icon" /> > <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/patch_content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/cssmanager.php?version=4.0.2-1288885946&files=/cas/newCss/css/myocean/print/print_003.css" media="print" />  <script type="text/javascript" src="js/common_rosters.js"></script> <style type="text/css">input.submit{cursor:pointer}</style> </head> <body id="cas" class="claro" onload="init();" > <div id="white_background_inner_page" style="height:105px;"></div> <div id="col3_content" class="clearfix"> <img src="/css/myocean/print/images/logo_myocean_print.png" width="1" height="1" align="left" id="logoPrint" alt="" /> <div id="breadcrumb"> <a href="http://marine.copernicus.eu/">Home</a> > Login </div> <h1>Welcome to the CMEMS Data Access Portal</h1> <br/> <p>Fill your login/password and click on <strong>Login</strong> to download data. If you are not registred yet click on <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent">Register</a></p> <p>If you have trouble logging in, make sure your browser is set to accept cookies.</p> <br/> <form id="authentification" name="com-login" class="fm-v clearfix" action="/cas/login;jsessionid=BA9435DA67876D405E6490F3DFC023A9?service=http://localhost:8080/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i.dds" method="post"> <fieldset class="input"> <label for="username">Username</label> <input id="username" name="username" class="required" tabindex="1" accesskey="i" type="text" value="" size="25" autocomplete="false"/><br /> <label for="password">Password</label> <input id="password" name="password" class="required" tabindex="2" accesskey="m" type="password" value="" size="25" autocomplete="off"/><br /> <input type="hidden" name="lt" value="_c4793F86B-451F-2DD7-ECFA-D395B0B99CF4_k3D0FA0E2-B51A-ACF8-EF1A-B307A76D684E" /> <input type="hidden" name="_eventId" value="submit" /> <div class="authentification_retrait_right"> <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent"><img src="/cas/newCss/css/myocean/screen/images/bt_register.gif" alt="Register" /></a> <input class="submit" name="submit" accesskey="l" value=" " type="submit" alt="Login" /> </div> </fieldset> <p> <br/><br/> <i>- You are registered but have forgotten your login/password?</i><br/> <i>- Any question?</i><br/> <b>⇒ CMEMS Service Desk : </b><A HREF="mailto:servicedesk.cmems@mercator-ocean.eu"><b>servicedesk.cmems@mercator-ocean.eu</b></A> </p> </form> <h3>Thank you for using CMEMS products</h3> <br/> <div id="sidebar"> <i>For security reasons, please Exit your web browser when you quit services requiring authentication!</i> </div> </div> </body></html> ncdump: https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i: NetCDF: Access failure

Of course, using python's modules inside the env yields similar results

from opendrift.readers import reader_netCDF_CF_generic
reader_netCDF_CF_generic.Reader('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i')

results in

Traceback (most recent call last):
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '297eccf1-3b01-443c-a779-283fa9800280']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 146, in __init__
    self.Dataset = xr.open_dataset(filename, decode_times=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/root/miniconda3/envs/opendrift/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 148, in __init__
    raise ValueError(e)
ValueError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

The only instalation that worked is the one via a docker file, but it doesn't suit my needs.

my conda environments is as follows:

     active environment : opendrift
    active env location : /root/miniconda3/envs/opendrift
            shell level : 2
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.28=0
                          __linux=4.18.0=0
                          __unix=0=0
       base environment : /root/miniconda3  (writable)
      conda av data dir : /root/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /root/miniconda3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Linux/4.18.0-425.19.2.el8_7.x86_64 rocky/8.7 glibc/2.28
                UID:GID : 0:0
             netrc file : /root/.netrc
           offline mode : False

# conda environments:
#
base                     /root/miniconda3
opendrift             *  /root/miniconda3/envs/opendrift

sys.version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GC...
sys.prefix: /root/miniconda3
sys.executable: /root/miniconda3/bin/python
conda location: /root/miniconda3/lib/python3.10/site-packages/conda
conda-build: None
conda-content-trust: /root/miniconda3/bin/conda-content-trust
conda-env: /root/miniconda3/bin/conda-env
user site dirs:

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: opendrift
CONDA_EXE: /root/miniconda3/bin/conda
CONDA_PREFIX: /root/miniconda3/envs/opendrift
CONDA_PREFIX_1: /root/miniconda3
CONDA_PROMPT_MODIFIER: (opendrift)
CONDA_PYTHON_EXE: /root/miniconda3/bin/python
CONDA_ROOT: /root/miniconda3
CONDA_SHLVL: 2
CURL_CA_BUNDLE: <not set>
GDAL_DRIVER_PATH: /root/miniconda3/envs/opendrift/lib/gdalplugins
LD_PRELOAD: <not set>
PATH: /root/miniconda3/envs/opendrift/bin:/root/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
http_proxy: <set>
https_proxy: <set>

Any help would be greatly appreciated, I have run out of ideas already.

Thank you very much

The text was updated successfully, but these errors were encountered:

gauteh · 2023-04-13T15:33:09Z

It looks like a recent bug in netcdf, maybe you can downgrade that package.

knutfrode · 2023-04-13T15:35:33Z

I also get the same problem when updating my environment, and so that netCDF version is 1.6.3
I tried then to downgrade to netCDF 1.6.2 (working on another machine), but same problem.
Also downgrading Xarray to 2022.12.0 (working on another machine) did not work.
Then not sure how to fix this?

Elcook · 2023-04-14T10:02:23Z

Would it be possible that the problem comes from the netCDF library, and not from the netCDF4 python module?

I'm assuming that because ncdump works outside the conda environment ( netcdf library version 4.7.0 ) but fails inside it (netcdf library version 4.9.2).

Anyway, I will try to test this and report here

vegardb · 2023-04-14T10:28:17Z

I can make this work for me with conda and netcdf4=1.6.1

Elcook · 2023-04-14T10:57:52Z

I can make this work for me with conda and netcdf4=1.6.1

You are correct, it worked for me too, even with netcdf library version 4.9.2.

knutfrode · 2023-04-14T11:14:54Z

Ok, I have now temporarily locked netcdf4 to <=1.6.1 in opendrift environment.yml, until bug is hopefully solved and backported by someone.
4626d20

AndresSepulveda · 2023-04-18T19:15:17Z

How do you downgrade to netcd4 1.6.1

knutfrode · 2023-04-18T19:38:15Z

If you update to latest, environment.yml contains netcdf4<=1.6.1
You can then do
conda env create -f environment.yml --force
followed by
pip install -e .
to reinstall dependencies

miguelcorreia19 · 2023-05-03T14:43:55Z

Is it possible to define the directory where the .dodsrc is present?
I want to change the directory where these files are present but the opendrift always will try on HOME dir. We can change this behavior?

knutfrode · 2023-05-03T14:47:19Z

.dodsrc and .netrc authentication are general, and not a part of OpenDrift.
You can see specifications here: https://docs.unidata.ucar.edu/nug/current/dap2.html

gauteh changed the title ~~Trouble accessing CMEMS sources when using conda environment~~ NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources Dec 12, 2023

gauteh added bug readers high-priority labels Dec 12, 2023

gauteh pinned this issue Dec 12, 2023

gauteh mentioned this issue Dec 12, 2023

netcdf > 1.6.1 does not read .netrc / .dodsrc Unidata/netcdf4-python#1299

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

Elcook commented Apr 13, 2023

gauteh commented Apr 13, 2023

knutfrode commented Apr 13, 2023

Elcook commented Apr 14, 2023

vegardb commented Apr 14, 2023

Elcook commented Apr 14, 2023

knutfrode commented Apr 14, 2023

AndresSepulveda commented Apr 18, 2023

knutfrode commented Apr 18, 2023 •

edited

Loading

miguelcorreia19 commented May 3, 2023

knutfrode commented May 3, 2023

NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

Comments

Elcook commented Apr 13, 2023

gauteh commented Apr 13, 2023

knutfrode commented Apr 13, 2023

Elcook commented Apr 14, 2023

vegardb commented Apr 14, 2023

Elcook commented Apr 14, 2023

knutfrode commented Apr 14, 2023

AndresSepulveda commented Apr 18, 2023

knutfrode commented Apr 18, 2023 • edited Loading

miguelcorreia19 commented May 3, 2023

knutfrode commented May 3, 2023

knutfrode commented Apr 18, 2023 •

edited

Loading