Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources #1083

Open
Elcook opened this issue Apr 13, 2023 · 10 comments
Open

Comments

@Elcook
Copy link

Elcook commented Apr 13, 2023

Hi,

I installed opendrift folowing instructions you provide. When using a conda environment I am not able to reach CMEMS source, no mather where I add my .dodsrc and .netrc file.

I have no problems using ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i' outside of conda environment but after doing:

conda activate opendrift
and
(opendrift) [root@opendrift-poc ~]# ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

I get the following error message:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: <!DOCTYPE^ html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <head> <title>Login</title> <link href="http://marine.copernicus.eu/favicon.ico" rel="shortcut icon" type="image/x-icon" /> <!-- link rel="icon" type="image/png" href="http://myo-wp.techworks.ie/devel/templates/myocean/favicon.png" /-->> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/patch_content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/cssmanager.php?version=4.0.2-1288885946&amp;files=/cas/newCss/css/myocean/print/print_003.css" media="print" /> <!--[if gte IE 6]><style type="text/css" media="screen">@import 'css/ie_cas.css';</style><![endif]--> <script type="text/javascript" src="js/common_rosters.js"></script> <style type="text/css">input.submit{cursor:pointer}</style> </head> <body id="cas" class="claro" onload="init();" > <div id="white_background_inner_page" style="height:105px;"></div> <div id="col3_content" class="clearfix"> <img src="/css/myocean/print/images/logo_myocean_print.png" width="1" height="1" align="left" id="logoPrint" alt="" /> <div id="breadcrumb"> <a href="http://marine.copernicus.eu/">Home</a> &gt; Login </div> <h1>Welcome to the CMEMS Data Access Portal</h1> <br/> <p>Fill your login/password and click on <strong>Login</strong> to download data. If you are not registred yet click on <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent">Register</a></p> <p>If you have trouble logging in, make sure your browser is set to accept cookies.</p> <br/> <form id="authentification" name="com-login" class="fm-v clearfix" action="/cas/login;jsessionid=BA9435DA67876D405E6490F3DFC023A9?service=http://localhost:8080/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i.dds" method="post"> <fieldset class="input"> <label for="username">Username</label> <input id="username" name="username" class="required" tabindex="1" accesskey="i" type="text" value="" size="25" autocomplete="false"/><br /> <label for="password">Password</label> <input id="password" name="password" class="required" tabindex="2" accesskey="m" type="password" value="" size="25" autocomplete="off"/><br /> <input type="hidden" name="lt" value="_c4793F86B-451F-2DD7-ECFA-D395B0B99CF4_k3D0FA0E2-B51A-ACF8-EF1A-B307A76D684E" /> <input type="hidden" name="_eventId" value="submit" /> <div class="authentification_retrait_right"> <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent"><img src="/cas/newCss/css/myocean/screen/images/bt_register.gif" alt="Register" /></a> <input class="submit" name="submit" accesskey="l" value=" " type="submit" alt="Login" /> </div> </fieldset> <p> <br/><br/> <i>- You are registered but have forgotten your login/password?</i><br/> <i>- Any question?</i><br/> <b>&rArr; CMEMS Service Desk : </b><A HREF="mailto:[email protected]"><b>[email protected]</b></A> </p> </form> <h3>Thank you for using CMEMS products</h3> <br/> <div id="sidebar"> <i>For security reasons, please Exit your web browser when you quit services requiring authentication!</i> </div> </div> </body></html> ncdump: https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i: NetCDF: Access failure

Of course, using python's modules inside the env yields similar results

from opendrift.readers import reader_netCDF_CF_generic
reader_netCDF_CF_generic.Reader('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i')

results in

Traceback (most recent call last):
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '297eccf1-3b01-443c-a779-283fa9800280']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 146, in __init__
    self.Dataset = xr.open_dataset(filename, decode_times=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/root/miniconda3/envs/opendrift/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 148, in __init__
    raise ValueError(e)
ValueError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'

The only instalation that worked is the one via a docker file, but it doesn't suit my needs.

my conda environments is as follows:

     active environment : opendrift
    active env location : /root/miniconda3/envs/opendrift
            shell level : 2
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.28=0
                          __linux=4.18.0=0
                          __unix=0=0
       base environment : /root/miniconda3  (writable)
      conda av data dir : /root/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /root/miniconda3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Linux/4.18.0-425.19.2.el8_7.x86_64 rocky/8.7 glibc/2.28
                UID:GID : 0:0
             netrc file : /root/.netrc
           offline mode : False

# conda environments:
#
base                     /root/miniconda3
opendrift             *  /root/miniconda3/envs/opendrift

sys.version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GC...
sys.prefix: /root/miniconda3
sys.executable: /root/miniconda3/bin/python
conda location: /root/miniconda3/lib/python3.10/site-packages/conda
conda-build: None
conda-content-trust: /root/miniconda3/bin/conda-content-trust
conda-env: /root/miniconda3/bin/conda-env
user site dirs:

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: opendrift
CONDA_EXE: /root/miniconda3/bin/conda
CONDA_PREFIX: /root/miniconda3/envs/opendrift
CONDA_PREFIX_1: /root/miniconda3
CONDA_PROMPT_MODIFIER: (opendrift)
CONDA_PYTHON_EXE: /root/miniconda3/bin/python
CONDA_ROOT: /root/miniconda3
CONDA_SHLVL: 2
CURL_CA_BUNDLE: <not set>
GDAL_DRIVER_PATH: /root/miniconda3/envs/opendrift/lib/gdalplugins
LD_PRELOAD: <not set>
PATH: /root/miniconda3/envs/opendrift/bin:/root/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
http_proxy: <set>
https_proxy: <set>

Any help would be greatly appreciated, I have run out of ideas already.

Thank you very much

@gauteh
Copy link
Member

gauteh commented Apr 13, 2023

It looks like a recent bug in netcdf, maybe you can downgrade that package.

@knutfrode
Copy link
Collaborator

I also get the same problem when updating my environment, and so that netCDF version is 1.6.3
I tried then to downgrade to netCDF 1.6.2 (working on another machine), but same problem.
Also downgrading Xarray to 2022.12.0 (working on another machine) did not work.
Then not sure how to fix this?

@Elcook
Copy link
Author

Elcook commented Apr 14, 2023

Would it be possible that the problem comes from the netCDF library, and not from the netCDF4 python module?

I'm assuming that because ncdump works outside the conda environment ( netcdf library version 4.7.0 ) but fails inside it (netcdf library version 4.9.2).

Anyway, I will try to test this and report here

@vegardb
Copy link

vegardb commented Apr 14, 2023

I can make this work for me with conda and netcdf4=1.6.1

@Elcook
Copy link
Author

Elcook commented Apr 14, 2023

I can make this work for me with conda and netcdf4=1.6.1

You are correct, it worked for me too, even with netcdf library version 4.9.2.

@knutfrode
Copy link
Collaborator

Ok, I have now temporarily locked netcdf4 to <=1.6.1 in opendrift environment.yml, until bug is hopefully solved and backported by someone.
4626d20

@AndresSepulveda
Copy link
Contributor

How do you downgrade to netcd4 1.6.1

@knutfrode
Copy link
Collaborator

knutfrode commented Apr 18, 2023

If you update to latest, environment.yml contains netcdf4<=1.6.1
You can then do
conda env create -f environment.yml --force
followed by
pip install -e .
to reinstall dependencies

@miguelcorreia19
Copy link

Is it possible to define the directory where the .dodsrc is present?
I want to change the directory where these files are present but the opendrift always will try on HOME dir. We can change this behavior?

@knutfrode
Copy link
Collaborator

.dodsrc and .netrc authentication are general, and not a part of OpenDrift.
You can see specifications here: https://docs.unidata.ucar.edu/nug/current/dap2.html

@gauteh gauteh changed the title Trouble accessing CMEMS sources when using conda environment NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources Dec 12, 2023
@gauteh gauteh pinned this issue Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants