curl authentication works with `--netrc` but not `--user`-CodePudding

With curl 7.74.0, the command

$ curl -b ~/.cookies -c ~/.cookies -L --user 'USERNAME:PASSWORD' https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt

yields HTTP 401 (verbose output listed below). However, the following works:

$ echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' >>~/.netrc
$ curl -b ~/.cookies -c ~/.cookies -L --netrc https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt

Why is that? Shouldn't these approaches be equivalent? Perhaps there's an extra option I need to convince curl to pass the credentials set with --user down to urs.earthdata.nasa.gov.

You can create an Earthdata account at https://urs.earthdata.nasa.gov/users/new to test these commands.

(N.B. The --user version works if you already have a valid login cookie from the --netrc version. The question is why login doesn't work.)

The -v output for the --user version is (authenticating cookies etc. censored with "XXX"):

*   Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
*  start date: Jul 14 00:00:00 2020 GMT
*  expire date: Oct 11 12:00:00 2022 GMT
*  subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Date: Mon, 25 Apr 2022 13:02:02 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Content-Length: 381
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Unauthorized</title>
</head><body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested.  Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>
* Connection #0 to host daac.ornl.gov left intact

The -v output for the --netrc version is:

* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
*   Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
*  start date: Jul 14 00:00:00 2020 GMT
*  expire date: Oct 11 12:00:00 2022 GMT
*  subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Location: https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXXA&response_type=code&redirect_uri=https://daac.ornl.gov/daacdata/doesntmater&state=XXX
< Content-Length: 518
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://urs.earthdata.nasa.gov/oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https://daac.ornl.gov/daacdata/doesntmater&state=XXX'
*   Trying 198.118.243.33:443...
* Connected to urs.earthdata.nasa.gov (198.118.243.33) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Maryland; L=Greenbelt; jurisdictionC=US; O=NASA (National Aeronautics and Space Administration); businessCategory=Government Entity; serialNumber=1958-07-29; CN=urs.earthdata.nasa.gov
*  start date: Aug 27 16:09:18 2021 GMT
*  expire date: Sep 12 16:09:18 2022 GMT
*  subjectAltName: host "urs.earthdata.nasa.gov" matched cert's "urs.earthdata.nasa.gov"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
*  SSL certificate verify ok.
* Server auth using Basic with user 'Kodiologist'
> GET /oauth/authorize?app_type=401&client_id=XXX&response_type=code&redirect_uri=https://daac.ornl.gov/daacdata/doesntmater&state=XXX HTTP/1.1
> Host: urs.earthdata.nasa.gov
> Authorization: Basic XXX
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.20.1
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Cache-Control: no-store
< Pragma: no-cache
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Access-Control-Allow-Origin: null
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Methods: GET, POST
< Access-Control-Expose-Headers: true
< Location: https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX
* Added cookie urs_user_already_logged="yes" for domain earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: urs_user_already_logged=yes; domain=earthdata.nasa.gov; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; secure; HttpOnly
* Added cookie _urs-gui_session="XXX" for domain urs.earthdata.nasa.gov, path /, expire 1650978188
< Set-Cookie: _urs-gui_session=XXX; path=/; expires=Tue, 26 Apr 2022 13:03:08 GMT; HttpOnly
< X-Request-Id: XXX
< X-Runtime: 0.053834
< Strict-Transport-Security: max-age=31536000
< 
* Ignoring the response-body
* Connection #1 to host urs.earthdata.nasa.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/doesntmater?code=XXX&state=XXX'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host daac.ornl.gov
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#0)
> GET /daacdata/doesntmater?code=XXX&state=XXX HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Mon, 25 Apr 2022 13:03:08 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
* Added cookie AppAuth="XXX" for domain daac.ornl.gov, path /, expire 0
< Set-Cookie: AppAuth=XXX; Path=/
< Location: https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
< Content-Length: 284
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
* Connection #0 to host daac.ornl.gov left intact
* Issue another request to this URL: 'https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt'
* Couldn't find host daac.ornl.gov in the .netrc file; using defaults
* Found bundle for host daac.ornl.gov: XXX [serially]
* Can not multiplex, even if we wanted to!
* Connection 0 seems to be dead!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Hostname daac.ornl.gov was found in DNS cache
*   Trying 160.91.19.24:443...
* Connected to daac.ornl.gov (160.91.19.24) port 443 (#2)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* SSL re-using session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Tennessee; L=Oak Ridge; O=Oak Ridge National Laboratory; OU=DAAC; CN=*.ornl.gov
*  start date: Jul 14 00:00:00 2020 GMT
*  expire date: Oct 11 12:00:00 2022 GMT
*  subjectAltName: host "daac.ornl.gov" matched cert's "*.ornl.gov"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt HTTP/1.1
> Host: daac.ornl.gov
> User-Agent: curl/7.74.0
> Accept: */*
> Cookie: AppAuth=XXX
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 25 Apr 2022 13:03:09 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000
< Last-Modified: Mon, 07 Dec 2020 22:32:53 GMT
< ETag: "5b5-5b5e76833e87e"
< Accept-Ranges: bytes
< Content-Length: 1461
< Vary: Accept-Encoding
< Content-Type: text/plain; charset=UTF-8
< 
station_id,station_name,stn_lat,stn_lon,stnz
RQC00660152,AGUIRRE,17.9556,-66.2222,7.6
RQC00664193,GUAYAMA 2E,17.9783,-66.0875,21.9
RQC00665693,MAGUEYES IS,17.9722,-67.0461,3.7
VQC00671740,CHRISTIANSTED FT,17.7469,-64.7014,9.1
VQW00011624,CHRISTIANSTED AP,17.7028,-64.8056,18.6
RQC00660061,ADJUNTAS SUBSTN,18.1747,-66.7978,557.8
RQC00660158,AIBONITO 1 S,18.1281,-66.2642,710.2
RQC00660426,ARECIBO OBSY,18.3494,-66.7525,323.1
RQC00662801,COLOSO,18.3808,-67.1569,12.2
RQC00663431,DOS BOCAS,18.3361,-66.6667,61.0
RQC00664614,HUMACAO NATURAL RESERVE,18.1506,-66.7719,2.7
RQC00664702,ISABELA SUBSTN,18.4653,-67.0525,128.0
RQC00665097,LAJAS SUBSTN,18.0331,-67.0722,27.4
RQC00665807,MANATI 2 E,18.4308,-66.4661,76.2
RQC00665908,MARICAO 2 SSW,18.1511,-66.9889,863.2
RQC00666730,PALMAREJO VEGA BAJA,18.385,-66.43,170.7
RQC00667292,PONCE 4 E,18.0258,-66.5253,21.3
RQC00668306,RIO PIEDRAS EXP STN,18.3906,-66.0542,28.0
RQC00669415,TOA BAJA LEVITTOWN,18.4356,-66.1678,8.5
RQC00669432,TORO NEGRO FOREST,18.1731,-66.4928,868.1
RQC00669521,TRUJILLO ALTO 2 SSW,18.3283,-66.0164,35.1
RQW00011641,SAN JUAN L M MARIN AP,18.4325,-66.0108,2.7
RQC00665064,JUNCOS 1 SE,18.2264,-65.9114,64.9
RQC00666343,CULEBRA HILL,18.2972,-65.29,71.3
RQC00666725,PALMA SOLA,18.3169,-65.8664,466.0
RQC00668814,WFO SAN JUAN,18.4311,-65.9917,3.0
RQW00011630,ROOSEVELT ROADS,18.255,-65.6408,10.1
VQC00677600,REDHOOK BAY ST THOMAS,18.3236,-64.8542,0.6
VQW00011640,CHARLOTTE AMALIE AP,18.3331,-64.9667,6.1
* Connection #2 to host daac.ornl.gov left intact

CodePudding user response：

You're using quotes around the username and password. In my experience this causes issues.

try

curl -b ~/.cookies -c ~/.cookies -L --user USERNAME:PASSWORD https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt

This has issues with your user and password being present in bash history.

This link has a lot of good info on other ways to do this safely too

https://stackoverflow.com/a/56130884/496405

CodePudding user response：

in your netrc you specifically say "this is the username:password for urs.earthdata.nasa.gov", and your command first connects to a different domain: daac.ornl.gov. so your netrc does not trigger on daac.ornl.gov, your "broken" curl command sends the username:password to daac.ornl.gov and gets rejected, while your working command does not send username:password to daac.ornl.gov, then daac.ornl.gov proceed to redirect you to to urs.earthdata.nasa.gov , here your netrc kicks in and your username:password gets sent to urs.earthdata.nasa.gov as it should..

unfortunately curl does not (currently?) support loading netrc from stdin, so your best bet is probably:

NETRCTMPFILE="$(mktemp)"
echo 'machine urs.earthdata.nasa.gov login USERNAME password PASSWORD' > "$NETRCTMPFILE";
curl -b ~/.cookies -c ~/.cookies -L --netrc-file "$NETRCTMPFILE" https://daac.ornl.gov/daacdata/daymet/Daymet_V4_Stn_Level_CrossVal/data/stnxvalmeta_tmin_pr_2016.txt
rm "$NETRCTMPFILE"