Home > Software engineering >  Downloading with curl when site uses cookies
Downloading with curl when site uses cookies

Time:11-19

I was able to download the data ( json response) from the below URL for a while via curl on Windows.

curl.exe enter image description here



The Win 10 curl response, same for all three requests.
The response below came from Cloudflare.
Based on the response headers I do not believe these requests were passed to the Host theocc.com.

Response Header

HTTP/1.1 403 Forbidden
Date: Sat, 19 Nov 2022 04:17:29 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
CF-Chl-Bypass: 1
Referrer-Policy: same-origin
Permissions-Policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
X-Frame-Options: SAMEORIGIN
Set-Cookie: __cf_bm=jNH8U9rGVke8A3M9NVoGyM_54P4yqbD6GUdBex2U9as-1668831449-0-Aajlm5as JsCCdzuStWeD6FNiZE6Q7BqSSGV axg1GOcbz3CREOYW gy0pq0iJpVVZ5HRo2RQA3AQaaFBjNnIsk=; path=/; expires=Sat, 19-Nov-22 04:47:29 GMT; domain=.theocc.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
CF-RAY: 76c622f05c20d9d1-MIA

Response HTML

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">


</head>
<body >
    <div  role="main">
    <div >
        <h1 >
            <img  src="/favicon.ico"
                 one rror="this.onerror=null;this.parentNode.removeChild(this)">
            www.theocc.com
        </h1>
        <h2  id="challenge-running">
            Checking if the site connection is secure
        </h2>
        <noscript>
            <div id="challenge-error-title">
                <div >
                    <span >
                        <div ></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=76c5e617baf0221a')"></div>
        <div id="challenge-body-text" >
            www.theocc.com needs to review the security of your connection before proceeding.
        </div>
        <form id="challenge-form" action="/mdapi/series-search?symbol_type=U\&amp;symbol=AA&amp;__cf_chl_f_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="C_LKeu2P5NrY0aR4BxvKBGqNiMKJ1potyy02pUTruns-1668828957-0-ASvIMrC-30PITC58EAtdhq56Jx7fME4EOUOUE-FMk5FBMP_d7Pfr7XFI_Qr6SAVlPAQCGA9_b9beL_agWaV1I44NJRLSb5IkRQ3E6icwIe8Eq7QVMQWYJ-KlvST_JI7znFj_MeqOMA50UlrgSYomFz8TuMTnjbbiiYo9FziI0gyhbYedZ0BtSiUakd9eMeV54maAR1UqybWW52Lj1qpMUZtAes6YaVvVNcjVLtBtjt5Dbspqa8XsoRwpk11VO5GnlLoz4uSqh-laMjyd6zOas7YCD7Qzs05AGIBGKLuPZ3xeyq43ZS0zcGLkxsxwgbF1IWdBz8br6tVuO6YFvY5qSsEeXvymMXbCltsj8tqKDb-XoCbfyFF_Gb33LwFmxYudOqrvZ-2miy-BbZAGrSRGe7_nNDfKf94f_ZRrpSu25VvNhvni46u1AUa4v4wy7O6ujeT9JHpwem_0Y2RUw5QJywl8ZXSFf9yGkYq9us8Dwbvkhcis7ovToZv3eP2Gf0MKjgSIj0twbMVIguUCrluO5vVhdzWu1HJfaZI0htq0Rn8ZWD9g0XF5amnaWs80iXocIIsJKPEhHDuh5jpNOH0suaXfZm7waukxfDxRUURM6Kg4BEEaJrNH0ZAsLLvaDfaQ-XCgr_s2VFRdLEQHMEScQ76P271i_MfhA23Xv0uSlLwPRjcnrrYhhfAOoYXMxUqMBjntLp52kLxAgBlV7fD4l8xe-JNpKYWaqi5IXQKLn87mG3XV4SCeeysfvhT26n4CHWXlgroGtv6AwcGtIR6Hkvg">
            <input type="hidden" name="r" value="NEE5A4XvbVzMlgGmfv94hrNQy8HlFwlN9MvsAnrBqEc-1668828957-0-AU6LelP5fDUpZVzOa9xe8l/70P6WeCebWpPrNZ5DR1Chl 2RZwXwTYc2PhlpBrSEc527YLgU biTV2VNZunN7E3l1/kL6wiimvLxzx46KIUC5IXDcXgDJ3uXX5P3eO/x5OE2RSxI4N07jV1pfmsKwsYf0hJtJzCraW rZt7/bVZVBmvFrYOSBTqI5jmZ Lsf4gF kLRBLKoheLKgvvblO8GLixR6SYt56sv8Hp9yS/DWmDoPC3yx8Z7DwvRFkpuvLdOCgqcTGNHGDfKjWQOwNFA1yB0Sc/6 8qD4mQgSPTMlPYdwiMgfERL9aHQy2A8/GQztGizyp nAceOiUhFripvog p3MWYdVNzMD4odVVakRdp/wva0LVzt8rtj1G5FBh6wVA94kCxlDRO6M7pYKQjz0c7ejdYUnFtQYFn4pR9B7H3eZNc2vIIJdTQpyxHNcIbjBWSTn5Bv9aZX/JrXZQE8bxK1PKD9wQtSy2k60cuotYOGmrwjNS4Jgk4bt3SaytN0t02mdvOZ6ZuLK9soRGqIN9iJvZUcU3TCvC pZdcm9x7DXTC9LCSjzkQWQygXwOAfqrf6floZcdmlJIzr6S2XgPN3OAy6Mz9hkc08Vu6qtAYjI2j1igUjLFZw07MIeluznsZCyLipxf/2LWpqMlotab3q8yT9W9WY8480IBsErIOmJ3c q2aptTpJM Cuayb9896SOl9niJP5m9UuwYdiyExE DieDGR2SIuAuzFxU2XuEgC1mgJmvcpkh jTlcvE/6rHbQBtZdY5xyMUSJI0u0G187QdjEYlGkTX7nL7t EKnK/bJCz0qFI/JWEBwY/j3OVAL/0DzI9H3d r6GnxqZZCUEwfBsl/3kLf6PdKzrtP otybkpG RvBd3lfuNx06hRVxmtfVAR3R3WfOCo9o8XOMORG1uvGPggT0DBBVJE8TKv0tbIyE3drvyRjJ4KqLxpPwYB2D/7rwDssEVlZhXf0AAu26R8ajqKWcjy/17tyX4tK8ERknmutokwP/hB4b2VEarMK4XgEapf7alPJvY7KB4NQ694DepNXMnhUncUTynw1T1H T7LTGaSFn9gcZFcNOPBWzIMElyUjfrB6xJEwG PcgitK9IWKZkHanvSeTTbPdkaCt6tEhk0LlR0IVPUCLhxUOKeo/1hLWah6oZU6LtF lHVJ0CF/pNK YnhREmbUaESoeiRs4Ill2CjS 69hl5CAuLtbCXdAoyZUHyBh3wrJBA u8EiqSZdKlXfxR85Y8jOO7td2XqS1tVDCLc4VODyXsjQ2CP2GKTThzlySl4s4aYlefAYb5/DDaFBnM1AKHM2TX/Sl90Pyzs7wAfx/UG9V1PW qNBbhizXJDyxYFBdRjK/dR0Mm4Diau5bv fbyfa1uWW6wywvl6 HwrCbMi7T2fdWzGUwhmyDZzqTlUBHB4ZR8eU0puljriB3uEeH9w11JK/2qQCviYRkvqfCm32lDPel4PIIAJlUb4qdFPyg0b7Faazh/UjyyS hK23EM1HVUCVX QY/aYj9dnzSIBxcy0UQDwmK8TpfcIPXD9iyYzrCQVCaa e5OtSP5ZJwrEX53f5vn/zFJWo0q4xUTMR5 mmrG QTVbfpRlRcbz3tI5qIQGUpSlg/MKSUNqernPf9b49FsHQM2NvssW/pA3XiJZQrVL2L8Xq/Ed0VbCIufQdOSvf1nQntUnGzKMbwordtTmsGA8z4Pogv5C7AoQcpPPbSNgiQL5D1Nvlz6Y1IlnG5ovU51a69ptkyX/CY7bFugJ66djMp9vFF2xrIVp0r1arEPvYzpS1DcVj2WQTdaWtXvj8OEAykHuBu1tD3UgtmAdEPE1aftX36eOhfveVgOUXhc47ObpQS54Vbl5vXPbIfdi3pnSEKyZY3CUtd8pNb7OhqqRw5UeRTN47JDhJITP5J4iLz GiPK7t2ORqDsTnm6T5s3/tksp/pa8/v ooA9hjSmkEssBgIShffOAUEwyIXJesXUoNx83pJyoHYcXoZI0ZfHsNNRoFn 9F TckR0U//xKdeNbkCbyXjJrB7yTOvKGolqx0LuAkx/SKW1isTCAdhs3CvbQNxb/GxlWi8D1PzUwVJeQ==">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cType: 'managed',
            cNounce: '32555',
            cRay: '76c5e617baf0221a',
            cHash: '7592d66e2323e80',
            cUPMDTk: "\/mdapi\/series-search?symbol_type=U\\&symbol=AA&__cf_chl_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0",
            cFPWv: 'b',
            cTTimeMs: '1000',
            cTplV: 4,
            cTplB: 'cf',
            cRq: {
                ru: 'aHR0cHM6Ly93d3cudGhlb2NjLmNvbS9tZGFwaS9zZXJpZXMtc2VhcmNoP3N5bWJvbF90eXBlPVVcJnN5bWJvbD1BQQ==',
                ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwNy4wLjAuMCBTYWZhcmkvNTM3LjM2IEVkZy8xMDcuMC4xNDE4LjQy',
                rm: 'R0VU',
                d: 'HOWi54 55wh9FM6AuZDX9ANcT34bxF2qiuKEraFDkFFxVWfBHI0H0RQrt4iKgIqp/rTy8AJZBTvoHTeHVUHEU8cHb4I9k9KfbQsqkWEYZbs82TTZ7KS7RE4eFUiXiDSp7Y0QjRU9BL8II412SDZ45C5FFFg8HXvVVScSCJoDt ARF1Ew3Jaw7unS6jp/2QqADMAF7HjZDag23KMOyCqhfXHrFkopkGie3S8ReM8KDMpt/WINtw/ZIztXsz/1gZvG9JYWW V8qsmBzlVrIHcoD70dm38ABF5ki4OZ34mXVsMrah0cRovMSqbMnbHDiOGr3lRnmAhcALHm8fzgAekQ1s95t7GfNd9mz Yextk5Ifx/XiwdGfksP67UBE svAbNUF/XtVzhCNqQjt6T025f DXfYNRCVK5i6ZpkHcZzndk x2uQO6lpzqa5w2yrPr7SMlbD b5vE5XZvNkHTchXBd74S0N68QjUIdxdrS679dCxfrv06qsk57teE5USCsYaxQNhj5n53vDx2FRTWlIJfo2vqHlzyXK2zFAry5MIEjmfYq9FYdciS5ngRM3AxTWA36so3 7syHH4uk3SZb5JQ579YnsmXEiA1ZGzN1h8Vpb9b0g2utYK6QYeRdyN7QZphZZ1Vzt/mMYIA96uakzHfNn3357jDIrZ3kQQDX7NXNo=',
                t: 'MTY2ODgyODk1Ny40MDQwMDA=',
                m: '/aGrMgSdZO6ogXdUQEEOQsDgNn UAxlig04 B a9YQY=',
                i1: 'wffO0nID3av5tavW3mXdtA==',
                i2: 'eIr4tQZjzWSBg3EFfM1UhA==',
                zh: 'rjV/vZpjqak/inbeUs7nHF3bR1cMk2CD6n2tHAx8WJg=',
                uh: 'p19SMC98WhDwvsQz Kqkj/EMDYzdbqDnaATK9trmhK0=',
                hh: 'rTcMqa1eOqviz82idTtNQqeGJGdhBGQmQRzkYbDEzA4=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=76c5e617baf0221a');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/managed/v1?ray=76c5e617baf0221a';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname   window._cf_chl_opt.cOgUQuery   window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/mdapi\/series-search?symbol_type=U\\&symbol=AA&__cf_chl_rt_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0"   window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


    <div  role="contentinfo">
        <div >
            <div >
                <div >Ray ID: <code>76c5e617baf0221a</code></div>
            </div>
            <div >Performance &amp; security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
        </div>
    </div>
</body>
</html>

I also used PHP's curl.

$headers = array();
//$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0';
$headers[] = 'User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0';
$headers[] = 'Accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8';
$headers[] = 'Accept-Language: en-US,en;q=0.5';
$headers[] = 'Accept-Encoding: gzip, deflate, br';
$headers[] = 'DNT: 1';
$headers[] = 'Connection: keep-alive';
$headers[] = 'Upgrade-Insecure-Requests: 1';
$headers[] = 'Sec-Fetch-Dest: document';
$headers[] = 'Sec-Fetch-Mode: navigate';
$headers[] = 'Sec-Fetch-Site: none';
$headers[] = 'Sec-Fetch-User: ?1';

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT,2);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch,  CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
echo "Response: $response\n";
echo "\nheader out: \n" . curl_getinfo($ch,CURLINFO_HEADER_OUT);
echo "curl Error: \n" . curl_error($ch);
echo "\n\ncurl_get_info:\n";
var_export(curl_getinfo($ch));

The above PHP code returned no response header or body, just a 403 Forbidden HTTP Status

Response: 

header out: 
GET /mdapi/series-search?symbol_type=U&symbol=AA HTTP/2
Host: www.theocc.com
user-agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1

curl Error: 
The requested URL returned error: 403

curl_get_info:
array (
  'url' => 'https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA',
  'content_type' => 'text/html; charset=UTF-8',
  'http_code' => 403,
  'header_size' => 954,
  'request_size' => 470,
  'filetime' => -1,
  'ssl_verify_result' => 20,
  'redirect_count' => 0,
  'total_time' => 0.082475,
  'namelookup_time' => 0.000513,
  'connect_time' => 0.016717,
  'pretransfer_time' => 0.039101,
  'size_upload' => 0.0,
  'size_download' => 0.0,
  'speed_download' => 0.0,
  'speed_upload' => 0.0,
  'download_content_length' => -1.0,
  'upload_content_length' => 0.0,
  'starttransfer_time' => 0.0824,
  'redirect_time' => 0.0,
  'redirect_url' => '',
  'primary_ip' => '2606:4700:90:0:3d7d:1d0b:b681:7270',
  'certinfo' => 
  array (
  ),
  'primary_port' => 443,
  'local_ip' => '2604:4500:0:8d:ec4:7aff:fe76:281a',
  'local_port' => 46564,
  'request_header' => 'GET /mdapi/series-search?symbol_type=U&symbol=AA HTTP/2
Host: www.theocc.com
user-agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1

',
)

I do not know if this helps, but this URL works in Win 10 cmd.
And works in PHP
NOTE: The escaped \& did not work on this URL.

This is not hosted on Cloudflare.

>curl "https://marketdata.theocc.com/series-search?symbolType=U&symbol=AA"

Response:

                Series/contract         Strike                  Open Interest
ProductSymbol   year    Month   Day     Integer Dec     C/P     Call    Put     Position Limit
AA              2022    11      18      17      500     C P     0       5950    25000000
AA              2022    11      18      20      000     C P     0       1553    25000000
AA              2022    11      18      22      500     C P     0       3145    25000000
AA              2022    11      18      25      000     C P     0       2714    25000000
AA              2022    11      18      30      000     C P     0       4645    25000000
AA              2022    11      18      32      000     C P     0       396     25000000
AA              2022    11      18      33      000     C P     0       409     25000000
AA              2022    11      18      34      000     C P     0       145     25000000
AA              2022    11      18      35      000     C P     0       3760    25000000
AA              2022    11      18      35      500     C P     0       101     25000000
AA              2022    11      18      36      000     C P     0       211     25000000
AA              2022    11      18      36      500     C P     0       163     25000000
  • Related