I was able to download the data ( json response) from the below URL for a while via curl on Windows.
The Win 10 curl response, same for all three requests.
The response below came from Cloudflare.
Based on the response headers I do not believe these requests were passed to the Host theocc.com
.
Response Header
HTTP/1.1 403 Forbidden
Date: Sat, 19 Nov 2022 04:17:29 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
CF-Chl-Bypass: 1
Referrer-Policy: same-origin
Permissions-Policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
X-Frame-Options: SAMEORIGIN
Set-Cookie: __cf_bm=jNH8U9rGVke8A3M9NVoGyM_54P4yqbD6GUdBex2U9as-1668831449-0-Aajlm5as JsCCdzuStWeD6FNiZE6Q7BqSSGV axg1GOcbz3CREOYW gy0pq0iJpVVZ5HRo2RQA3AQaaFBjNnIsk=; path=/; expires=Sat, 19-Nov-22 04:47:29 GMT; domain=.theocc.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
CF-RAY: 76c622f05c20d9d1-MIA
Response HTML
<!DOCTYPE html>
<html lang="en-US">
<head>
<title>Just a moment...</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<meta name="robots" content="noindex,nofollow">
<meta name="viewport" content="width=device-width,initial-scale=1">
<link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
</head>
<body >
<div role="main">
<div >
<h1 >
<img src="/favicon.ico"
one rror="this.onerror=null;this.parentNode.removeChild(this)">
www.theocc.com
</h1>
<h2 id="challenge-running">
Checking if the site connection is secure
</h2>
<noscript>
<div id="challenge-error-title">
<div >
<span >
<div ></div>
</span>
<span id="challenge-error-text">
Enable JavaScript and cookies to continue
</span>
</div>
</div>
</noscript>
<div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=76c5e617baf0221a')"></div>
<div id="challenge-body-text" >
www.theocc.com needs to review the security of your connection before proceeding.
</div>
<form id="challenge-form" action="/mdapi/series-search?symbol_type=U\&symbol=AA&__cf_chl_f_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0" method="POST" enctype="application/x-www-form-urlencoded">
<input type="hidden" name="md" value="C_LKeu2P5NrY0aR4BxvKBGqNiMKJ1potyy02pUTruns-1668828957-0-ASvIMrC-30PITC58EAtdhq56Jx7fME4EOUOUE-FMk5FBMP_d7Pfr7XFI_Qr6SAVlPAQCGA9_b9beL_agWaV1I44NJRLSb5IkRQ3E6icwIe8Eq7QVMQWYJ-KlvST_JI7znFj_MeqOMA50UlrgSYomFz8TuMTnjbbiiYo9FziI0gyhbYedZ0BtSiUakd9eMeV54maAR1UqybWW52Lj1qpMUZtAes6YaVvVNcjVLtBtjt5Dbspqa8XsoRwpk11VO5GnlLoz4uSqh-laMjyd6zOas7YCD7Qzs05AGIBGKLuPZ3xeyq43ZS0zcGLkxsxwgbF1IWdBz8br6tVuO6YFvY5qSsEeXvymMXbCltsj8tqKDb-XoCbfyFF_Gb33LwFmxYudOqrvZ-2miy-BbZAGrSRGe7_nNDfKf94f_ZRrpSu25VvNhvni46u1AUa4v4wy7O6ujeT9JHpwem_0Y2RUw5QJywl8ZXSFf9yGkYq9us8Dwbvkhcis7ovToZv3eP2Gf0MKjgSIj0twbMVIguUCrluO5vVhdzWu1HJfaZI0htq0Rn8ZWD9g0XF5amnaWs80iXocIIsJKPEhHDuh5jpNOH0suaXfZm7waukxfDxRUURM6Kg4BEEaJrNH0ZAsLLvaDfaQ-XCgr_s2VFRdLEQHMEScQ76P271i_MfhA23Xv0uSlLwPRjcnrrYhhfAOoYXMxUqMBjntLp52kLxAgBlV7fD4l8xe-JNpKYWaqi5IXQKLn87mG3XV4SCeeysfvhT26n4CHWXlgroGtv6AwcGtIR6Hkvg">
<input type="hidden" name="r" value="NEE5A4XvbVzMlgGmfv94hrNQy8HlFwlN9MvsAnrBqEc-1668828957-0-AU6LelP5fDUpZVzOa9xe8l/70P6WeCebWpPrNZ5DR1Chl 2RZwXwTYc2PhlpBrSEc527YLgU biTV2VNZunN7E3l1/kL6wiimvLxzx46KIUC5IXDcXgDJ3uXX5P3eO/x5OE2RSxI4N07jV1pfmsKwsYf0hJtJzCraW rZt7/bVZVBmvFrYOSBTqI5jmZ Lsf4gF kLRBLKoheLKgvvblO8GLixR6SYt56sv8Hp9yS/DWmDoPC3yx8Z7DwvRFkpuvLdOCgqcTGNHGDfKjWQOwNFA1yB0Sc/6 8qD4mQgSPTMlPYdwiMgfERL9aHQy2A8/GQztGizyp nAceOiUhFripvog p3MWYdVNzMD4odVVakRdp/wva0LVzt8rtj1G5FBh6wVA94kCxlDRO6M7pYKQjz0c7ejdYUnFtQYFn4pR9B7H3eZNc2vIIJdTQpyxHNcIbjBWSTn5Bv9aZX/JrXZQE8bxK1PKD9wQtSy2k60cuotYOGmrwjNS4Jgk4bt3SaytN0t02mdvOZ6ZuLK9soRGqIN9iJvZUcU3TCvC pZdcm9x7DXTC9LCSjzkQWQygXwOAfqrf6floZcdmlJIzr6S2XgPN3OAy6Mz9hkc08Vu6qtAYjI2j1igUjLFZw07MIeluznsZCyLipxf/2LWpqMlotab3q8yT9W9WY8480IBsErIOmJ3c q2aptTpJM Cuayb9896SOl9niJP5m9UuwYdiyExE DieDGR2SIuAuzFxU2XuEgC1mgJmvcpkh jTlcvE/6rHbQBtZdY5xyMUSJI0u0G187QdjEYlGkTX7nL7t EKnK/bJCz0qFI/JWEBwY/j3OVAL/0DzI9H3d r6GnxqZZCUEwfBsl/3kLf6PdKzrtP otybkpG RvBd3lfuNx06hRVxmtfVAR3R3WfOCo9o8XOMORG1uvGPggT0DBBVJE8TKv0tbIyE3drvyRjJ4KqLxpPwYB2D/7rwDssEVlZhXf0AAu26R8ajqKWcjy/17tyX4tK8ERknmutokwP/hB4b2VEarMK4XgEapf7alPJvY7KB4NQ694DepNXMnhUncUTynw1T1H T7LTGaSFn9gcZFcNOPBWzIMElyUjfrB6xJEwG PcgitK9IWKZkHanvSeTTbPdkaCt6tEhk0LlR0IVPUCLhxUOKeo/1hLWah6oZU6LtF lHVJ0CF/pNK YnhREmbUaESoeiRs4Ill2CjS 69hl5CAuLtbCXdAoyZUHyBh3wrJBA u8EiqSZdKlXfxR85Y8jOO7td2XqS1tVDCLc4VODyXsjQ2CP2GKTThzlySl4s4aYlefAYb5/DDaFBnM1AKHM2TX/Sl90Pyzs7wAfx/UG9V1PW qNBbhizXJDyxYFBdRjK/dR0Mm4Diau5bv fbyfa1uWW6wywvl6 HwrCbMi7T2fdWzGUwhmyDZzqTlUBHB4ZR8eU0puljriB3uEeH9w11JK/2qQCviYRkvqfCm32lDPel4PIIAJlUb4qdFPyg0b7Faazh/UjyyS hK23EM1HVUCVX QY/aYj9dnzSIBxcy0UQDwmK8TpfcIPXD9iyYzrCQVCaa e5OtSP5ZJwrEX53f5vn/zFJWo0q4xUTMR5 mmrG QTVbfpRlRcbz3tI5qIQGUpSlg/MKSUNqernPf9b49FsHQM2NvssW/pA3XiJZQrVL2L8Xq/Ed0VbCIufQdOSvf1nQntUnGzKMbwordtTmsGA8z4Pogv5C7AoQcpPPbSNgiQL5D1Nvlz6Y1IlnG5ovU51a69ptkyX/CY7bFugJ66djMp9vFF2xrIVp0r1arEPvYzpS1DcVj2WQTdaWtXvj8OEAykHuBu1tD3UgtmAdEPE1aftX36eOhfveVgOUXhc47ObpQS54Vbl5vXPbIfdi3pnSEKyZY3CUtd8pNb7OhqqRw5UeRTN47JDhJITP5J4iLz GiPK7t2ORqDsTnm6T5s3/tksp/pa8/v ooA9hjSmkEssBgIShffOAUEwyIXJesXUoNx83pJyoHYcXoZI0ZfHsNNRoFn 9F TckR0U//xKdeNbkCbyXjJrB7yTOvKGolqx0LuAkx/SKW1isTCAdhs3CvbQNxb/GxlWi8D1PzUwVJeQ==">
</form>
</div>
</div>
<script>
(function(){
window._cf_chl_opt={
cvId: '2',
cType: 'managed',
cNounce: '32555',
cRay: '76c5e617baf0221a',
cHash: '7592d66e2323e80',
cUPMDTk: "\/mdapi\/series-search?symbol_type=U\\&symbol=AA&__cf_chl_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0",
cFPWv: 'b',
cTTimeMs: '1000',
cTplV: 4,
cTplB: 'cf',
cRq: {
ru: 'aHR0cHM6Ly93d3cudGhlb2NjLmNvbS9tZGFwaS9zZXJpZXMtc2VhcmNoP3N5bWJvbF90eXBlPVVcJnN5bWJvbD1BQQ==',
ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwNy4wLjAuMCBTYWZhcmkvNTM3LjM2IEVkZy8xMDcuMC4xNDE4LjQy',
rm: 'R0VU',
d: 'HOWi54 55wh9FM6AuZDX9ANcT34bxF2qiuKEraFDkFFxVWfBHI0H0RQrt4iKgIqp/rTy8AJZBTvoHTeHVUHEU8cHb4I9k9KfbQsqkWEYZbs82TTZ7KS7RE4eFUiXiDSp7Y0QjRU9BL8II412SDZ45C5FFFg8HXvVVScSCJoDt ARF1Ew3Jaw7unS6jp/2QqADMAF7HjZDag23KMOyCqhfXHrFkopkGie3S8ReM8KDMpt/WINtw/ZIztXsz/1gZvG9JYWW V8qsmBzlVrIHcoD70dm38ABF5ki4OZ34mXVsMrah0cRovMSqbMnbHDiOGr3lRnmAhcALHm8fzgAekQ1s95t7GfNd9mz Yextk5Ifx/XiwdGfksP67UBE svAbNUF/XtVzhCNqQjt6T025f DXfYNRCVK5i6ZpkHcZzndk x2uQO6lpzqa5w2yrPr7SMlbD b5vE5XZvNkHTchXBd74S0N68QjUIdxdrS679dCxfrv06qsk57teE5USCsYaxQNhj5n53vDx2FRTWlIJfo2vqHlzyXK2zFAry5MIEjmfYq9FYdciS5ngRM3AxTWA36so3 7syHH4uk3SZb5JQ579YnsmXEiA1ZGzN1h8Vpb9b0g2utYK6QYeRdyN7QZphZZ1Vzt/mMYIA96uakzHfNn3357jDIrZ3kQQDX7NXNo=',
t: 'MTY2ODgyODk1Ny40MDQwMDA=',
m: '/aGrMgSdZO6ogXdUQEEOQsDgNn UAxlig04 B a9YQY=',
i1: 'wffO0nID3av5tavW3mXdtA==',
i2: 'eIr4tQZjzWSBg3EFfM1UhA==',
zh: 'rjV/vZpjqak/inbeUs7nHF3bR1cMk2CD6n2tHAx8WJg=',
uh: 'p19SMC98WhDwvsQz Kqkj/EMDYzdbqDnaATK9trmhK0=',
hh: 'rTcMqa1eOqviz82idTtNQqeGJGdhBGQmQRzkYbDEzA4=',
}
};
var trkjs = document.createElement('img');
trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=76c5e617baf0221a');
trkjs.setAttribute('style', 'display: none');
document.body.appendChild(trkjs);
var cpo = document.createElement('script');
cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/managed/v1?ray=76c5e617baf0221a';
window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
if (window.history && window.history.replaceState) {
var ogU = location.pathname window._cf_chl_opt.cOgUQuery window._cf_chl_opt.cOgUHash;
history.replaceState(null, null, "\/mdapi\/series-search?symbol_type=U\\&symbol=AA&__cf_chl_rt_tk=kUJFHohnkrGLP_aT8f5l1edyZE38Iuvbqeguf9EaPyM-1668828957-0-gaNycGzNCH0" window._cf_chl_opt.cOgUHash);
cpo.onload = function() {
history.replaceState(null, null, ogU);
};
}
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
</script>
<div role="contentinfo">
<div >
<div >
<div >Ray ID: <code>76c5e617baf0221a</code></div>
</div>
<div >Performance & security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
</div>
</div>
</body>
</html>
I also used PHP's curl.
$headers = array();
//$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:107.0) Gecko/20100101 Firefox/107.0';
$headers[] = 'User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0';
$headers[] = 'Accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8';
$headers[] = 'Accept-Language: en-US,en;q=0.5';
$headers[] = 'Accept-Encoding: gzip, deflate, br';
$headers[] = 'DNT: 1';
$headers[] = 'Connection: keep-alive';
$headers[] = 'Upgrade-Insecure-Requests: 1';
$headers[] = 'Sec-Fetch-Dest: document';
$headers[] = 'Sec-Fetch-Mode: navigate';
$headers[] = 'Sec-Fetch-Site: none';
$headers[] = 'Sec-Fetch-User: ?1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT,2);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
echo "Response: $response\n";
echo "\nheader out: \n" . curl_getinfo($ch,CURLINFO_HEADER_OUT);
echo "curl Error: \n" . curl_error($ch);
echo "\n\ncurl_get_info:\n";
var_export(curl_getinfo($ch));
The above PHP code returned no response header or body, just a 403 Forbidden HTTP Status
Response:
header out:
GET /mdapi/series-search?symbol_type=U&symbol=AA HTTP/2
Host: www.theocc.com
user-agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
curl Error:
The requested URL returned error: 403
curl_get_info:
array (
'url' => 'https://www.theocc.com/mdapi/series-search?symbol_type=U&symbol=AA',
'content_type' => 'text/html; charset=UTF-8',
'http_code' => 403,
'header_size' => 954,
'request_size' => 470,
'filetime' => -1,
'ssl_verify_result' => 20,
'redirect_count' => 0,
'total_time' => 0.082475,
'namelookup_time' => 0.000513,
'connect_time' => 0.016717,
'pretransfer_time' => 0.039101,
'size_upload' => 0.0,
'size_download' => 0.0,
'speed_download' => 0.0,
'speed_upload' => 0.0,
'download_content_length' => -1.0,
'upload_content_length' => 0.0,
'starttransfer_time' => 0.0824,
'redirect_time' => 0.0,
'redirect_url' => '',
'primary_ip' => '2606:4700:90:0:3d7d:1d0b:b681:7270',
'certinfo' =>
array (
),
'primary_port' => 443,
'local_ip' => '2604:4500:0:8d:ec4:7aff:fe76:281a',
'local_port' => 46564,
'request_header' => 'GET /mdapi/series-search?symbol_type=U&symbol=AA HTTP/2
Host: www.theocc.com
user-agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
connection: keep-alive
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
',
)
I do not know if this helps, but this URL works in Win 10 cmd.
And works in PHP
NOTE: The escaped \&
did not work on this URL.
This is not hosted on Cloudflare.
>curl "https://marketdata.theocc.com/series-search?symbolType=U&symbol=AA"
Response:
Series/contract Strike Open Interest
ProductSymbol year Month Day Integer Dec C/P Call Put Position Limit
AA 2022 11 18 17 500 C P 0 5950 25000000
AA 2022 11 18 20 000 C P 0 1553 25000000
AA 2022 11 18 22 500 C P 0 3145 25000000
AA 2022 11 18 25 000 C P 0 2714 25000000
AA 2022 11 18 30 000 C P 0 4645 25000000
AA 2022 11 18 32 000 C P 0 396 25000000
AA 2022 11 18 33 000 C P 0 409 25000000
AA 2022 11 18 34 000 C P 0 145 25000000
AA 2022 11 18 35 000 C P 0 3760 25000000
AA 2022 11 18 35 500 C P 0 101 25000000
AA 2022 11 18 36 000 C P 0 211 25000000
AA 2022 11 18 36 500 C P 0 163 25000000