When I make request I get this.
I want the html to load and the run the scripts to create the full body before I parse it.
I working with flutter(dart).
I tried using the webview but I'm not getting the html from after the page is loaded.
When I make the request to the site, this is what the response looks like.
I need some help guys.
How should I go about it?
<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<title>Just a moment...</title>
<style type="text/css">
html,
body {
width: 100%;
height: 100%;
margin: 0;
padding: 0;
}
body {
background-color: #ffffff;
color: #000000;
font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Helvetica Neue", Arial, sans-serif;
font-size: 16px;
line-height: 1.7em;
-webkit-font-smoothing: antialiased;
}
h1 {
text-align: center;
font-weight: 700;
margin: 16px 0;
font-size: 32px;
color: #000000;
line-height: 1.25;
}
p {
font-size: 20px;
font-weight: 400;
margin: 8px 0;
}
p,
.attribution,
{
text-align: center;
}
#spinner {
margin: 0 auto 30px auto;
display: block;
}
.attribution {
margin-top: 32px;
}
@keyframes fader {
0% {
opacity: 0.2;
}
50% {
opacity: 1.0;
}
100% {
opacity: 0.2;
}
}
@-webkit-keyframes fader {
0% {
opacity: 0.2;
}
50% {
opacity: 1.0;
}
100% {
opacity: 0.2;
}
}
#cf-bubbles>.bubbles {
animation: fader 1.6s infinite;
}
#cf-bubbles>.bubbles:nth-child(2) {
animation-delay: .2s;
}
#cf-bubbles>.bubbles:nth-child(3) {
animation-delay: .4s;
}
.bubbles {
background-color: #f58220;
width: 20px;
height: 20px;
margin: 2px;
border-radius: 100%;
display: inline-block;
}
a {
color: #2c7cb0;
text-decoration: none;
-moz-transition: color 0.15s ease;
-o-transition: color 0.15s ease;
-webkit-transition: color 0.15s ease;
transition: color 0.15s ease;
}
a:hover {
color: #f4a15d
}
.attribution {
font-size: 16px;
line-height: 1.5;
}
.ray_id {
display: block;
margin-top: 8px;
}
#cf-wrapper #challenge-form {
padding-top: 25px;
padding-bottom: 25px;
}
#cf-hcaptcha-container {
text-align: center;
}
#cf-hcaptcha-container iframe {
display: inline-block;
}
</style>
<meta http-equiv="refresh" content="35">
<script type="text/javascript">
//<![CDATA[
(function(){
window._cf_chl_opt={
cvId: "2",
cType: "non-interactive",
cNounce: "86849",
cRay: "6a97b753b854da86",
cHash: "31c7b92e6efc7fe",
cPMDTk: "",
cFPWv: "b",
cTTimeMs: "1000",
cRq: {
ru: "aHR0cHM6Ly9nZy5jby51ay9yYWNpbmcvMjctb2N0LTIwMjEvZmFrZW5oYW0tMTI1NQ==",
ra: "UG9zdG1hblJ1bnRpbWUvNy4yOC40",
rm: "R0VU",
d: "kPxE7HAIMaSY0 Q26IXxpuTBsg9MYQ7J/ICHGaynkKTojWIXlwrmVPwT F9kXDNE8Jc60RxAMi3ET5lG3UfaNS8h7kz8aJbykZ4EwheY/kJcnVBvvndt9WGhXuDvU0FOh9cZ9SAgRuYgtmkn hRxxi5owt/uhRmgg5MfZWETvbbMY/jCMrFpPhenjJFusWE1eM0Uz1E7p5hOK/W9P9wDRmC4A8f4z9PTWHMJY6EjsHR5GNEOZzHvWUd5uKRmIbQxylvGKD8smzOR1mUYrI hlAUB8U6wptxbbpDZDcxkEYk7M/bp4oIkQrJNBkPvVGmnVcBvZN2QrSJdUIVFNA9kjKh4AfzpxPMkfqhoJIXOhMQK lr1R7T3wHf/qRMpzvgN5HD8wSySaGq ZMUraHmms1bvI3p25Ril83l3CvfGA573nzB0Pl9wBt1Ilj et9TonO/yCPUT9ncqu/HsuK6zoTsYs1XO9emsj8iGHyH20tEDL/QIQu0Xm Na2nREmz4LLYNg/y/S/IlGc8LjDv548dVWGJxGuPSzeDWod49nzqXZR6u4otbfcHIIMdEgPjTa",
t: "MTYzNjEzMjQ0Mi4yMDcwMDA=",
m: "UHh8TWbrY8rIVckQYr2DEHf3xtqOy3IWbHpurhKo0Dg=",
i1: "bPb8laPvucq92fuTb8UHJQ==",
i2: "uyNfsEipqDYhS6 qFdYUkA==",
zh: "C039uzYEN6IwJDV7FwZVYsBSveHeK850qjEkkfVJ6vw=",
uh: "q2fUGDQwB25BKEmSOHJOj1NUZxjS6q3TVwtPqObOVZM=",
hh: "nfF1CEMggKTkMnWiuwyIjIi54NW2HaYcu4gV2CdK4ks=",
}
}
window._cf_chl_enter = function(){window._cf_chl_opt.p=1};
})();
//]]>
</script>
</head>
<body>
<table width="100%" height="100%" cellpadding="20">
<tr>
<td align="center" valign="middle">
<div class="cf-browser-verification cf-im-under-attack">
<noscript>
<h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the
page.</h1>
</noscript>
<div id="cf-content" style="display:none">
<div id="cf-bubbles">
<div class="bubbles"></div>
<div class="bubbles"></div>
<div class="bubbles"></div>
</div>
<h1><span data-translate="checking_browser">Checking your browser before accessing</span>
gg.co.uk.</h1>
<div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies"
style="display:none">
<p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload
the page.</p>
</div>
<p data-translate="process_is_automatic">This process is automatic. Your browser will redirect
to your requested content shortly.</p>
<p data-translate="allow_5_secs" id="cf-spinner-allow-5-secs">Please allow up to 5
seconds…</p>
<p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">
Redirecting…</p>
</div>
<form class="challenge-form" id="challenge-form"
action="/racing/27-oct-2021/fakenham-1255?__cf_chl_jschl_tk__=I4acmyvWUrWbLXAGjYC5HXUh31EzMntIEES9DIs8Yg8-1636132442-0-gaNycGzNBmU"
method="POST" enctype="application/x-www-form-urlencoded">
<input type="hidden" name="md" value="WYgGQTwD0.PlAYN7zN9M7XpYgnUJxpt6W_rYCboneoQ-1636132442-0-AfDISHvT2GBIpWp1EPYu9cdxCuFQpLT_u7UKTH9TL-mj0X8Xxw9raGV2KUlDtB4DX7Z3h-LYRsfmLlNmrV-WstInbyHg2H3gWXOuMU2mnguq0TdSnDmB0MWffohG7RydsDYePGxxBYBAXhJMAYyRLWlbcprMPKCaHVm9zAxWf5tZFgYfCLP-CLdoXXS_HuKNhA0iXqanK-LYFcSJXKuvTr0M7oAx-vhljyX8JW3sJ2sgJlV7vLLB9MPl3l7WgWBZ4q0bW8KKVhNi3WXReGdIZUpIirNA6D2xHAe5oyfDMqUfEyopDE2EFGeNxKb2yI0qPvmmaup4WjHIG0DangljT0VcnK8oR2-lz7tVreiQQS50DcnX_EiDswWlbfXDvz83RI3IDOmSjBJXQZBSClMjhkCPs9yf5wOGXtq2jjd2sIFFWgw3zS5Q0n9dsTfFCNSFRGtOtHmCM2ztI8ivbpfvf27CNK55Xb7b2BPCOICarJUqM16ELz7AAQ6lS0OJymzCU0G1eC58DDvCiMCx6AC7nAI2ce38OjfjYHO5Wq_lI8u8VFrGaS4nZyySvn6-0gGqRA" />
<input type="hidden" name="r" value="i7Mahy4asxjhgyvRAVdQk8RCAO5shxagJMSNb2vkdXM-1636132442-0-Acl9elCj95M0xGc183cGdekXyIhJ RXABhl9YAfeIxJZd5a0u4EX8e1wb7wJdeYsBwO/e42SiuMP3O5t3PoeEibaVnEIPnLgipqmvVXoSiOuK302AlzY5JmI0Xy80sRBsoyWLglYAEPZft7cFoS XK3ayn3GMqgoQpnf8nhjeVmOx6g1v25gqwGYoz6MdvE/7Z2GAwvCnqP2Bk7fjp7gBBQ1pWSgzCEP2jXsbYuqcBE7HHlNbMyJF1tqOIAd1Hm7S8KtAQUllGHNfJy7s/L6Ibvg55SCP8XHUlAqASeMvqbJors5HimAx iZgpjfYhetytGpQTqh8m7BlVyZBUkvVl6HA3r0eR0OSVOOUf/8BmuYTMrZKqkElg BY4sNTy0UqLeuyLH5RS3zMLdNDx75IkGexDVPx2KXBxPwdmPhVO2jmoJdSl ocBTzfPqjsXw XAyo9Sm5id7TZ/Z xxXYA2RMCpBiGBXPdfu2U CJI4/S/6TWfmunwpMwRIfEby1fQNmvNCJWJPIybXpZR0tOH/eln7CoeR8qaSrCUjP7H5fNSgiIi/gtH2oxm3D20y5T4Hd1EGEtKf0E1pLUGSeZY8LBsIMtvPlnXr5srsz4965 WZu87RK9w8filse 9rtIT6KM55QcgIE7Cvb4yfI74x7y uJ0bjUoc8D3ke5GjK4isNsMlh0kxgY9gVJmzxReZ9n9ahT77FOcOJh8eQEQOCPZQ8J9DqUduOkH7775AvPseKj1CPb3 EJvoaZGdNIfYvUV6PjMAsnthNOaCKDQ8oHiYZLW8A8YxF2AUrQxMBGjeIjvv8OCAkFwk4RhX/JSqNXgWE mj6x ySSNFtBJp7cOEdB27Teyx11ImRvTXBZdv09XFIgYRO7WYmI/1sB/qfn65uL1Pr0chh5mCX8FgX1r/CRw/IkvJhLpDPNxq0MiXJK/gUuZK7URNCQI3ep Dv 7lyuvi9sQgkXl2ne8fSp8Nbh3MogLWFAYx0o7Jz1iZY4Vtryw87I e6Ditw9eklqURwEiSE7yIJ5foZGLZwClJs9W1pfpZ2G2OIU8HBvN/O4ERCqT2nLRVJH513AUzPv9DpowCMUxANLCAIvJL9XmSbVeUBvUQmdpOK grZlgFRx6d8/kKepJStd8iuN/P9SGFPmOGKQHmECFiu70JPk4QCS3J08BjhuIpC4uvfLNmAzQi1yYEOpwSfv1XPf4DLS/GO59qg/jbgu0UZyRBQDL/KE4wOiE8Dtw7ACjkUtaON1NDgOS5nKkUu0diqIIRvreQdLCy4ehjuvG7LuWPPdrEmU4ZYS3R IPHNBEBbak2pszPGm8Xc0GtVWldmtAeRr sN0K8JeRPIbORtPuhwMPpGGosCiPmqxHshlyS0qPB0kYsykHvjDXzI63E3EccRyo353ByRSLe0wcz0k32Ku4I7uDGN/YmQEcggLEVsShXXUFA FOgQzuM6PE9o NPgke/M1mJBIuyyOrBtQFNskIgXQA0jPVVpJz84OeR3ylkTsVDGOvnhj5nYPEk27gDxHVoYc2C6Omi/FwRaZcGQrBrTxCTejlnmEnGTUd M"/>
<input type="hidden" value="3734809bfba8e7119ff4a9e747f0a631" id="jschl-vc" name="jschl_vc"/>
<!-- <input type="hidden" value="" id="jschl-vc" name="jschl_vc"/> -->
<input type="hidden" name="pass" value="1636132443.207-qh8v/uVVqU"/>
<input type="hidden" id="jschl-answer" name="jschl_answer"/>
</form>
<script type="text/javascript">
//<![CDATA[
(function(){
var a = document.getElementById('cf-content');
a.style.display = 'block';
var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
var trkjs = isIE ? new Image() : document.createElement('img');
trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=6a97b753b854da86");
trkjs.id = "trk_jschal_js";
trkjs.setAttribute("alt", "");
document.body.appendChild(trkjs);
var cpo=document.createElement('script');
cpo.type='text/javascript';
cpo.src="/cdn-cgi/challenge-platform/h/b/orchestrate/jsch/v1?ray=6a97b753b854da86";
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
//]]>
</script>
<div id="trk_jschal_nojs"
style="background-image:url('/cdn-cgi/images/trace/jschal/nojs/transparent.gif?ray=6a97b753b854da86')">
</div>
</div>
<div class="attribution">
DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/"
target="_blank">Cloudflare</a>
<br />
<span class="ray_id">Ray ID: <code>6a97b753b854da86</code></span>
</div>
</td>
</tr>
</table>
</body>
</html>CodePudding user response：
try to move <script> at the top of your app
CodePudding user response：
I used the flutter WebView and it worked.
I wrote the full js script in a file.
The script does the parsing and returns the required data as json.
So I pass the script file content to my WebView controller's evaluateJavaScript and the result will be json. So I can just take that data and work with it.
// webview's controller
WebViewController? _controller;
// setup your webview and get the controller
WebView(
initialUrl: widget.url ?? 'https://www.google.com.tr',
onWebViewCreated: (controller) => _controller = controller,
javascriptMode: JavascriptMode.unrestricted,
gestureNavigationEnabled: true,
onPageFinished: (_) => readJS(),
);
//
void readJS() async {
String js = await fileReader.readFile("lib/js/main.js"); // read script file
String data = await _controller!.evaluateJavascript(js);
print("data\n$data");
}