there is a website called

In my script I dump out the form and the input fields required with the following:

my @forms = $mech->forms;
foreach my $form (@forms) {
  my @inputfields = $form->param;
  print Dumper \@inputfields;
}

resulting in

$VAR1 = [
          '__EVENTTARGET',
          '__EVENTARGUMENT',
          '__LASTFOCUS',
          '__VIEWSTATE',
          '__VIEWSTATEGENERATOR',
          '__EVENTVALIDATION',
          'txtCount',
          'chkUppercase',
          'chkBrackets',
          'chkHypens',
          'chkBase64',
          'chkRFC7515',
          'chkURL',
          'LocalTimestampValue',
          'btnGenerate',
          'txtResults'
        ];

and this is the post

my $mainpage = "https://www.guidgenerator.com/online-guid-generator.aspx";
$mech->post( "$mainpage",
     fields      => {
       'txtCount' => "1",
       'chkBase64' => "on",
       'LocalTimestampValue' => "Date().getTime()",
       'btnGenerate' => "Generate some GUIDs!",
       'txtResults' => "",
       '__EVENTTARGET' => 'on',
       '__EVENTARGUMENT', => 'on',
       '__LASTFOCUS', => 'on',
       '__VIEWSTATEGENERATOR' => "247C709F",
       '__VIEWSTATE' => 'on',
       '__EVENTVALIDATION' => 'on',
       'chkUppercase' => 'off',
       'chkBrackets' => 'off',
       'chkHypens' => 'off',
       'chkRFC7515' => 'off',
       'chkURL' => 'off',
     },
);

When I do the trace on the website I get the headers but there is another tab called Payload. That contains most of the fields listed above. I try to input these fields into the POST but not sure if I should be doing this differently or it doesn't matter because its javascript?

I know this is a lot of information. I'm not even sure that perl's mechanize can pull this information. Any help would be appreciated. Please let me know any other data you want me to post here.

CodePudding user response：

You can use Mech's built-in stuff to do this. There is no need to submit any extra fields or headers.

use strict;
use warnings;
use feature 'say';

use WWW::Mechanize;

my $mech = WWW::Mechanize->new;
$mech->get('https://www.guidgenerator.com/online-guid-generator.aspx');
$mech->field( txtCount => 10 );
$mech->click;

say $mech->value('txtResults');

This will output something like:

$ perl mech.pl                                  
211b3cad1665483ca303360bdbda0c61
ecc3348d83cb4bb5bdcb11c6148c5ae1
0a3f2fe5748946a1888a4a5bde8ef2e6
acb26deb9fda4411aa64638cdd1ec5f1
2afe609c355b4a10b6a0ae8c74d3aef1
30fd89ab170147cfb24f131346a203e3
2301d258e1d045aa8f0682f2ea14464c
f064507ca3e14a4eb860b0a30ba096ed
9a42b15d5c79420c921dcc07c306459b
5bea2e345f75453caaf795681963866a

The crux here was that you cannot use $mech->submit as that wouldn't submit the value of the submit button. That's a bit annoying. So instead, you have to use $mech->click, which pretends the default form's default submit button was clicked, hence submitting that value as well. That's just how buttons work on forms, and in this case the backend checks the value to see which one was clicked.

You can then use $mech->value to get the field value out. You'd probably want to split it to process it further.

The JavaScript in this page is actually completely irrelevant to the functionality. All it does is save and restore the settings you've chosen in a cookie, so that when you come back the same checkboxes will be ticked. That's nice, but nowadays probably better done with local storage on the frontend side. However you don't have to deal with the JS at all to crawl this page. The main functionality is backend side.

You might also be interested in $mech->dump_forms, which is a nice debugging aid that prints out all the forms with fields and values. Another good debugging aid when working with Mech (or any LWP based class) is LWP::ConsoleLogger::Everywhere. That's what I used to compare my program's request with my browser's one to find the missing button form field.

_{Disclaimer: I am a maintainer of WWW::Mechanize and I wrote LWP::ConsoleLogger::Everywhere.}