So I'm trying to web scrape a site which has what I would consider a form - several inputs to specialize search such as <select>s and text inputs - that is not within <form> tags.
Rather, they seem to be grouped in <div>s.

The site queries using JavaScript from there, so essentially all I want to do is fill out the 'form' (various inputs) and then hit the submit button and safe the info returned.

However, I can't find a way in mechanize to grab buttons, selects, etc. that aren't in forms. Is there a way?

CodePudding user response：

Mechanize cannot work on JS interactive websites.

For web scraping, I would suggest using Kimurai framework. It is built on top of Mechanize.

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites.

Use any of the JS engines available while scrapping.

You can execute JS which triggers an event to show form and there on you can use capybara helpers to fill in the form and submit

All the power of Capybara: use methods like click_on, fill_in, select, choose, set, go_back, etc. to