I have a scrape on a JS
driven HTML
page that updates found products on the same page. I want to grab the products and parse them out. I want to then move onto the next page, which I have already managed using Watir
. The problem is that Watir
cannot see the first page's items. Each item is formatted in the HTML
below. Here is my code after, then the result:
## HTML
<div data-jplist-item="" >
<div >
<a href="/en/Products/All/Tipless Chine" target="_blank" title="Click here to learn more about this product">
<div >(1711) Red Chine, Tips Removed<span style="display:none;">boat part, DELTAUPDATE</span></div>
<div style="display:none;">25.575</div>
<div style="display:none;">1711</div>
<div >SALE</div></a>
</div>
</div>
## code
##### a is the browser
item = a.div(class: "itemSpace")
ap item.inspect
puts item.class
puts
item2 = a.div(class: "itemContainer")
ap item2.inspect
puts item2.class
## result
"#<Watir::Div: located: false; {:class=>\"itemSpace\", :tag_name=>\"div\"}>"
Watir::Div
"#<Watir::Div: located: false; {:class=>\"itemContainer\", :tag_name=>\"div\"}>"
Watir::Div
As you can see, the div
is not located. I'm under the impression Watir
manages JS
returns, so i don't know why it isn't visible.
Anybody know what I'm doing wrong? Different gem (Ferrum
) needs to be used?
CodePudding user response:
The located: false
does not mean that Watir cannot see the element or that the element does not exist. It just means that the Watir has not tried to find the element yet. Once you perform an action on the element, then it will be located: true
.
For example, if you inspected the class
attribute (rather than the object's class), the :located
will change to true
.
item = a.div(class: "itemSpace")
p item.inspect
#=> "#<Watir::Div: located: false; {:class=>\"itemSpace\", :tag_name=>\"div\"}>"
p item.class_name # note that this is #class_name not #class
#=> "itemSpace"
p item.inspect
#=> "#<Watir::Div: located: true; {:class=>\"itemSpace\", :tag_name=>\"div\"}>"
As to why Watir no longer sees the first page of results, that would depend on what happens to them when loading the second page of results. If the first page of results is removed from the page (ie user can no longer see them), then Watir would not be able to see it either. You'd want to capture the first page of results before going to the next page.