How to retrieve a Span Value by class name on a webpage in Python using beautiful soup?-CodePudding

I am trying to access a price value by class name on a web page in python but I am running into some issues. First of all, the source code I see when I inspect the page directly from the page itself is different from the result I get in python after parsing the page as seen in my code below:

import requests
from bs4 import BeautifulSoup

# Make a request
page = requests.get(
    "https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662")
soup = BeautifulSoup(page.content, 'html.parser')
print(soup)

The result of soup is:

C:\Users\Netchoice\PycharmProjects\selenium\venv\Scripts\python.exe C:/Users/Netchoice/PycharmProjects/selenium/main.py
 <!DOCTYPE html>

<html dir="ltr" lang="en">
<head>
<link as="script" href="https://shopfront.takealot.com/static/js/app-loader.js" rel="preload"/>
<link crossorigin="" href="https://media.takealot.com" rel="preconnect"/>
<title>Takealot</title>
<meta charset="utf-8"/>
<meta content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0" name="viewport"/>
<meta content="ie=edge" http-equiv="X-UA-Compatible"/>
<link href="data:image/x-icon;base64,AAABAAEAEBAAAAEACABoBQAAFgAAAC\
gAAAAQAAAAIAAAAAEACAAAAAAAAAEA\
AAAAAAAAAAAAAAEAAAAAAADWrHgA1a\
p1AL16JgDmzLcAtnYSAP3//wD //8A\
t3YSAP///wDu1ssA vDuAPXi2wDCgz\
sAtnYRAP3//gD //4AuHYRALd1EwD/\
//4Au3okALl2EQC4dRMAuXUTALp1Ew\
D///0At3USALh2EAC4dRIA//7/ALl2\
EAC5dRIA69LDAP76 QDYsX4A2bF AO\
nTxwD//v4AuHURALl1EQDPn2UA0Z9l\
ANiwfwDZsH8At3QSALd1FgC5dhQAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAGxsbGxsbJSsbLRUeGxsbGx\
sbGxsbGxYCHyMBLBsbGxsbGxsbGxsT\
GAgIHAobGxsbGxsbGxsbCwYICAYMGx\
sbGxsbGxsbGyAcHAMHJhsbGxsbGxsb\
GxsICAghFCUbGxsbGxsbGxsbCAgIIh\
sbGxsbGxsbGxsbGwgICCIbGxsbGxsb\
GxsbFhUIBiQpFiUbGxsbGxsbGxAICA\
gGCBIEGxsbGxsbGxsRJAUICAYGJhsb\
GxsbGxsbGSgSCA4JJxAbGxsbGxsbGx\
sbCAgIABsbGxsbGxsbGxsbGwgICAAb\
GxsbGxsbGxsbGxsGDwYqGxsbGxsbGx\
sbGxsbGhcdDRsbGxsbGwAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\
AAAAAAAAAAAAAAA=" rel="icon" sizes="32x32" type="image/x-icon">
<link href="/favicon.ico" rel="shortcut icon" sizes="16x16" type="image/x-icon"/>
<link href="/static/images/apple-touch-icon.png" rel="apple-touch-icon" sizes="180x180" type="image/png"/>
<style>
        html, body {
          background-color: #f4f4f4;
        }
        .sf {
          min-height: 400px;
        }
      </style>
</link></head>
<body id="body">
<div  id="shopfront-app"></div>
<script onl oad="window.loadShopfront('https://shopfront.takealot.com');" src="https://shopfront.takealot.com/static/js/app-loader.js" type="text/javascript">
</script>
</body>
</html>


Process finished with exit code 0

But When I inspect directly on the page I see everything (all the tags) including the one I am targeting which is as follows:

<span >R 137</span>

That is just my first issue. Secondly I am failing to get the value "R137" in python. How can I retrieve this price value and store it in a variable called price based on my initial code? I would really appreciate it if somebody could help me with this.

CodePudding user response：

Content is provided dynamically by JavaScript - Knowing the PLID (end of url) of the product you could use its api and extract all information needed:

import requests, json

url = 'https://api.takealot.com/rest/v-1-10-0/product-details/PLID90167662?platform=desktop'
jsonData = requests.get(url).json()

This will give you all information that is used to display the product:

{'title': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6', 'desktop_href': 'https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'core': {'id': 90167662, 'title': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6', 'subtitle': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6', 'brand': None, 'brand_url': None, 'reviews': 0, 'star_rating': 0.0, 'slug': 'hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6', 'authors': [], 'formats': []}, 'badges': {'items': [{'id': 'badge-0', 'type': 'saving', 'value': '57% off'}]}, 'buybox': {'plid': 90167662, 'product_line_id': 90167662, 'product_id': 200259873, 'tsin': 90190801, 'add_to_cart_text': 'Add to Cart', 'listing_price': 320.0, 'prices': [137], 'loyalty_prices': [{'id': 'loyaltyprices-0', 'display_text': 'eB1,370', 'prices': [1370], 'info_text': None, 'description': None, 'info_mode': None}, {'id': 'loyaltyprices-1', 'display_text': 'Discovery Miles 1,370', 'prices': [1370], 'info_text': None, 'description': None, 'info_mode': None}], 'pretty_price': 'R 137', 'delivery_charges': {}, 'is_add_to_cart_available': True, 'is_add_to_wishlist_available': True, 'variants_call_to_action': None, 'is_free_shipping_available': False, 'is_digital': False, 'is_preorder': False, 'multibuy_display': False, 'multibuy_label': None, 'promotion_qty': None, 'promotion_qty_display_text': None, 'sponsored_ads_seller_id': 'M29838012'}, 'breadcrumbs': {'items': [{'slug': 'computers', 'type': 'department', 'id': 13, 'name': 'Computers & Tablets'}, {'slug': 'accessories-26394', 'type': 'category', 'id': 26394, 'name': 'Accessories'}, {'slug': 'cable-adapters-26395', 'type': 'category', 'id': 26395, 'name': 'Cable Adapters'}]}, 'stock_availability': {'status': 'In stock', 'is_leadtime': False, 'is_imported': False, 'distribution_centres': [{'id': 'distribution_centre_0', 'text': 'JHB', 'description': 'This item is in stock in our Johannesburg warehouse and can be shipped from there.', 'info_type': 'short'}], 'when_do_i_get_it_text': 'When do I get it?', 'when_do_i_get_it_info': '## When do I get it?\n\nOur picking, packing and delivery is linked to what you order, where you want it delivered and how you choose to pay, so delivery times can vary.\nEstimates are provided below to help you shop, but always check your order’s Estimated Delivery Date during checkout for the most accurate delivery information.\n\n### Standard Delivery\n\nDelivery destination              | Delivery time\n----------------------------------|-----------------\nCape Town, Johannesburg, Pretoria | 2-5 working days\nRegional Area                     | 2-6 working days\nRemote Area                       | 3-7 working days\n\n### Premium Delivery Options\n\nOnly available in Main Centres (Cape Town, Johannesburg & Pretoria)\n\nDelivery type               | Delivery time                       | Cut off time\n----------------------------|-------------------------------------|-----------------------------\nSame Day Delivery           | Same Day between 3pm - 7pm          | Pay before 10:30am weekdays\nNext Business Day Delivery  | Next Working Day between 7am - 7pm  | Pay before 11:59pm previous day\nSaturday Delivery           | Saturday between 11am - 7pm         | Pay before 11:59pm Friday\nSunday Delivery             | Sunday between 11am - 7pm           | Pay before 11am Saturday\n\n### Standard Collect\n\nCollect location                                | Collect time\n------------------------------------------------|-----------------\nPickup Point: Cape Town, Johannesburg, Pretoria | 2-5 working days\nPickup Point: Regional Area                     | 2-6 working days\nPickup Point: Remote Area                       | 3-7 working days\n\n### Premium Collect Options\n\nOnly available in Main Centres (Cape Town, Johannesburg & Pretoria)\n\nCollect type     | Collect time       | Cut off time\n-----------------|--------------------|---------------------------------\nSame Day Collect | Same Day from 3pm  | Pay before 10:30am Mon-Sat\nNext Day Collect | Next Day from 10am | Pay before 11:59pm previous day\n\nCollection times vary, please wait for your Ready to Collect email before visiting the pickup point.\n', 'display_seasonal_message': False, 'seasonal_message_text': None, 'seasonal_message_info': None}, 'gallery': {'images': ['https://media.takealot.com/covers_images/64cdaee47c164f908504e0b1d3515e69/s-{size}.file', 'https://media.takealot.com/covers_images/6d0cbf60b46d46d8a204beee758727b8/s-{size}.file', 'https://media.takealot.com/covers_images/1b2c6a62af614ea092c669091c5276fb/s-{size}.file'], 'size_guide': None}, 'product_information': {'tab_title': 'Product Information', 'categories': {'display_name': 'Categories', 'source': 'productline-categories', 'content_type': 'text/markdown', 'displayable_text': '- [Computers & Tablets](/computers) / [Accessories](/computers/accessories-26394) / [Cable Adapters](/computers/cable-adapters-26395)', 'value': [[{'id': 13, 'name': 'Computers & Tablets', 'slug': 'computers'}, {'id': 26394, 'dept': 13, 'name': 'Accessories', 'slug': 'accessories-26394', 'level': 2, 'parent': None, 'children': []}, {'id': 26395, 'dept': 13, 'name': 'Cable Adapters', 'slug': 'cable-adapters-26395', 'level': 1, 'parent': 26394, 'children': []}]], 'id': 'categories-0'}, 'items': [{'display_name': 'Categories', 'source': 'productline-categories', 'content_type': 'text/markdown', 'displayable_text': '- [Computers & Tablets](/computers) / [Accessories](/computers/accessories-26394) / [Cable Adapters](/computers/cable-adapters-26395)', 'value': [[{'id': 13, 'name': 'Computers & Tablets', 'slug': 'computers'}, {'id': 26394, 'dept': 13, 'name': 'Accessories', 'slug': 'accessories-26394', 'level': 2, 'parent': None, 'children': []}, {'id': 26395, 'dept': 13, 'name': 'Cable Adapters', 'slug': 'cable-adapters-26395', 'level': 1, 'parent': 26394, 'children': []}]], 'id': 'categories-0'}, {'display_name': 'Warranty', 'source': 'product-line-attribute-service', 'content_type': 'text/markdown', 'displayable_text': 'Limited \\(6 months\\)', 'value': {'type': {'id': 2, 'name': 'Limited', 'sort_order': 200}, 'period': {'unit': 'm', 'value': 6}}, 'id': 'attributes-0', 'item_type': 'warranty'}, {'display_name': 'TV Audio Adapter', 'source': 'product-line-attribute-service', 'content_type': 'text/markdown', 'displayable_text': 'HDMI to HDMI', 'value': {'id': 20, 'name': 'HDMI to HDMI', 'sort_order': 2000}, 'id': 'attributes-1', 'item_type': 'tv_audio_adapter'}, {'display_name': "What's in the box", 'source': 'product-line-attribute-service', 'content_type': 'text/markdown', 'displayable_text': '1 x HDMI Over CAT5e/6 Transmitter  \n1 x HDMI Over CAT5e/6 Receiver', 'value': '1 x HDMI Over CAT5e/6 Transmitter\n1 x HDMI Over CAT5e/6 Receiver', 'id': 'attributes-2', 'item_type': 'what_s_in_the_box'}, {'display_name': 'Basic Colours', 'source': 'tsin-attribute-service', 'content_type': 'text/markdown', 'displayable_text': 'Blue', 'value': [{'id': 2, 'name': 'Blue', 'sort_order': 200}], 'id': 'attributes-3', 'item_type': 'basic_colors'}, {'display_name': 'Barcode', 'source': 'tsin-barcodes', 'content_type': 'text/markdown', 'displayable_text': '- 539824528628', 'value': ['539824528628'], 'id': 'attributes-4', 'item_type': 'barcode'}]}, 'description': {'tab_title': 'Description', 'html': "<p>ThHDMI Extender 1080P RJ45 Ports LAN Network HDMI Extension up to 30m Over CAT5e/6 UTP LAN Ethernet Cable for HDTV Monitor 1 Pair<br />\nThe HDMI Extender of the CAT-5e/6 cable can only use one of the CAT-5e/6 cables to extend the 1080P HDTV display to 30 meters. The adapter can not only break the limitation of the transmission length of the HDMI cable but also have greater flexibility and convenience in cable construction.</p>\n\n<p>Product characteristics<br />\nA pair of full functional modules, without setting.<br />\n1. Connect the RJ45 port of transmitter and receiver with CAT-5E/6 network wire<br />\n2. CAT-5e/6 cable can replace HDMI cable for long-distance transmission.<br />\n3. The network line follows the standard of IEEE-568B. Using CAT-6 cable, the transmission distance can reach 30 meters.<br />\n5. The product is compact and convenient.<br />\n6. The transmitter and receiver are used in pairs, without setting, without an external power supply.<br />\n7. Signal rate up to 2.5 Gbps, support 1080P, this section does not support HDCP;</p>\n\n<p>The HDMI Extender of the CAT-5e/6 cable can only use one of the CAT-5e/6 cables to extend the 1080P HDTV display to 30 meters. The adapter can not only break the limitation of the transmission length of the HDMI cable but also have greater flexibility and convenience in cable construction.<br />\nProduct characteristics<br />\nA pair of full functional modules, without setting.<br />\n1. Connect the RJ45 port of transmitter and receiver with CAT-5E/6 network wire<br />\n2. CAT-5e/6 cable can replace HDMI cable for long-distance transmission.<br />\n3. The network line follows the standard of IEEE-568B. Using CAT-6 cable, the transmission distance can reach 30 meters.<br />\ne product is compact and convenient.<br />\n6. The transmitter and receiver are used in pairs, without setting, without an external power supply.<br />\n7. Signal rate up to 2.5 Gbps, support 1080P, this section does not support HDCP;</p><p>What's in the box<br />\n1 x HDMI Over CAT5e/6 Transmitter<br />\n1 x HDMI Over CAT5e/6 Receiver</p>"}, 'bullet_point_attributes': {'items': [{'description': '## What does this mean?\n\nYou can pay for this item with cash when your driver delivers your order.\n\n### How does it work?\n\nSelect Cash On Delivery (COD) as your payment option when you checkout.\nHave the exact amount of cash ready for our friendly delivery driver when he arrives.\n\n### Is COD available on every order?\n\nNo. We don’t offer COD for:\n\n-   Digital products (e.g. eBooks, gift vouchers or airtime)\n-   Customised products (e.g. a sports shirt printed with a personalised name)\n-   Orders totalling R5,000 or more\n-   Orders being collected from Takealot pickup points\n-   Delivery addresses falling outside of our Takealot Delivery Team area\n\n\n### If your order is eligible for COD, it will be displayed as a payment option at checkout.\n', 'positive': True, 'text': 'Eligible for Cash on Delivery.', 'info_mode': 'long', 'type': 'eligible_for_cod', 'id': 'bullet-point-0'}, {'description': '## Hassle-Free Exchanges & Returns for 30 Days\n\n### **Returns**\n\nIf you are not completely satisfied, simply return the items to us in their original condition and packaging within 30 days of receipt and we will issue a credit which can be used to place a new order.\n\n### Exchanges  (Fashion, sportswear and shoes only)\n\nThe item must be unworn and unused (other than trials) with the original tags still attached. Packaging should be returned in an undamaged condition with the item.\n\n### When does this policy not apply?\n\nIf the item is:\n- older than 30 days\n- opened, unsealed, used or missing any accessories\n- a digital product such as an eBook, electronic voucher, gaming code or other digital downloads;\n- an audio or video recording or computer software that has been unsealed;\n- a newspaper, periodical or magazine;\n- a foodstuff, beverage, or other product intended for everyday consumption;\n- a nursing or maternity product that has been unsealed, including (but not limited to) breast pumps, bottles, formula, maternity underwear, nappies and wipes;\n- a beauty product or fragrance which has been used;\n- a product which has been personalised or made to your specifications.\n\n\nFind out more about [Exchanges & Returns](https://secure.takealot.com/help#returns).\n', 'positive': True, 'text': 'Hassle-Free Exchanges & Returns for 30 Days.', 'info_mode': 'long', 'type': 'hassle_free_exchanges_and_returns', 'id': 'bullet-point-1'}, {'description': 'Limited warranty, with certain exclusions, as defined by the manufacturer. Please consult the manufacturer for further details.', 'positive': True, 'text': '6-Month Limited Warranty.', 'info_mode': 'short', 'type': 'warranty', 'id': 'bullet-point-2'}]}, 'reviews': {'count': 0, 'href': 'https://api.takealot.com/rest/v-1-10-0/product-reviews/plid/90167662', 'tab_title': 'Reviews', 'link_data': {'path': '/rest/{api_version}/product-reviews/plid/{plid}', 'fields': {'api_version': 'v-1-10-0', 'plid': '90167662'}}, 'star_rating': 0.0, 'distribution': {'num_1_star_ratings': 0, 'num_2_star_ratings': 0, 'num_3_star_ratings': 0, 'num_4_star_ratings': 0, 'num_5_star_ratings': 0}}, 'seller_detail': {'seller_id': 29838012, 'display_name': 'Gapfilled', 'fulfilled_by_takealot': '- Product sold by trusted Seller\n- Packed & shipped by Takealot\n- Customer Service and returns handled by Takealot\n', 'link_data': {'path': '/seller/{seller_slug}?sellers={seller_id}', 'fields': {'seller_id': '29838012', 'seller_slug': 'gapfilled'}}}, 'meta': {'identifier': 'PLID90167662', 'href': 'https://api.takealot.com/rest/v-1-10-0/product-details/PLID90167662?platform=desktop', 'link_data': {'path': '/rest/{api_version}/product-details/{plid}?platform={platform}', 'fields': {'api_version': 'v-1-10-0', 'plid': 'PLID90167662', 'platform': 'desktop'}}, 'date_retrieved': '2022-03-28T20:56:48.089279', 'display': True, 'type': 'ProductDetails'}, 'frequently_bought_together': {'href': 'https://api.takealot.com/rest/v-1-10-0/product-details/PLID90167662/frequently-bought-together?platform=desktop', 'link_data': {'path': '/rest/{api_version}/product-details/{plid_str}/frequently-bought-together?platform={platform}', 'fields': {'api_version': 'v-1-10-0', 'plid_str': 'PLID90167662', 'platform': 'desktop'}}}, 'customers_also_bought': {'href': 'https://api.takealot.com/rest/v-1-10-0/product-details/PLID90167662/customers-also-bought?platform=desktop', 'link_data': {'path': '/rest/{api_version}/product-details/{plid_str}/customers-also-bought?platform={platform}', 'fields': {'api_version': 'v-1-10-0', 'plid_str': 'PLID90167662', 'platform': 'desktop'}}}, 'top_navigation': {'items': [{'id': 'topnav-98', 'text': 'Computers', 'href': '/computers', 'foreground': '#333333', 'background': '#ededed', 'font_styling': []}, {'id': 'topnav-242', 'text': 'Laptops', 'href': '/computers/laptops', 'foreground': '#333', 'background': '#ededed', 'font_styling': []}, {'id': 'topnav-103', 'text': 'Accessories', 'href': '/computers/accessories', 'foreground': '#333333', 'background': '#ededed', 'font_styling': []}, {'id': 'topnav-99', 'text': 'Tablets & Kindles', 'href': '/computers/tablets_and_kindles', 'foreground': '#333333', 'background': '#ededed', 'font_styling': []}, {'id': 'topnav-100', 'text': 'Networking', 'href': '/computers/networking-26387', 'foreground': '#333333', 'background': '#ededed', 'font_styling': []}, {'id': 'topnav-101', 'text': 'Data Storage', 'href': '/computers/external_storage', 'foreground': '#333333', 'background': '#ededed', 'font_styling': []}]}, 'exchanges_and_returns': {'tab_title': 'Exchanges and Returns', 'copy': '### Free exchanges & returns for 30 days\n\nWrong size? Not quite what you wanted? You can exchange or return your purchase up to 30 days from the day you received it.\n\n1. Log your exchange or return request via your [Takealot Account profile](https://secure.takealot.com/account/).\n2. Prepare your return parcel.\n3. Wait for courier collection or drop off the parcel yourself.\n4. If approved, your account will be credited.\n\n[Find out more](https://www.takealot.com/help/fashion-exchanges-returns)\n'}, 'seo': {'title': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6 | Buy Online in South Africa | takealot.com', 'description': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6 available to buy online at takealot.com. Many ways to pay. Eligible for Cash on Delivery. Hassle-Free Exchanges & Returns for 30 Days. 6-Month Limited Warranty. We offer fast, reliable delivery to your door.', 'canonical': 'https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'alternate': {'handheld': 'https://m.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'android': 'android-app://fi.android.takealot/app/takealot.com/product/PLID90167662'}, 'x_robots_tag': 'index, follow, archive, max-image-preview: large', 'last_modified': 'Fri, 04 Mar 2022 14:21:51 GMT'}, 'pdp_ad_slots': {'ad_slots': [{'targeting': {'category': ['26395', '26394'], 'brand': 'generic', 'environment': 'production', 'platform': 'desktop', 'plid': '90167662', 'type': ['13'], 'price': 'Under 1000'}, 'id': 'pdp_slot_1', 'supported_sizes': [{'viewport': [1259, 0], 'sizes': [[300, 50]]}, {'viewport': [1024, 0], 'sizes': [[970, 90]]}, {'viewport': [768, 0], 'sizes': [[728, 90]]}, {'viewport': [490, 0], 'sizes': [[468, 60]]}, {'viewport': [300, 0], 'sizes': [[300, 50]]}, {'viewport': [0, 0], 'sizes': []}], 'type': 'GoogleAdSlot', 'div_id': 'div-gpt-ad-275239196329189789119376002312684635017', 'ad_unit': '/21681910165/pdp_slot_1'}, {'targeting': {'category': ['26395', '26394'], 'brand': 'generic', 'environment': 'production', 'platform': 'desktop', 'plid': '90167662', 'type': ['13'], 'price': 'Under 1000'}, 'id': 'pdp_slot_2', 'supported_sizes': [{'viewport': [1259, 0], 'sizes': [[300, 250]]}, {'viewport': [1024, 0], 'sizes': [[970, 90]]}, {'viewport': [768, 0], 'sizes': [[728, 90]]}, {'viewport': [490, 0], 'sizes': [[468, 60]]}, {'viewport': [300, 0], 'sizes': [[300, 250]]}, {'viewport': [0, 0], 'sizes': []}], 'type': 'GoogleAdSlot', 'div_id': 'div-gpt-ad-66983877907575230555233183003546139299', 'ad_unit': '/21681910165/pdp_slot_2'}]}, 'sharing': {'url': 'https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'enabled': ['facebook', 'twitter', 'pinterest', 'email', 'whatsapp'], 'copy': {'body': {'email': 'Check this out on takealot: HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6
https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'twitter': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6 available to buy online at @TAKEALOT We offer fast, reliable delivery to your door.', 'facebook': 'I love this product on takealot.com: HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6 https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'whatsapp': 'Check this out on takealot.com, HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6 https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662', 'pinterest': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6 | Buy Online in South Africa | https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662'}, 'subject': {'email': 'Check this out on takealot: HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6'}}}, 'data_layer': {'name': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6', 'sku': 90190801, 'event': 'product', 'quantity': 1, 'categoryname': ['Accessories', 'Cable Adapters'], 'totalPrice': 137, 'productlineSku': 'PLID90167662-90190801', 'departmentname': 'Computers & Tablets', 'departmentid': 13, 'pageType': 'product', 'categoryid': [26394, 26395], 'prodid': 'PLID90167662'}, 'event_data': {'documents': {'product': {'product_line_id': 90167662, 'in_stock': True, 'market_place_listing': True, 'sku_id': 200259873, 'purchase_price': 137, 'original_price': 320.0, 'lead_time': 'In stock - shipped from JHB'}}}, 'flixmedia': {'tab_title': 'Featured Content', 'distributor': '6320', 'brand': '', 'mpn': '539824528628', 'ean': '539824528628', 'language': 'za', 'fallback_language': 'en'}, 'enhanced_ecommerce_detail': {'ecommerce': {'detail': {'products': [{'id': 'PLID90167662', 'name': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6', 'brand': None, 'category': 'Computers & Tablets/Accessories/Cable Adapters', 'price': 137, 'quantity': 1, 'position': 0, 'variant': None, 'dimension1': None, 'dimension2': 200259873}], 'actionField': {'list': 'product'}}}, 'event': 'eec.detail'}, 'enhanced_ecommerce_add_to_cart': {'ecommerce': {'add': {'products': [{'id': 'PLID90167662', 'name': 'HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6', 'brand': None, 'category': 'Computers & Tablets/Accessories/Cable Adapters', 'price': 137, 'quantity': 1, 'position': 0, 'variant': None, 'dimension1': None, 'dimension2': 200259873}]}, 'currencyCode': 'ZAR'}, 'event': 'addToCart'}}

Access data needed like:

jsonData['title']
### HDMI Extender LAN Network HDMI Extension up to 30m CAT5e/6

jsonData['buybox']['pretty_price']
### R 137

jsonData['buybox']['listing_price']
### 320.0

CodePudding user response：

The url is entirely dynamic . In this case, you can apply an automation tool something like selenium to grab your desired data. You can just run the code to see the result.

from bs4 import BeautifulSoup
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager


url = 'https://www.takealot.com/hdmi-extender-lan-network-hdmi-extension-up-to-30m-cat5e-6/PLID90167662'

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(5)

soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.close()

price =soup.find("span",class_="currency plus currency-module_currency_29IIm").text
print(price)

Output:

R 137