Home > Mobile >  Beautiful Soup Python How to scrape specific value in multiple divs
Beautiful Soup Python How to scrape specific value in multiple divs

Time:07-09

I am trying to get the value from one of the tspans at the bottom of this html code but have gotten stuck, so far I have got this far but I know I'm not even close.

Many thanks in advance

soup = bs(s.get(URL).text, 'html.parser')
divTag = soup.find_all("div", {"class": "tablebox"})
results = soup.find(id="exoTooltip")
elements = results.find_all("g", class_="exoTooltip undefined")

I've gotten to the point of being able to log in and scrape but am stuck on how to get the specific value

<body style="background: rgb(71, 71, 71);">
   <div id="exo-charts">
      <div>
         <div id="exo-center" style="position: absolute; z-index: 1; top: 0px; left: 45px; width: 657px; height: 691px;">
            <div id="exo-top" style="position: absolute; top: 0px; left: 0px; width: 657px; height: 35px; background: rgb(38, 38, 38);">
               <div  style="height: 35px;">
                  <div  style="color: rgb(228, 228, 228); border-right: 1px solid rgb(71, 71, 71);">
                     <div  title="Ticker">
                        <div >
                           <div id="topPeriodSelector"  style="color: rgb(228, 228, 228); min-width: 50px;"><span  style="color: rgb(228, 228, 228);">&nbsp;</span>Bybit: BTCUSD</div>
                        </div>
                     </div>
                  </div>
                  <div  style="color: rgb(228, 228, 228); border-right: 1px solid rgb(71, 71, 71);">
                     <div  title="Rotation period">
                        <div >
                           <div id="topPeriodSelector"  style="color: rgb(228, 228, 228); min-width: 50px;">
                              <span  style="color: rgb(228, 228, 228);">&nbsp;</span>
                              <div style="color: rgb(228, 228, 228);">TR-210-140</div>
                           </div>
                        </div>
                     </div>
                  </div>
                  <div  style="color: rgb(228, 228, 228); border-right: 1px solid rgb(71, 71, 71);">
                     <div  title="Tick per bar multiplier">
                        <div >
                           <div >
                              Tick *&nbsp;<input  maxlength="5" type="text" tabindex="-1" spellcheck="false" value="20" style="font-family: &quot;Trebuchet MS&quot;;">
                              <div id="topTickMultiplierMore"  style="color: rgb(228, 228, 228);"><span >&nbsp;</span></div>
                              <div id="topTickMultiplierLess"  style="color: rgb(228, 228, 228);"><span >&nbsp;</span></div>
                              <div style="width: 80px; height: 10px; font-size: 9px; position: absolute; bottom: -1px; margin-left: 1px; z-index: 999999999; color: rgb(135, 135, 135);">1 tick = 0.5</div>
                           </div>
                        </div>
                     </div>
                  </div>
                  <div>&nbsp;</div>
                  <div  style="color: rgb(228, 228, 228);">
                     <div  title="Cluster mode">
                        <div style="width: 69px; height: 10px; font-size: 9px; position: absolute; top: 1px; margin-left: 7px; z-index: 999999999; color: rgb(135, 135, 135);">Cluster</div>
                        <div  style="border-right: 1px solid rgb(71, 71, 71);">
                           <div id="topChartTypeSelector"  style="color: rgb(228, 228, 228); border: 1px solid rgb(71, 71, 71); min-width: 105px;">Delta Profile&nbsp;<span  style="color: rgb(228, 228, 228);">&nbsp;</span></div>
                        </div>
                     </div>
                  </div>
                  <div  style="color: rgb(228, 228, 228);">
                     <div  title="Text mode">
                        <div style="width: 69px; height: 10px; font-size: 9px; position: absolute; top: 1px; margin-left: 7px; z-index: 999999999; color: rgb(135, 135, 135);">Text</div>
                        <div  style="border-right: 1px solid rgb(71, 71, 71);">
                           <div id="topTextTypeSelector"  style="color: rgb(228, 228, 228); min-width: 81px;">Bid-Ask&nbsp;<span  style="color: rgb(228, 228, 228);">&nbsp;</span></div>
                        </div>
                     </div>
                  </div>
                  <div  title="Direction candle" style="color: rgb(228, 228, 228);">
                     <div >
                        <div style="width: 69px; height: 10px; font-size: 9px; position: absolute; top: 1px; margin-left: 7px; z-index: 999999999; color: rgb(135, 135, 135);">Candle</div>
                        <div  style="border-right: 1px solid rgb(71, 71, 71);">
                           <div id="topDCTypeSelector"  style="color: rgb(228, 228, 228); min-width: 92px;">Mono Candle&nbsp;<span  style="color: rgb(228, 228, 228);">&nbsp;</span></div>
                        </div>
                     </div>
                  </div>
                  <div  style="color: rgb(135, 135, 135); margin-left: 1px;">
                     <div >
                        <div  title="Volume">
                           <div  style="color: rgb(135, 135, 135);">Vol</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Trades Count Sell/Buy">
                           <div  style="color: rgb(135, 135, 135);">TCount</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Liquidations">
                           <div  style="color: rgb(228, 228, 228);">Rekt</div>
                        </div>
                     </div>
                     <div  style="border-right: 1px solid rgb(71, 71, 71);">&nbsp;</div>
                     <div >
                        <div  title="Footprint Bar Statistics">
                           <div  style="color: rgb(228, 228, 228);">FPBS</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Trade Size">
                           <div  style="color: rgb(135, 135, 135);">TSize</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Candle Statistics">
                           <div  style="color: rgb(228, 228, 228);">CS</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Delta bars">
                           <div  style="color: rgb(228, 228, 228);">DBars</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Open Interest">
                           <div  style="color: rgb(228, 228, 228);">OI</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Net Longs">
                           <div  style="color: rgb(135, 135, 135);">NL</div>
                        </div>
                     </div>
                     <div >
                        <div  title="Net Shorts">
                           <div  style="color: rgb(135, 135, 135);">NS</div>
                        </div>
                     </div>
                     <div >
                        <div  title="vWap">
                           <div  style="color: rgb(228, 228, 228);">vWap</div>
                        </div>
                     </div>
                  </div>
                  <div data-no-selfie="true"  style="color: rgb(170, 170, 170);">
                     <div >
                        <div ></div>
                     </div>
                  </div>
                  <span data-yes-selfie="true" id="exoName" style="color: rgb(170, 170, 170); display: none; position: fixed; font-size: 10px; top: 10px; right: 10px;">exocharts.com</span><span data-no-selfie="true" title="Save as default preset. Applies to local browser when opening new tab" style="color: rgb(170, 170, 170); width: 50px; height: 22px; position: fixed; right: 50px; top: 12px; cursor: pointer;">Default</span><span data-no-selfie="true" title="Save into selected template (overwrites)" style="color: rgb(170, 170, 170); width: 50px; height: 22px; position: fixed; right: 110px; top: 12px; cursor: pointer;">Save tpl</span>
                  <span title="Take screen shot">
                     <svg data-no-selfie="true" data-place="left" data-offset="{'top': 0, 'left': 5}" id="Kappa_1" xmlns="http://www.w3.org/2000/svg" x="0px" y="0px" viewBox="0 0 297 297" style="color: rgb(170, 170, 170); width: 22px; height: 22px; position: fixed; right: 10px; top: 5px; cursor: pointer;">
                        <g>
                           <path d="M297,69.439c0-33.65-27.376-61.027-61.026-61.027c-33.651,0-61.028,27.377-61.028,61.027
                              c0,23.979,13.907,44.763,34.074,54.733v81.815c0,5.617,4.555,10.171,10.172,10.171h33.564c5.617,0,10.172-4.554,10.172-10.171
                              v-81.815C283.094,114.201,297,93.418,297,69.439z M242.585,195.817h-13.223v-65.714c2.174,0.234,4.377,0.363,6.611,0.363
                              c2.234,0,4.438-0.129,6.611-0.363V195.817z M235.974,110.124c-22.435,0-40.686-18.251-40.686-40.685
                              c0-22.434,18.251-40.685,40.686-40.685c22.434,0,40.684,18.251,40.684,40.685C276.657,91.873,258.407,110.124,235.974,110.124z"></path>
                           <path d="M194.779,150.767h-19.707c0.022-0.28,0.043-0.562,0.043-0.848c0-5.617-4.555-10.171-10.171-10.171h-16.783
                              c-5.617,0-10.171,4.554-10.171,10.171c0,0.286,0.02,0.567,0.043,0.848h-3.29c-0.093-0.09-0.179-0.188-0.275-0.274l-16.409-16.411
                              c-1.907-1.906-4.494-2.979-7.192-2.979H94.084c-2.698,0-5.285,1.072-7.192,2.979l-16.685,16.686H43.655
                              c-0.271-2.252-1.292-4.394-2.902-6.004c-1.892-1.893-4.506-2.98-7.19-2.98c-2.675,0-5.3,1.088-7.191,2.98
                              c-1.611,1.61-2.632,3.752-2.901,6.004H10.171C4.554,150.767,0,155.321,0,160.938v117.479c0,5.617,4.554,10.171,10.171,10.171
                              h184.608c5.617,0,10.171-4.554,10.171-10.171V160.938C204.95,155.321,200.396,150.767,194.779,150.767z M184.608,268.245H20.343
                              v-97.136h164.266V268.245z"></path>
                           <path d="M102.476,267.482c26.359,0,47.805-21.444,47.805-47.805c0-26.36-21.445-47.805-47.805-47.805
                              c-26.36,0-47.805,21.444-47.805,47.805C54.671,246.038,76.115,267.482,102.476,267.482z M102.476,192.216
                              c15.143,0,27.462,12.319,27.462,27.462c0,15.143-12.319,27.462-27.462,27.462s-27.463-12.319-27.463-27.462
                              C75.013,204.535,87.333,192.216,102.476,192.216z"></path>
                           <circle cx="102.477" cy="219.678" r="10.171"></circle>
                        </g>
                     </svg>
                  </span>
               </div>
            </div>
            <div style="height: 655px; margin-top: 36px; background-image: linear-gradient(rgb(32, 32, 32), rgb(1, 1, 1)); border-radius: 0px 0px 1px 1px;">
               <div  style="position: relative; width: 657px; height: 655px; float: left;">
                  <div style="position: absolute; z-index: 1;">
                     <canvas id="exoGrid" width="1314" height="1310" style="position: absolute; width: 657px; height: 655px; z-index: 0;"></canvas>
                     <canvas id="exoMain" width="1314" height="1310" style="position: absolute; width: 657px; height: 655px; z-index: 1;"></canvas>
                     <canvas id="exoOverlayCPL" width="1314" height="1310" style="position: absolute; width: 657px; height: 655px; z-index: 2;"></canvas>
                     <canvas id="exoOverlayCPR" width="1314" height="1310" style="position: absolute; width: 657px; height: 655px; z-index: 3;"></canvas>
                     <canvas id="exoOverlay" width="1314" height="1310" style="position: absolute; width: 657px; height: 655px; z-index: 4;"></canvas>
                     <canvas data-no-selfie="true" id="exoCapture" width="1314" height="1310" style="position: absolute; width: 657px; height: 655px; z-index: 5;"></canvas>
                  </div>
                  <svg  width="657" height="655" style="position: absolute; z-index: 6;">
                     <style type="text/css">
                        .exoGrabCursor {
                        pointer-events: all;
                        cursor: -moz-grabbing;
                        cursor: -webkit-grabbing;
                        cursor: grabbing;
                        }
                        .exoCrosHairCursor {
                        pointer-events: all;
                        cursor: default;
                        }
                        .exoTooltip {
                        pointer-events: all;
                        cursor: pointer;
                        }
                        .exoAvoidInter {
                        pointer-events: none;
                        }
                        .exoEnableInter {
                        pointer-events: all;
                        }
                        .exoTooltip {
                        pointer-events: all;
                        cursor: pointer;
                        }
                        .exoDefCursor {
                        cursor: default;
                        }
                        .exoMoveCursor {
                        cursor: move;
                        }
                        .exoPointerCursor {
                        cursor: pointer;
                        }
                        .exoResizeCursor{
                        cursor: n-resize;
                        }
                        .exoEastWestResizeCursor {
                        cursor: ew-resize;
                        }
                     </style>
                     <defs>
                        <clipPath id="chart-area-clip">
                           <rect x="0" y="0" width="582" height="633"></rect>
                        </clipPath>
                        <clipPath id="chart-area-clip-1">
                           <rect x="0" y="0" width="582" height="435"></rect>
                        </clipPath>
                        <clipPath id="chart-area-clip-57">
                           <rect x="0" y="0" width="582" height="69"></rect>
                        </clipPath>
                        <clipPath id="chart-area-clip-56">
                           <rect x="0" y="0" width="582" height="67"></rect>
                        </clipPath>
                        <clipPath id="chart-area-clip-54">
                           <rect x="0" y="0" width="582" height="62"></rect>
                        </clipPath>
                     </defs>
                     <g transform="translate(0, 0)">
                        <rect  width="582" height="633" style="opacity: 0;"></rect>
                        <g >
                           <g transform="translate(0, 0)">
                              <g>
                                 <g  transform="translate(8, 16)">
                                    <text font-size="12" x="0" y="0" font-family="Trebuchet MS" >
                                       <tspan fill="#f3f3f3">O</tspan>
                                       <tspan fill="#E93C3C">21,823&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">H</tspan>
                                       <tspan fill="#E93C3C">21,834&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">L</tspan>
                                       <tspan fill="#E93C3C">21,733&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">C</tspan>
                                       <tspan fill="#E93C3C">21,780&nbsp;&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">Vol</tspan>
                                       <tspan fill="#E93C3C">7.3M&nbsp;&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">D</tspan>
                                       <tspan fill="#E93C3C">-3.4M&nbsp;&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">Trades b:279&nbsp;</tspan>
                                       <tspan fill="#f3f3f3">a:224&nbsp;</tspan>
                                    </text>
                                 </g>
                              </g>
                              <g>
                                 <g transform="translate(8, 32)">
                                    <text  x="0" y="0" fill="#f3f3f3" font-family="Trebuchet MS" font-size="11">Delta Profile , Adaptive, S: 3</text>
                                 </g>
                              </g>
                              <g transform="translate(582, 0)">
                                 <rect  x="0" y="0" opacity="0" height="435" width="100"></rect>
                              </g>
                              <g transform="translate(0, 0)" height="100%" width="100%">
                                 <line x1="51" y1="0" x2="51" y2="435"  style="stroke: transparent; stroke-width: 10; cursor: col-resize;"></line>
                              </g>
                              <g transform="translate(0, 0)" height="100%" width="100%">
                                 <line x1="400" y1="0" x2="400" y2="435"  style="stroke: transparent; stroke-width: 10; cursor: col-resize;"></line>
                              </g>
                              <g></g>
                           </g>
                           0
                           <g transform="translate(0, 435)">
                              <g transform="translate(0, 0)" height="100%" width="100%">
                                 <line x1="0" y1="0" x2="582" y2="0"  style="stroke: transparent; stroke-width: 5; cursor: row-resize;"></line>
                              </g>
                              <g>
                                 <g transform="translate(5, 13)">
                                    <svg xmlns="http://www.w3.org/2000/svg" width="11" height="11" viewBox="0 0 24 24" fill="none" stroke="#f3f3f3" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"  x="0" y="-9">
                                       <circle cx="12" cy="12" r="3"></circle>
                                       <path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1 0 2.83 2 2 0 0 1-2.83 0l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83 0 2 2 0 0 1 0-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1-2-2 2 2 0 0 1 2-2h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 0-2.83 2 2 0 0 1 2.83 0l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 2-2 2 2 0 0 1 2 2v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 0 2 2 0 0 1 0 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-.09a1.65 1.65 0 0 0-1.51 1z"></path>
                                    </svg>
                                    <text  font-family="Trebuchet MS" font-size="11" fill="#f3f3f3" x="18" y="0">OI - Open Interest, OI Delta</text>
                                 </g>
                              </g>
                              <g transform="translate(582, 0)">
                                 <rect  x="0" y="0" opacity="0" height="69" width="100"></rect>
                              </g>
                              <g transform="translate(0, 0)">
                                 <line x1="0" y1="70" x2="582" y2="70" style="stroke: rgb(66, 71, 73); stroke-width: 2;"></line>
                                 <line x1="582" y1="0" x2="582" y2="71" style="stroke: rgb(66, 71, 73); stroke-width: 2;"></line>
                              </g>
                              <g>
                                 <g transform="translate(568, 3)">
                                    <svg xmlns="http://www.w3.org/2000/svg" width="11" height="11" viewBox="0 0 24 24" fill="none" stroke="#f3f3f3" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"  opacity="0.2" x="0" y="0">
                                       <line x1="12" y1="19" x2="12" y2="5"></line>
                                       <polyline points="5 12 12 5 19 12"></polyline>
                                    </svg>
                                 </g>
                              </g>
                              <g>
                                 <g transform="translate(554, 3)">
                                    <svg xmlns="http://www.w3.org/2000/svg" width="11" height="11" viewBox="0 0 24 24" fill="none" stroke="#f3f3f3" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"  opacity="0.6" x="0" y="0">
                                       <line x1="12" y1="5" x2="12" y2="19"></line>
                                       <polyline points="19 12 12 19 5 12"></polyline>
                                    </svg>
                                 </g>
                              </g>
                           </g>
                           <g transform="translate(0, 504)">
                              <g transform="translate(0, 0)" height="100%" width="100%">
                                 <line x1="0" y1="0" x2="582" y2="0"  style="stroke: transparent; stroke-width: 5; cursor: row-resize;"></line>
                              </g>
                              <g>
                                 <g transform="translate(5, 13)">
                                    <svg xmlns="http://www.w3.org/2000/svg" width="11" height="11" viewBox="0 0 24 24" fill="none" stroke="#f3f3f3" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"  x="0" y="-9">
                                       <circle cx="12" cy="12" r="3"></circle>
                                       <path d="M19.4 15a1.65 1.65 0 0 0 .33 1.82l.06.06a2 2 0 0 1 0 2.83 2 2 0 0 1-2.83 0l-.06-.06a1.65 1.65 0 0 0-1.82-.33 1.65 1.65 0 0 0-1 1.51V21a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-.09A1.65 1.65 0 0 0 9 19.4a1.65 1.65 0 0 0-1.82.33l-.06.06a2 2 0 0 1-2.83 0 2 2 0 0 1 0-2.83l.06-.06a1.65 1.65 0 0 0 .33-1.82 1.65 1.65 0 0 0-1.51-1H3a2 2 0 0 1-2-2 2 2 0 0 1 2-2h.09A1.65 1.65 0 0 0 4.6 9a1.65 1.65 0 0 0-.33-1.82l-.06-.06a2 2 0 0 1 0-2.83 2 2 0 0 1 2.83 0l.06.06a1.65 1.65 0 0 0 1.82.33H9a1.65 1.65 0 0 0 1-1.51V3a2 2 0 0 1 2-2 2 2 0 0 1 2 2v.09a1.65 1.65 0 0 0 1 1.51 1.65 1.65 0 0 0 1.82-.33l.06-.06a2 2 0 0 1 2.83 0 2 2 0 0 1 0 2.83l-.06.06a1.65 1.65 0 0 0-.33 1.82V9a1.65 1.65 0 0 0 1.51 1H21a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-.09a1.65 1.65 0 0 0-1.51 1z"></path>
                                    </svg>
                                    <text  font-family="Trebuchet MS" font-size="11" fill="#f3f3f3" x="18" y="0">Delta bars</text>
                                 </g>
                              </g>
                              <g transform="translate(582, 0)">
                                 <rect  x="0" y="0" opacity="0" height="67" width="100"></rect>
                              </g>
                              <g transform="translate(0, 0)">
                                 <line x1="0" y1="68" x2="582" y2="68" style="stroke: rgb(66, 71, 73); stroke-width: 2;"></line>
                                 <line x1="582" y1="0" x2="582" y2="69" style="stroke: rgb(66, 71, 73); stroke-width: 2;"></line>
                              </g>
                              <g>
                                 <g transform="translate(568, 3)">
                                    <svg xmlns="http://www.w3.org/2000/svg" width="11" height="11" viewBox="0 0 24 24" fill="none" stroke="#f3f3f3" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"  opacity="0.6" x="0" y="0">
                                       <line x1="12" y1="19" x2="12" y2="5"></line>
                                       <polyline points="5 12 12 5 19 12"></polyline>
                                    </svg>
                                 </g>
                              </g>
                              <g>
                                 <g transform="translate(554, 3)">
                                    <svg xmlns="http://www.w3.org/2000/svg" width="11" height="11" viewBox="0 0 24 24" fill="none" stroke="#f3f3f3" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"  opacity="0.2" x="0" y="0">
                                       <line x1="12" y1="5" x2="12" y2="19"></line>
                                       <polyline points="19 12 12 19 5 12"></polyline>
                                    </svg>
                                 </g>
                              </g>
                           </g>
                           <g transform="translate(0, 571)">
                              <g>
                                 <g transform="translate(5, 13)">
                                    <svg 

https://i.stack.imgur.com/jTJrA.png screenshot of the html if needed

CodePudding user response:

You can use a CSS selector find the text "Trades" and then call find_previous() to get the previous tspan:


soup = BeautifulSoup(html, 'html.parser')
print(
    soup.select_one('.exoTooltip.undefined tspan:-soup-contains("Trades")').find_previous('tspan').text
)

Prints:

-3.4M
  • Related