'Connection reset by peer' error with web scraping-CodePudding

I am trying to get HTML data using urllib but I keep getting Connection reset by peer error. I have tried to use header option to circumvent proxy related issue, but seems like it is not working. Any help would be highly appreciated

import urllib.request
url = "https://wolt.com/az/aze/baku/restaurant/qutabxana"
hdr = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' }

req = urllib.request.Request(url, headers=hdr)
response = urllib.request.urlopen(req)
response.read()

Error response:

---------------------------------------------------------------------------
   
File /opt/conda/envs/bifrost/lib/python3.8/ssl.py:1309, in SSLSocket.do_handshake(self, block)
   1308         self.settimeout(None)
-> 1309     self._sslobj.do_handshake()
   1310 finally:

ConnectionResetError: [Errno 104] Connection reset by peer

CodePudding user response：

Use requests module and for parsing use Beautiful soap package for python.

https://beautiful-soup-4.readthedocs.io/en/latest/

Here is the example for web scraper https://realpython.com/beautiful-soup-web-scraper-python/

CodePudding user response：

Use direct requests module along with .get() method

import requests
res= requests.get('https://wolt.com/az/aze/baku/restaurant/qutabxana')
print(res)

Output:

<Response [200]>

Full working code with an example:

from bs4 import BeautifulSoup 
import requests
res= requests.get('https://wolt.com/az/aze/baku/restaurant/qutabxana')

soup = BeautifulSoup(res.text,'lxml')
for title in soup.select('[] p'):
    print(title.text)

Output:

Qarın Qutabı
Qarın, qursaq, iç piyi, duz, istiot
Göy Qutabı
İspanaq, keşniş, cincilim, gicitkən, turşəng, kəvər, soğan göy
Pendirli Göy Qutabı
Ağ pendir, göyərtilər
Pendir Qutabı
Ağ pendir
Ət Qutabı
Ət, quyruq, duz istiot
Kartof Qutabı
Kartof pure
Sarı Noxud Qutabı
Noxud puresi
Pendirli (Holland) Qutab
Holland pendiri
Ətli və Holland Pendiri ilə Qutab
Ət ve holland pendiri
Kartoflu Pirojki
Kartof pure
Ciyər ilə Pirojki
Ağ və qara ciyər
Ətli Pirojki
Ət
Sarı Noxud ilə Pirojki
Noxud
Cemli Ponçik
Alma cemi
Ətli Blinçik
Ət, düyü
Kəsmikli Blinçik
Kəsmik
Kartoflu Blinçik
Kartof
Acika
Tomat, duz, istiot
Ayran Ev Sayağı
Fanta® 500 ml
Pepsi® 500 ml
Coca-Cola® 500 ml