"Cannot allocate memory" requests

I have a script similar to the one below. It is located on a server with 0.5 GB of RAM. After about a day of stable operation, this error appears: [Errno 12] Cannot allocate memory. I believe that I am closing everything unnecessary, but the problem remains.

from requests_html import HTMLSession
from bs4 import BeautifulSoup as BS
def get_site(session, site, params):
    try:
       r = session.get(site_ad, headers=headers, verify=False)
       script = """
                () =>{
                     $('.WEB_APPOINT_FORWARDBUTTON').click();
                }
                """
       r.html.render(script=script, sleep=5 number_of_attempts*2, keep_page=True)
       soup = BS(r.html.html, 'html.parser')
       r.close()
       info = soup.find("div", {"id": "location"}).find_all("td", {"class": "calendar"})
       return info
   except:
       return None
headers = {'accept': '*/*', 'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36'}
ses = HTMLSession()
url = "my_url"
while True:
   params = "same info"
   rez = get_site(ses, url, params)
   time.sleep(30)
ses.close()

CodePudding user response：

Well, evidently you're leaking memory and 512 megs is not much at all, and things crash. Possibly recycling the session every now and then could help, but it's impossible to tell without a profiler.

Since this looks like a program that just does a thing on a schedule over and over, it should probably be safe to just restart it if/when it crashes, so set up your server to do so.

If, for instance, you're running your script using a Systemd unit, you can just add Restart=always.

For a more rudimentary approach, you could just do it in the shell with something like

while true; do python myscript.py; sleep 30; done

CodePudding user response：

I don't see an obvious resource hog but infinite loops make me uncomfortable.

Maybe try scheduling a cron job every minute and skip the loop so your script can properly terminate and free the allocated memory?