Selenium gets response code of 429 but firefox private mode does not-CodePudding

The

Firefox with persistent log:

CodePudding user response：

This is just my huch after working with selenium and webdriver for a while; I suspect that it is due to the default user agent of selenium being set to something lame by default and that the server side recognizes this and provides you with a silly HTTP code and a blank page as a result.

Try setting the user agent to something reasonable and/or disable selenium's interfering with defaults.

Another tips is to look at the request using wireshark or similar to see exactly what is sent over the wire.

CodePudding user response：

429 Too Many Requests

The HTTP 429 Too Many Requests response status code indicates the user has sent too many requests within a short period of time. The 429 status code is intended for use with rate-limiting schemes.

Root Cause

When your server detects that a user agent is trying to access a specific page too often in a short period of time, it triggers a rate-limiting feature. The most common example of this is when a user (or an attacker) repeatedly tries to log into a web application.

The server can also identify a bot with cookies, rather than by their login credentials. Requests may also be counted on a per-request basis, across your server, or across several servers. So there are a variety of situations that can result in you seeing an error like one of these:

429 Too Many Requests

429 Error

HTTP 429

Error 429 (Too Many Requests)

This usecase

This usecase seems to be a classical case of Selenium driven GeckoDriver initiated firefox Browsing Context getting detected as a bot due to the fact:

Selenium identifies itself

References

You can find a couple of relevant detailed discussions in:

How to Conceal WebDriver in Geckodriver from BotD in Java?

How can I make a Selenium script undetectable using GeckoDriver and Firefox through Python?