Is it possible to identify the client / library which sent a HTTP request?

I am trying to fetch some data via an API and it is possible to query the API via cURL and python, but when I try to use node (doesn't matter which library, axios requests, unirest, native, ...) or wget I get a proprietary error back from the backend.

Now I am wondering, if the backend is able to identify, which library I am using?

More information:

• The requests are exactly the same, so no way to distinguish them
• The user-agent header field is set and overwritten for all requests
• I already tried to monitor the traffic in wireshark, but couldn't find any differences with the packets on HTTP layer (only the order of the header fields is different, that according to the standard this shouldn't make a difference)

CodePudding user response：

Nodejs uses google V8 JS engine, V8 based http request clients will not allow you to override headers that would compromise 'web safety', so for example if you are setting "Origin, Host, Referrer" headers, node might refuse to do so. I had the same issue previously.

Un-opinionated http clients, such as the ones written in C (curl) and python won't 'web safety' check your requests, so that is what is causing the difference in behavior.

In my case I used a C library that I called from javascript to make my 'unsafe' requests and the problem was solved.

CodePudding user response：

It turns out that the problem was TLS fingerprinting.

See: https://httptoolkit.tech/blog/tls-fingerprinting-node-js/