How would tracking pixels implement rate limiting?-CodePudding

How would a company like Facebook or Snapchat implement rate limiting on their tracking pixel? I imagine that they must have rate limiting on the pixel route since it's a public API that anyone can call and could lead to malicious attacks. However, how would you implement a system to rate limit since you can't have a generic limit for all advertisers, mom and pop shops may get 100 pixel requests a day, whereas Nike would get 100M pixel requests a day for example.

If you try to have a rate limit for each advertiser Id, then another situation is that you have dozens of new advertisers joining the platform every day and how would you assign a rate limit to their requests without knowing the type of traffic they would receive from them (for example Nike signing up vs mom and pop shop would need different rate limits). Would this lead to manual work every time you need to assign one or is there some way of dynamically assign a rate limit?

Also this type of scenario is different than a regular API since you don't want to ever lose requests from these advertisers (because that lost request could be a potential attribution). I guess you want to receive as much as possible but also want to be able to control malicious requests from overloading your API.

I've been thinking about this for a bit and can't seem to come up with a way to design a rate limiter for this usage.

CodePudding user response：

Facebook does not have a rate-limiting on the pixels.

All API requests are subject to rate limits. Graph API and Instagram Basic Display API requests are subject to Platform Rate Limits, while Marketing API and Instagram Graph API requests are subject to Business Use Case (BUC) Rate Limits.

You can check the following links to get a better explanation:

CodePudding user response：

Most common approach to rate limiting is token buckets (or leaking bucket). Many (most?) major services use those. The reason for rate limiting may be because of business (when a client X has Y requests allowed per W of time) or just general protection for the system - as no system is truly unlimited.

For example, lets say a service has a dependency and that dependency may support up to X calls per second. Then the system would add a token bucket in front of it (incl. a distributed version) and when there are no more resources, the system would cheaply throttle, instead of having an overload related outage.

More interesting example comes when limit is based on some business reason - like we give each client a specific limit in a multi-tenant system. In this particular case, every client would have their own token bucket and one more bucket to protect system as a whole (as described in previous paragraph). Basically, for a client request to be executed, two buckets should be non empty.

Even more interesting (and more practical) example is shaping traffic like this: we give each client X tps, but we allow overflow - e.g. we allow X Y tps for some limited window, e.g. 10 seconds.

To implement that approach, the system would use four token buckets:

One is per client bucket to handle X tps per client.
If a particular client has first bucket empty, the second will kick in, that one will allow up to Y*(window size) requests to come through, and this bucket will have very slow refill factor
To make sure not all tickets from bucket two can be executed immediately - a third per-client bucket is managed; it also has Y limit, and different refill rate.
And the last bucket works as usual - protects from whatever true limits on the system are.

So for a request to come through, one of two conditions should be true:

Bucket one and four should have at least one token each
Bucket two, three and four should have at least one token each