There is no published limit. There are three things you can read instead: the site's robots.txt, the rate limit its CDN is running, and the status code it sends back when it has had enough.
There is no published limit, so the honest answer is that you can check a website without getting blocked about as often as that site's defences will tolerate, and almost nobody prints the number. Three things are readable: the site's robots.txt, the rate limit its CDN is running, and the status code it sends back when it has had enough. Every figure below was read from its source on 12 September 2026.
No. RFC 9309, the robots exclusion protocol, defines user-agent, allow and disallow and stops there. Crawl-delay is not in it, whatever a hundred SEO blogs imply.
What the standard does settle is what to do when robots.txt will not load, and the two failure directions are opposite. A server error means the file "is undefined and the crawler MUST assume complete disallow". A 404 is the reverse, and the crawler "MAY access any resources on the server". Cached copies have a shelf life too. Google's documentation gives the same 24 hours the standard does.
Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable.
So the file tells you what you may fetch. It does not tell you how often, and where it tries to, most of the web is not listening.
That depends entirely on who is reading it, which is why quoting a crawl-delay as "the site's limit" is wrong more often than it is right.
| Reader | Honours it? | What the source says |
|---|---|---|
| RFC 9309, the standard | Not defined | Only user-agent, allow and disallow are defined fields |
| No | "other fields such as `crawl-delay` aren't supported" | |
| Bing (msnbot) | Yes | Throttling relative to its default rate: 1 slow, 5 very slow, 10 extremely slow, and Bing advises "the lowest value possible" |
Waterstones is a good worked example, because its file says what it means. Under `user-agent: Bingbot` it sets `Crawl-delay: 0.2`, and the comment directly above reads "Crawl Delay - 5 URL max per second". One shop has told one crawler, in public, that five requests a second is acceptable. Google is given no number at all, because Google would ignore one. None of that is a promise to you, but it beats guessing, and reading it costs ten seconds. We have a separate page on how Waterstones handles signed editions, which is the sort of listing that makes people want a short interval in the first place.
The mildest version is a polite refusal: 429 Too Many Requests, usually with a Retry-After header saying how long to wait. MDN is clear about who gets counted. "Typically, rate-limiting restrictions are based on a client's IP but can be specific to users or authorized applications if requests are authenticated or contain a cookie." Share an address with an office, a VPN or somebody else's monitor and you share the budget.
Above that sits the bot wall: a challenge page, a captcha, an interstitial that never resolves. That is a different mechanism from a rate limit, and why a website says verify you are human covers it. Cloudflare's own documentation shows how little a site needs to set one up. The free plan allows a single rate limiting rule, matched on path, counted by IP, over a 10-second window with a 10-second timeout. A burst trips a window that short. A steady trickle does not.
The version that actually costs people money is the site that keeps serving you a page with the answer taken out. Distill's guide to watching US visa appointments recommends checks "no less than 4hrs" apart and says the site "is known to restrict access for users who check too frequently". The page still loads. The slots are simply not in it, and a monitor in that state reports success forever without ever firing.
Read robots.txt before anything else
It sits at /robots.txt on every site and occasionally hands you a number, the way Waterstones does. It also tells you which paths are off limits, which is the part that is actually standardised.
Ask for the smallest thing that answers the question
A shop's product JSON is a fraction of the page built around it, and the server does far less work to produce it. Fetching a whole rendered page to read one stock field is the expensive way to ask.
Use conditional requests where the server supports them
Send If-None-Match with the ETag you already hold. If nothing has changed you get a 304 Not Modified with an empty body, so you still learn the answer and the transfer costs almost nothing.
Read the response, not only the content
A 429 with a Retry-After header is an instruction. Back off for exactly that long instead of retrying harder, which is what turns a temporary limit into a lasting one.
Spread short intervals across addresses
Limits are counted per IP, so ten addresses checking every 50 seconds look nothing like one address checking every 5. That is the entire argument for a proxy group.
Match the interval to how fast the page moves
A Pokemon drop is decided inside a minute. A council planning page is not. The same number for both is wrong twice.
It depends on what the page does, and the only certainly wrong answer is one number for everything. A price that moves weekly gains nothing from a 5-second check, and a 4-hour check on a restock is decoration.
The rough shape: seconds for drops and restocks where the stock lasts minutes, a minute or two for prices and sale pages, an hour or more for terms, policy pages and job listings. Our own intervals and the plan limits behind them are on the polling intervals help page, and why restock alerts arrive too late covers what a slow interval costs you in practice.
For scale, a monitor on a 5-second interval makes 12 requests a minute against one URL. One person refreshing three tabs during a drop will beat that without trying.
It is not part of the standard and Google ignores it: "other fields such as `crawl-delay` aren't supported". Bing does read it. Treat it as the site saying what it would prefer, which is worth knowing even when nothing enforces it.
Usually not. It means you went over a limit inside a window, and most limits reset on their own. Look for a Retry-After header, wait that long, and come back slower. A ban looks different: a 403 that never clears, or a challenge page that never resolves.
They split the count, which is not the same as removing it. MDN says rate limits are usually applied per IP, so ten addresses checking once a minute each read as ten visitors on a one-minute interval. That helps with a rate limit. It does nothing about a bot wall that has already decided what your requests look like.
Read the status code and the size of the response rather than checking that something came back. A block is often an HTTP 200 carrying a challenge page, and a filtered page is a 200 carrying real HTML with the part you wanted missing.
robots.txt is a request rather than a law, and the site's own terms are the document that decides. The practical test is whether you would be comfortable telling the shop what you are doing. One product page every few seconds is nothing to a shop that survives Black Friday. Crawling an entire catalogue at that rate is a different conversation.
Watch a page and pick your own interval Free monitors check every 60 seconds. Standard is 15 seconds, Power is 5.