It is not a test you passed or failed. It is a score, and the wall appears when your request lands on the wrong side of a threshold somebody chose.
A website says "verify you are human" when the security service in front of it has scored your request as possibly automated and decided to check before letting it through. It is not a judgement about you, and there is usually nothing wrong with your browser. Cloudflare, which runs the most familiar version of this, describes the score as a number from 1 to 99 saying how likely the request came from a bot, and the site owner picks the threshold at which a visitor gets challenged.
Between you and the website sits a security service. It looks at the request before the site ever sees it, gives it a score, and compares that score to a rule the site owner wrote. Cloudflare publishes how the scoring works, and it uses several engines at once.
| Engine | What it looks at |
|---|---|
| Heuristics | "Pattern matching against a database of known malicious fingerprints" |
| Machine learning | "Analyzing request features across billions of daily requests", producing most scores between 2 and 99 |
| JavaScript detections | "Catches headless browsers (browsers controlled by software, with no visible window or human operator) and other automation tools" |
The score then feeds a rule. Cloudflare's documentation is plain about who decides: you can use bot scores in firewall rules to "block, challenge, or allow requests based on their score". So two people on the same broadband line can get different treatment on two different sites, because the sites chose different thresholds. Nothing about your machine changed in between.
Most of the time you will not see a puzzle. Cloudflare's challenge page runs JavaScript in the background, and its own docs say the visitor "must wait until their browser finishes processing the JavaScript, which typically takes less than five seconds". The older wording for this was "checking your browser before accessing", which is the same thing with a different label.
Because passing it stores a cookie, and anything that clears or hides that cookie sends you back to the start. Cloudflare names it directly: a successful check "issues a persistent clearance cookie (`cf_clearance`)".
That single detail explains most of the repeat offenders. A private window starts with no cookies, so every visit is a first visit. A VPN or a mobile network that moves you between addresses can invalidate it. Browser settings or extensions that block cookies stop it being stored at all. And a cookie set on one site does nothing for the next one, because each site runs its own rules.
If you are seeing it on every site and every visit, the common factor is nearly always cookie storage or a shared address, rather than anything about the sites themselves.
This is the part that surprises people, and Cloudflare states it outright. A challenge works by sending back a whole HTML page for a browser to render and solve. Its documentation says: "This mechanism fails when the browser expects a non-HTML response, such as an AJAX or XHR (fetch) request."
Challenge Pages interrupt the request flow by returning a full HTML page for the user's browser to render and solve. This mechanism fails when the browser expects a non-HTML response, such as an AJAX or XHR (fetch) request.
So a script asking for a product feed does not get a puzzle it could theoretically solve. It gets a page of HTML where it expected data, and no way through. Worse, that response often arrives with a perfectly normal status code, so a naive tool records it as a success and quietly reports that the shop has no products. Anyone building this sort of thing learns to check what came back rather than what the status said.
One of the three detection engines above makes this harder still. JavaScript detections exist specifically to catch "headless browsers", which is the standard way an automated tool renders a page. The thing you would reach for to solve the challenge is the thing being looked for.
No, and this is why a monitor can fail without anything appearing on screen. DataDome, another common service, runs what it calls a device check that the visitor never sees. Its documentation says it "runs on the end user's device, without the need for any user interaction", and that it triggers when "a bot is detected by DataDome, but the evidence of bot activity is not strong enough to block it".
It inspects screen details, environment and hardware information, and runs JavaScript challenges including canvas rendering and timing tests. DataDome also states that "no personal information is collected by Device Check. Only technical details are inspected."
The practical consequence is that a page can be refused in several different ways: an obvious challenge, a silent check that fails, an empty result that looks like an out-of-stock shop, or a plain refusal to answer. Only the first of those looks like a block.
A 429 is a different animal and much easier to read. MDN describes it as meaning "the client has sent too many requests in a given amount of time", and notes that a `Retry-After` header "may be included to this response to indicate how long a client should wait before making the request again". It also says restrictions are typically based on the client's address. That is a rate problem, not an identity problem, and the fix is to slow down and honour the header.
Robots.txt is worth reading before you point anything at a site, but expect less from it than people assume. RFC 9309 standardises only user-agent, allow and disallow. `crawl-delay` is not in the standard, and Google's own documentation confirms it "supports the following fields (other fields such as crawl-delay aren't supported): user-agent, allow, disallow, sitemap". So a crawl-delay line is a request that many crawlers simply ignore.
Two rules in the RFC are worth knowing anyway. Crawlers "SHOULD NOT use the cached version for more than 24 hours", which is also roughly how long Google caches it. And the two failure directions are opposite: on a 4xx "the crawler MAY access any resources on the server", while on a 5xx "the crawler MUST assume complete disallow". A site with a broken robots.txt is asking to be left alone entirely.
Check whether the site publishes the data another way
A feed, a public API, or a sitemap is meant to be read and will not fight you. This is faster and more reliable than reading the page, and it is the first thing worth looking for.
Read robots.txt and respect what it disallows
It only carries user-agent, allow and disallow reliably, but those are the site's stated wishes. If it disallows the path, that is an answer.
Slow down before you do anything else
Most blocks that appear after a while are rate related. A 429 with a Retry-After header is telling you exactly what to do, and checking every few minutes rather than every few seconds fixes more problems than any clever workaround.
Check what actually came back, not the status code
A challenge can arrive as a normal-looking response containing HTML where you expected data. Treat an unexpected shape as a failure rather than as an empty result, or you will record a healthy shop as sold out.
Accept when the answer is no
Some sites do not want automated readers, and that is theirs to decide. If a site consistently refuses, the honest options are its own alerts, its newsletter, or checking by hand.
If you are using StockWatch and a monitor has gone quiet or started erroring, the same logic applies in a more specific form: our troubleshooting notes cover what a block looks like from inside the app, check intervals covers going slower, and proxy groups covers the case where the site is refusing one address rather than everybody. The wider picture of what page monitoring can and cannot do is in how to monitor a website for changes.
Because a security service scored the request as possibly automated and the site owner set a rule to challenge requests at that score. Cloudflare scores requests from 1 to 99 for how likely they came from a bot, and the owner chooses whether to block, challenge or allow at a given score.
Passing stores a clearance cookie, which Cloudflare calls cf_clearance. A private window, cleared cookies, a cookie-blocking extension or a changing IP address all mean the cookie is missing next time, so the check runs again.
No. It is a threshold, not an accusation. A shared or mobile network address, a VPN, an unusual browser configuration or simply a site with strict settings will all trigger it.
Because the challenge is an HTML page meant to be rendered. Cloudflare's own documentation says the mechanism fails when the request expects a non-HTML response such as a fetch call, so a script asking for data gets a wall rather than a solvable puzzle.
No. DataDome's device check runs in the page without any user interaction, so a request can be refused with nothing on screen. A silent failure often looks like an empty page or a shop with no products rather than an error.
Not reliably. Crawl-delay is not part of RFC 9309, and Google states it does not support the field. Only user-agent, allow and disallow are standard, so slowing down has to be your own choice rather than something the file enforces.
Watch a page without building any of this Five pages free, checked every 60 seconds, alerts to Discord, email or Telegram.