Why scrapers die in month three

There is a rhythm to scraping projects that anyone who has run several will recognise.

Week one, someone writes a scraper. It works. Everyone is pleased, and slightly surprised at how easy it was. Week four, it is quietly feeding a dashboard and nobody thinks about it. Week ten, a report looks wrong. Someone checks. The scraper has been returning empty results for nine days and nothing anywhere raised a hand.

This is not a story about bad engineering. The scraper was fine. The problem is that a scraper and a pipeline are different objects, and most projects budget for the first while needing the second.

What actually changes

Nothing dramatic happens in month three. That is the point — the failures are mundane and cumulative.

A site rolls out a redesign, and the selector that found the price now finds nothing. A CDN turns on bot protection that was previously in monitoring mode. A block of proxy addresses gets flagged somewhere else entirely and inherits a reputation you had no part in creating. Someone adds a cookie consent interstitial that only appears for visitors from certain countries.

None of these are attacks on you specifically. They are ordinary infrastructure decisions made by people who are not thinking about you at all. And each one has the same effect: the collector keeps running, returns a valid HTTP 200, and delivers nothing useful.

The dangerous failure mode is not the one that throws an exception. It is the one that returns an empty array and exits zero.

The failures that don’t announce themselves

A pipeline that crashes is a good pipeline. Something is on fire, alerts go off, someone fixes it within the hour.

The expensive failures are the quiet ones:

• Partial collection. The scraper gets 60% of the catalogue and reports success. Nobody notices until someone asks why a product category looks thin.
• Stale-but-valid data. A cache layer starts serving yesterday’s page. The data parses correctly, the schema validates, the numbers are simply wrong.
• Silent geographic drift. Requests start being routed through a different region and the site serves different prices. Everything works, everything is wrong.
• Slow degradation. Success rate drops from 99% to 94% to 88% over six weeks. No single day looks like a problem.

Each of these passes every check that a naive monitoring setup performs. HTTP status is fine. The job completed. The rows landed in the table.

What a pipeline has that a scraper doesn’t

The difference is not sophistication. It is that a pipeline is built assuming it will be attacked by entropy, and instrumented accordingly.

Volume baselines. You know that this source produces roughly 40,000 records a day. When it produces 24,000, that is an alert — even though nothing technically failed.

Structural fingerprinting. You store a hash of the page structure, not just the extracted values. When the layout changes, you know before the data goes wrong, not after.

Per-source rotation profiles. Different sites tolerate different request patterns. A global rotation policy is a compromise that is wrong for every source simultaneously. Rotation tuned per source costs more to set up and stops costing anything afterwards.

Failure that is loud. When something breaks, the pipeline stops and says so. It does not deliver a smaller dataset and hope nobody checks.

Validation at ingest. A price of zero, a date in 1970, a string where a number should be — caught at the boundary, not three transformations downstream where it has already contaminated an aggregate.

The economics nobody plans for

Here is the split we see across pipelines we have run for a year or more:

Phase	Share of total effort
Writing the collectors	~15%
Keeping them alive	~55%
Validation and monitoring	~30%

The first column is what gets estimated. The second is what gets discovered.

This is why the honest question to ask a vendor is not “can you scrape this site”. Almost anyone can. The question is what happens on the Sunday in month seven when the source changes its defences — and whether you find out from an alert or from a client.

What we do about it

We run pipelines at 99.9% uptime on sources we do not control. That number is not a claim about clever parsing. It is the output of treating collection as an operational discipline rather than a delivery milestone: baselines, structural monitoring, per-source rotation, alerting that fires before the data degrades rather than after.

Anyone can write a scraper. Keeping it running while the target changes its defences every quarter is a different job, and it is the one that decides whether the data still arrives in month twenty-four.