Travel data collection at scale: the scraper is the easy part

Anyone can build the first version of a scraper.

Point a script at a page. Find the fields. Parse the response. Save the result.

For a small dataset, that can be enough. At scale, it is usually just the beginning.

The difficult part starts when the source changes its HTML, moves data behind another request, introduces regional behaviour, modifies anti-bot protection, starts returning partial responses, or quietly stops exposing one field.

The collector may still be running. The database may still be receiving rows. And the product depending on that data may already be broken.

That distinction matters particularly in travel.

Airlines, hotels, tours and activities are dynamic products. Prices change. Availability disappears. Schedules move. New products appear. Content is updated. Different markets can expose different results.

So the real engineering question is not:

Can we extract the data?

It is:

Can the data product remain complete and trustworthy when the systems underneath it keep changing?

A scraper and a production data pipeline solve different problems

A scraper answers a narrow question: can data be extracted from this source?

A production pipeline has to answer many more:

• Did we receive the amount of data we expected?
• Did one destination, airline or category disappear?
• Are critical fields still populated?
• Did the source change its structure?
• Is the data fresh enough for the product using it?
• How quickly will degradation be detected?
• How quickly can the source recover?
• Can one failing source be isolated without breaking the entire product?

This is why we think of external data collection as data infrastructure, not a collection of scripts.

At Integrality, one production aviation workflow processes roughly 600K records per month under SLA across multiple independent sources. Volume is only part of the story. The difficult work is keeping that pipeline alive as upstream behaviour changes.

If you want the operational version of this argument, see Why scrapers die in month three.

Travel data can fail without throwing an error

The easiest failure is obvious:

500 Internal Server Error

Much more dangerous is:

200 OK

followed by incomplete or distorted data.

A source may normally return thousands of products for a market and suddenly return hundreds. Prices may still exist, but the field used to distinguish adult and child rates may disappear. Results for several destinations may vanish while aggregate volume remains almost unchanged.

Nothing necessarily crashes.

A production system therefore needs to monitor the shape of the data, not only the health of the infrastructure.

That means baselines for record volume, field completeness, important segments, freshness and source-specific distributions. It also means treating a technically successful request as only the first step in determining whether the pipeline is healthy.

We go deeper on that in HTTP 200 is not uptime.

Anti-bot handling belongs inside the architecture

When a source actively resists automated access, proxy rotation, session behaviour, browser execution, retry strategy, rate controls and source-specific limits are not optional extras.

They affect whether the dataset exists at all.

More importantly, different sources behave differently. Trying to force twenty travel sites through one universal collection configuration often looks elegant in a diagram and becomes fragile in production.

The more resilient pattern is to isolate source-specific collection logic from the normalized downstream model.

If one source changes, its adapter changes.

The rest of the product should not need to know.

Validation belongs at ingest

A reliable pipeline should reject or quarantine obviously implausible data as early as possible.

Examples include:

• a price that suddenly becomes zero across a market;
• a date that falls outside the expected travel window;
• an availability response with no bookable options;
• a carrier or destination disappearing from a source that normally covers it;
• a schema field changing type;
• a large jump in null values for a commercially important field.

The further bad data travels downstream, the more expensive it becomes to diagnose.

A pipeline that catches a malformed record at ingest creates one local incident.

A pipeline that lets it contaminate caches, analytics and customer-facing recommendations creates several.

The output should be a data product

A customer generally does not need a scraper.

They need complete records, predictable schemas, acceptable latency, freshness and reliability.

That changes what should be measured.

Useful operational metrics include:

coverage · completeness · freshness · latency · recovery time · error rate · uptime

Those metrics sit much closer to the business value of the system than “number of collectors written.”

The collector itself can be rewritten.

The reliable dataset is the product.

Where APIs fit

None of this means web collection should replace official APIs.

Quite the opposite.

The strongest architecture normally uses the best authoritative or licensed source available for each requirement, then fills specific gaps where necessary.

For example, OAG currently publishes global schedule data for 900+ airlines and says its schedules are updated every 15 minutes. Products like that are the right foundation for many aviation requirements.

Likewise, experience marketplaces expose structured APIs for product and availability data. Viator’s Partner API separates product-content ingestion from availability schedules and provides real-time availability and pricing checks near booking.

External collection becomes valuable when a product needs data that is unavailable from the primary feed, needs independent verification, depends on a particular market or source, or needs information exposed only through another surface.

The engineering job is not to choose a religion called API or scraping.

It is to construct the most reliable data layer for the actual product.

A practical architecture

A production travel-data pipeline often looks closer to this:

Travel sources → source adapters → access/anti-bot layer → validation → normalization → canonical store/API → monitoring

Each layer answers a different failure mode.

The adapter knows how a source behaves.

The access layer keeps requests viable.

Validation determines whether the returned data is plausible.

Normalization makes several sources usable through one schema.

Monitoring detects both hard failures and silent drift.

That separation is what allows the system to evolve without making every source change a product-wide incident.

The question to ask before outsourcing travel data collection

Do not ask only:

Can you scrape this source?

Ask:

• What happens when the source changes?
• How is partial collection detected?
• What is monitored besides HTTP status?
• How is source-specific behaviour isolated?
• What does recovery look like?
• What data-quality metrics are part of the SLA?
• How does the normalized output remain stable while source schemas change?

Those questions tell you whether you are buying a script or an operating data product.

Building a travel product that depends on data you do not control?

Integrality builds and operates external data pipelines for travel, marketplaces and other high-change sources.

If one source, market or dataset is disproportionately expensive to keep alive, show us the gap. We can usually determine quickly whether the problem is collection, normalization, observability — or something that should not be custom-built at all.