HTTP 200 is not uptime: the silent failures that break travel data pipelines

The job started.

Requests returned successfully.

No uncaught exception appeared.

Rows reached the database.

Monitoring is green.

And the dataset is wrong.

This is one of the most expensive failure modes in external data collection because the infrastructure reports success while the product quietly deteriorates.

Travel data makes the problem particularly visible.

One market disappears.

A price field moves.

Availability is returned differently.

A source introduces a new page variant.

One airline starts serving a fallback response.

None of those events necessarily produces a server error.

Infrastructure uptime and data uptime are different metrics

Imagine a collector with 99.99% infrastructure availability.

Now imagine that 7% of its records have lost a critical price field for four days.

From the infrastructure perspective, the system is healthy.

From the customer’s perspective, it is not.

This is why a serious external-data pipeline needs several levels of monitoring.

Layer 1: transport

First ask whether the request itself succeeded.

Monitor:

• response codes;
• timeouts;
• network errors;
• block rates;
• retry volume;
• latency;
• proxy or session failures.

This catches hard failures.

It does not tell you whether the returned data is useful.

Layer 2: parser

Next ask whether the response still contains the structures you expect.

Useful signals include:

• required selectors;
• JSON keys;
• schema validation;
• parser error rates;
• payload type;
• expected embedded data blocks.

This catches many source changes.

It still does not guarantee a healthy dataset.

A parser can successfully extract a smaller, incomplete or stale result set.

Layer 3: completeness

Now ask whether you received approximately the data you expected.

Examples:

• total records;
• records per airline;
• records per geography;
• activities per destination;
• percentage of products with price;
• percentage with availability;
• missing-value rates;
• number of expected options per product.

A major drop should be investigated even when every request was technically successful.

This is where a production pipeline starts monitoring business-shaped data rather than infrastructure-shaped data.

Layer 4: distribution

Aggregate volume can hide local failures.

Suppose overall activity volume remains stable because Paris grows by 10%.

Meanwhile Rome loses half its listings.

Total row count looks normal.

The dataset is not.

Monitoring therefore needs to understand meaningful segments:

source · market · destination · carrier · category · product type

depending on the domain.

The right segmentation is product-specific.

The principle is universal: a healthy total can hide an unhealthy slice.

Layer 5: freshness

Correct data can still be unusable when it is old.

Travel marketplaces make this especially obvious.

Viator’s Partner API separates availability schedules from the real-time availability and pricing check used near booking. Its documentation warns that schedule data can rapidly become out of date and recommends using the real-time check to determine bookability in the transaction flow.

See Viator’s technical documentation.

That is a useful reminder:

Freshness is part of correctness.

The acceptable age depends on the field.

A description can be days old and still be fine.

Availability can become wrong in seconds.

Layer 6: cross-source disagreement

When several sources describe the same entity, disagreement itself becomes a signal.

Suppose two aviation sources agree on a route and one suddenly stops returning it.

Did the route disappear?

Is one source delayed?

Did a point-of-sale or market rule change?

Did the collector lose coverage?

Cross-source comparison can reveal degradation before any single source violates a simple threshold.

The goal is not to force every source to agree.

It is to know when disagreement is unusual enough to investigate.

Monitor ratios, not only counts

Absolute counts are useful, but ratios can be even more informative.

Examples:

• percentage of results with price;
• percentage of products with at least one bookable option;
• percentage of successful requests that produce usable records;
• percentage of routes represented in all expected sources;
• percentage of products whose key fields changed;
• percentage of records rejected by validation.

Ratios make it easier to detect slow degradation.

A source that moves from 99% usable output to 96%, then 92%, then 88% may never trigger a simple “job failed” alert.

It is still becoming operationally expensive.

Use baselines that understand seasonality

Travel data has natural cycles.

More activity availability may appear during peak seasons.

Airline schedules change by season and timetable period.

A fixed threshold can therefore produce either noise or blind spots.

Better baselines compare the current shape with relevant history:

• same weekday;
• same market;
• same time of day;
• recent rolling distribution;
• expected seasonal range.

Anomaly detection does not need to be complicated to be useful.

Often a few well-chosen baselines catch more real incidents than a sophisticated system monitoring the wrong variables.

Fail loudly when the data is unsafe

There is a strong temptation to keep delivering partial output because “some data is better than none.”

Sometimes that is true.

Sometimes it is dangerous.

The pipeline should have an explicit policy for degraded data:

• continue with a warning;
• quarantine one source;
• mark records as stale;
• use a fallback source;
• stop downstream delivery;
• escalate for manual review.

The right choice depends on the product.

The important part is that the choice is intentional.

A silent partial dataset should not be the accidental default.

Measure time-to-detection and time-to-recovery

Uptime is useful, but two operational metrics often explain customer experience better:

Time to detection: how long can bad data exist before the system knows?

Time to recovery: once detected, how long until trustworthy data is flowing again?

A pipeline that fails for fifteen minutes, alerts immediately and recovers cleanly may be more reliable in practice than one that quietly delivers incomplete data for two days.

The customer should never be your monitoring system

A data platform has failed twice if the customer is the first person to notice.

First the data degraded.

Then observability failed to catch it.

For production external-data systems, the operating question should therefore be:

How long can bad data exist before someone knows?

Not merely:

Does the server respond?

This is why we describe external collection in terms of operating reliability rather than how quickly a scraper can be written.

The collector is relatively easy to demonstrate.

Reliability is what has to survive production.

For the broader architecture, see Travel data collection at scale: the scraper is the easy part.

Have a pipeline that is technically green but still creates data-quality incidents?

Integrality can benchmark one source against coverage, freshness, completeness and recovery time before discussing a larger rebuild.

Talk to an engineer about the specific source, not a generic transformation project.