Modern aviation data pipelines: APIs, NDC and the gaps between them

“Why collect airline data externally when aviation APIs already exist?”

It is a reasonable question.

And the wrong answer would be:

APIs are not good enough, so scrape everything.

Modern aviation already has excellent specialist data infrastructure.

OAG currently publishes forward-looking and historical schedules for 900+ airlines and says its schedules are updated every 15 minutes. Products like that are the right primary source for many schedule and operational requirements.

The interesting engineering problem begins when a travel product needs something beyond one standard feed.

Aviation products increasingly combine several data layers

Depending on the use case, a product may need:

• schedules;
• operational flight status;
• routes and connections;
• fares;
• airline retail offers;
• ancillary information;
• market-specific content;
• partner data;
• historical observations;
• independent validation;
• information exposed only through a particular source.

A single product can therefore consume several types of aviation data without any one source being “wrong.”

The relevant question is:

Which source is authoritative for which fact?

NDC is about richer airline retailing

IATA defines New Distribution Capability as an industry-supported data exchange standard that enables airlines to create and distribute richer offers across distribution channels.

Its current NDC overview emphasizes richer air content, product differentiation and a more transparent shopping experience.

But NDC is now part of a broader transition.

IATA’s Modern Airline Retailing program describes the industry’s movement toward 100% Offers and Orders, supported by NDC, ONE Order and related standards.

That transition makes airline retailing richer.

It does not automatically make downstream data architecture simpler.

Different airlines, aggregators, sellers and IT providers move at different speeds. A travel product still has to reconcile the sources it consumes.

External collection should be the gap layer, not a reflex

There are several situations where an external source can still be useful around a strong licensed or API foundation.

Coverage gaps

The product needs a field or observation that is not available through its primary provider.

Independent validation

The product needs to compare what appears on a public sales surface with what an upstream feed reports.

Point-of-sale or market-specific behaviour

The same airline content can behave differently by geography, language, currency or point of sale.

Long-tail sources

A niche airline, route, ancillary or regional source may not justify another large commercial integration.

Competitive or market intelligence

The requirement is observation rather than transaction processing.

These are very different use cases from attempting to recreate a complete commercial aviation dataset by scraping airline websites.

The distinction matters.

Use the best source for the job

A modern architecture might combine:

commercial aviation data provider

for broad standardized schedule or status coverage;

NDC / airline API

for richer airline retail offers and direct distribution capabilities;

partner or supplier feed

for contractual product or inventory data;

targeted external source

for a gap, verification or observation requirement.

The architecture should make those roles explicit.

If two sources disagree, the system needs rules about authority, freshness and provenance rather than an arbitrary “last write wins.”

Normalize before the product sees the sources

Ten aviation sources should not produce ten incompatible schemas inside the application.

A data layer should map them into a canonical model.

For example:

source_airline_name → airline

departure_airport_code → origin

arrival_airport_code → destination

market_currency → currency

observed_fare → price

source_timestamp → observed_at

with provenance retained so the system still knows where each value came from.

The normalization layer then owns:

• mapping;
• validation;
• deduplication;
• timestamps;
• source authority;
• provenance;
• source-specific exceptions.

That is where a pile of feeds becomes a data product.

Isolate source adapters from the product

A resilient architecture should make source changes local.

If airline A changes its flow:

adapter A changes.

The normalized model remains stable.

The search experience, analytics layer or AI agent consuming the data should not need a redesign.

This becomes more important while airline retailing itself continues to evolve.

IATA’s current Modern Airline Retailing material describes an ongoing industry transition toward Offers and Orders and modular, interoperable capabilities across the value chain.

Your product should expect upstream interfaces to change over time.

The best defense is architectural isolation.

Provenance should be first-class data

When several sources describe the same flight, fare or offer, the system needs to know:

• which source produced a value;
• when it was observed;
• whether it was transformed;
• how confident the mapping is;
• which source is authoritative for the field;
• whether a fallback source was used.

Without provenance, multi-source data becomes hard to debug.

With provenance, disagreement becomes observable.

That matters for both operational reliability and product trust.

Validation should happen before normalization hides the problem

Normalization makes data easier to consume, but it can also hide source errors if applied carelessly.

A source-specific adapter should validate enough of the raw response to detect:

• unexpected schema changes;
• missing required fields;
• implausible values;
• large coverage drops;
• market-specific anomalies.

Only then should the data be mapped into the canonical model.

Otherwise, a clean normalized object can give false confidence about a broken upstream source.

For more on this, see HTTP 200 is not uptime.

Where Integrality fits

Integrality operates a production aviation pipeline processing roughly 600K records per month under SLA across multiple independent sources.

Our position is not:

Replace every commercial aviation API.

It is:

Build the collection, normalization, monitoring and integration layer around the sources your product actually needs.

Sometimes the correct answer is a commercial API.

Sometimes it is a direct airline or NDC integration.

Sometimes it is a targeted external collector.

Frequently it is a combination.

The broader engineering pattern is covered in Travel data collection at scale.

Already have strong primary feeds but still carry awkward airline-specific integrations?

That is usually a better conversation for us than “we need flight data.”

Show us the gap: the carrier, market, point-of-sale difference or source your core team would rather not spend the next quarter maintaining.