A dashboard is only as current as the data reaching it. An AI assistant can confidently recommend a product that sold out minutes ago if its information pipeline has stopped updating. Data pipelines move, transform and coordinate information between systems. Their most valuable work is often invisible: handling changes, recovering from failures and delivering dependable data without a team constantly repairing the connections.
The investment question
The investment question is which parts of integration remain worth paying for as cloud platforms automate more data movement. Simple transfers face commoditisation. Maintaining reliable access to many changing systems, preserving meaning and meeting production service levels can still be demanding and valuable.
Our view is that the strongest suppliers sell dependable outcomes across heterogeneous environments. Connector counts are less persuasive than coverage of difficult sources, recovery behaviour, data quality and operational support. AI increases the need for fresh context, but it also gives established platforms another reason to bundle ingestion and transformation into broader contracts.
How pipelines work
Extraction obtains data from applications, databases, files or event streams. Loading places it in a destination. Transformation changes structure, combines records or applies business logic. ETL transforms before loading into the target analytical store; ELT loads first and uses destination compute for transformation. Actual architectures may combine both approaches.
Change data capture, or CDC, identifies changes rather than repeatedly copying an entire database. Debezium’s architecture documentation describes connectors that capture database changes and publish events. Log-based capture can reduce repeated full scans, but permissions, initial snapshots, schema changes and recovery still require careful management.
Streaming handles continuing events, while batch processing works on bounded groups of records. Orchestration coordinates dependencies, retries and schedules. Apache Airflow is a workflow orchestration platform, not itself a general substitute for a continuously running streaming engine. Keeping these roles distinct helps explain why companies may need several components even within an integrated service.
Market structure and competitive advantage
| Pipeline function | Customer requirement | Defensible capability |
|---|---|---|
| Application and database ingestion | Reliable access to changing sources | Connector maintenance and difficult source coverage |
| Transformation | Consistent business logic and tested models | Reusable development practices and dependencies |
| Streaming and CDC | Timely, recoverable propagation of changes | Throughput, resilience and operational tooling |
| Orchestration and observability | Coordinated execution and rapid incident resolution | Clear lineage, diagnostics and ownership |
Commercial products compete with open-source tools, internal engineering and native cloud services. Fivetran and dbt now sit within a combined business following their merger, while Confluent is part of IBM. Hyperscalers and data platforms provide overlapping ingestion, streaming and transformation services, often close to the destination where data will be processed.
A long-lived integration can create switching costs through mappings, schedules, permissions and operational procedures. The supplier must continually maintain that integration as upstream APIs, schemas and access rules change. This maintenance burden can create a moat, but it also creates substantial support and engineering costs that weaken the economics of poorly priced connectors.
Economics: moving less can be worth more
Pricing may depend on changed rows, data volume, compute time, connector count or reserved capacity. These measures do not always track business value. A frequently updated source can generate a large bill even if only a small subset of its information affects decisions. Customers need to understand exclusions, repeated updates and historical reload behaviour.
For illustration, imagine a pipeline transferring 100 gigabytes daily when only five gigabytes of records change. Incremental processing could sharply reduce movement, but savings depend on capture, transformation and destination costs. If a supplier charges by transferred volume, improved efficiency may reduce revenue while making the service more attractive. This example is conceptual, not a pricing estimate.
The total cost includes the engineers who monitor failures, reconcile missing records and respond when a source changes. A more expensive managed connector may be economical if it removes recurring operational work. Conversely, a low-complexity internal integration can be sensible when its scope is narrow and stable.
Freshness has a price. Seconds-level updates can require continuously running resources, stronger monitoring and more complex state management. A monthly reporting process may gain little from that expense. The economically rational pipeline meets the decision’s freshness requirement with appropriate reliability, rather than pursuing real time for every dataset.
AI and hyperscalers: context must keep changing
AI pipelines extend beyond moving tabular rows. Documents may need parsing, deduplication, chunking, embedding and indexing. Changes in access permissions or source deletion must propagate to derived representations. Otherwise, an assistant can continue retrieving information that has become obsolete or inaccessible in the original system.
Agents also create events and operational state. A reliable architecture must distinguish observing a change from safely acting on it. Delivery guarantees within one streaming component do not automatically guarantee that an external action, such as sending an order, occurs exactly once. Idempotency and coordination with destination systems remain essential design concerns.
Hyperscalers can integrate source databases, streams, storage and model services within one operational environment. Independent platforms have a stronger case when data spans clouds, enterprise applications and legacy systems. Their value is greatest where they reduce the complexity that no single destination provider fully controls.
Current market debates — September 2026
Consolidation is reshaping the sector. Fivetran announced the completion of its merger with dbt Labs on 1 June 2026. IBM completed its acquisition of Confluent on 17 March 2026. Both combinations position integration as part of a broader data and AI proposition. The strategic rationale is credible; actual product integration and customer outcomes require separate evidence.
The second debate concerns zero-ETL. AWS’s Aurora zero-ETL documentation describes managed integrations that reduce the need to build selected movement pipelines. This can remove genuine work. It does not eliminate source modelling, data quality, permissions, destination costs or every transformation required by a business. Supported combinations and service limitations matter more than the label.
The third debate is whether AI-generated pipeline code weakens commercial tools. Coding assistance can reduce development time, especially for straightforward transformations. Production value still depends on tests, monitoring, schema handling, recovery and ownership. The likely competitive pressure is strongest on undifferentiated authoring features and weaker where vendors assume difficult ongoing operational responsibilities.
Structural debates: the value of a neutral integration layer
As destinations absorb common ingestion functions, independent suppliers must justify an additional platform. Neutrality can matter when customers want consistent movement and transformations across competing environments. Yet neutrality is useful only if the integration works reliably and does not introduce excessive cost or another fragmented governance system.
Streaming also raises a structural trade-off between freshness and complexity. A business event can arrive late, out of order or more than once. Correctness requires explicit rules for time, replay and duplicate handling. These requirements create room for sophisticated products, but they also mean that streaming adoption should follow a real business need.
Finally, the most durable integration assets may be business definitions and operational knowledge rather than transfer code. A pipeline that accurately reconciles customers across systems is harder to replace than one that merely copies a table. Suppliers that make those definitions transparent and maintainable can create trust without relying solely on technical lock-in.
What to watch
Watch source-to-destination freshness, failure rates, recovery time, data completeness and the engineering hours needed to keep pipelines running. Compare customer spending against changes actually used downstream. For vendors, follow retention, support intensity, connector utilisation and the effect of native integrations on new business.
For AI, test whether updates, deletions and permissions reach retrieval systems reliably. Track production use rather than pipeline demonstrations. The enduring opportunity is to deliver trustworthy context at the moment it matters, with a cost and operating burden that customers can sustain.
Explore this sector
Data platforms & analytics — sector overview
Related sectors: Semiconductors & chipmaking · AI infrastructure & data centres
