A data pipeline is a system that moves data from one place to another, transforming it along the way. This definition is simple but hides substantial complexity: the source and destination may have very different schemas, the data may need validation and cleaning, the volume may be enormous, and the pipeline may need to run reliably for years without intervention.
The basic structure
Every data pipeline has the same fundamental structure:
Extract: Read data from one or more sources. Sources might be relational databases, APIs, files, message queues, or event streams.
Transform: Modify the data — cleaning, filtering, reshaping, aggregating, enriching, joining across sources.
Load: Write the transformed data to a destination. Destinations might be data warehouses, databases, files, or other pipelines.
This ETL pattern is well-established, though in practice the boundaries are often blurry. Some systems use ELT (Extract-Load-Transform), where raw data is loaded first and transformation happens inside the destination using SQL.
Batch vs. streaming
The fundamental choice in pipeline design is whether to process data in batches or as a continuous stream.
Batch pipelines process data in discrete chunks at scheduled intervals — every hour, every day, every month. The data warehouse is updated when the batch runs. Batch pipelines are simpler to build, easier to reason about, and sufficient when you do not need real-time results.
Streaming pipelines process data as it arrives, event by event. They are necessary when you need low latency — real-time dashboards, fraud detection, live recommendations. But they are more complex: you need to handle late-arriving events, manage state across a distributed system, and deal with "exactly once" processing guarantees.
Hybrid architectures (sometimes called the Lambda or Kappa architecture) combine both: a streaming layer for real-time results and a batch layer for historical accuracy, with the results merged at query time.
Key components
A modern data pipeline typically involves several distinct systems:
Message queues / event streaming: Kafka, Kinesis, or Pub/Sub serve as the backbone of streaming systems — durable, ordered, distributed event logs that decouple producers from consumers.
Orchestration: Tools like Apache Airflow, Dagster, or Prefect manage the scheduling, execution, and monitoring of pipeline tasks. They define dependencies between steps and handle retries and failures.
Transformation: dbt (data build tool) has become the standard for SQL-based transformation inside data warehouses. For Python-based transformation, tools like Spark, Pandas, or Polars are common.
Storage: Data lakes (object storage like S3 or GCS) store raw data at low cost. Data warehouses (BigQuery, Snowflake, Redshift) provide fast analytical query capability over structured data.
Observability: Monitoring data quality, catching schema changes, alerting on pipeline failures — all critical for production pipelines.
Schema evolution
One of the most persistent challenges in data pipelines is dealing with schema changes in source systems. If an upstream API changes the structure of its response, pipelines that depend on specific fields will break.
Strategies for handling schema evolution:
- Schema registries: Centralize schema definitions and validate messages against them. Confluent Schema Registry for Kafka, or similar tools.
- Versioning: Include version numbers in data and handle multiple versions explicitly.
- Schema validation: Validate incoming data against expected schemas before processing.
- Flexible formats: Use formats like Parquet or Avro that support schema evolution natively.
Idempotency and re-processing
A well-designed pipeline can be safely re-run without producing duplicate or incorrect results. This property — idempotency — is essential for reliability: when a pipeline fails partway through, you want to be able to restart it without fear of corruption.
Achieving idempotency typically requires:
- Upsert operations rather than pure inserts (update existing records, don't duplicate them)
- Partition-based processing where each run owns specific time windows
- Careful handling of timestamps and deduplication keys
Monitoring and data quality
Pipelines that run silently and produce incorrect data are worse than pipelines that fail loudly. Production data pipelines should monitor:
- Volume: Is the expected amount of data arriving?
- Freshness: Is data arriving on schedule?
- Distribution: Are key field distributions within expected ranges?
- Null rates: Are required fields suddenly null?
- Referential integrity: Do foreign keys resolve?
Tools like Great Expectations, dbt tests, or Soda can encode data quality rules that run as part of the pipeline.
When to build vs. buy
Not every data movement problem requires building a custom pipeline. Managed connectors from Fivetran, Airbyte, or similar tools handle many standard source-to-warehouse integrations out of the box. Building custom pipelines makes sense when:
- Your source or destination is nonstandard
- You need processing logic that off-the-shelf connectors cannot provide
- Volume or cost constraints make managed services impractical
Summary
Data pipelines extract data from sources, transform it, and load it to destinations. The choice between batch and streaming depends on latency requirements. Key components include orchestration tools, transformation frameworks, storage systems, and data quality monitoring. Idempotency — the ability to safely re-run pipelines — is a design goal worth planning for from the beginning.