Use streaming ETL when the business action needs continuously updated data and the source, processing and destination can meet that freshness contract. Use batch ETL when scheduled processing meets the need with a simpler operating model. Micro-batching sits between these choices. Select the pipeline from the required decision time, recovery behavior and cost—not from the label “real time.”

Key takeaways

• Budget end-to-end freshness, including publication and downstream consumption.
• Throughput, latency and recovery time are separate measures.
• Verify delivery guarantees across the complete source-to-destination path.
• Continuous processing needs ownership for state, lag, replay and evolving schemas.

Streaming vs batch ETL architectures

Batch ETL

Scheduled boundary → bounded source extraction
                   → transformation + validation
                   → accepted target publication
                   → consumer refresh

A batch job processes a defined slice of work. Its source boundary might be a file manifest, a snapshot or a tracked change range. Define that boundary and the point where the output becomes accepted. A schedule alone does not guarantee that the job sees a consistent source state.

Streaming ETL

Source changes → durable event transport
               → incremental processing + state
               → validated destination updates
               → downstream serving and action
                    ↕ checkpoints / replay / reconciliation

A streaming design reacts to arriving data rather than waiting for a large scheduled interval. That does not imply each record is processed independently or that the destination is immediately current. Spark 3.5.6 Structured Streaming documentation, for example, describes a default micro-batch execution model. Match implementation details to the actual runtime and connectors.

Micro-batching

Frequent bounded batches can satisfy a short freshness window without requiring the lowest possible per-event latency. Test startup, minimum compute and small-file costs at the intended cadence. Running an expensive full scan more often is not the same as an efficient incremental pipeline.

Build a freshness budget from the business action backward

Ask when someone or something must act on the information. Distinguish the event’s business time from when the source emits it, when the pipeline accepts it and when the consumer can use it. A fast processor cannot recover time already spent waiting in the source.

End-to-end freshness delay =
  source emission delay
  + ingestion/queue waiting
  + processing and validation
  + destination publication
  + consumer refresh or serving delay

This is an accounting model for the pipeline path, not a performance guarantee. Measure each component under normal, burst and recovery conditions. Record percentile distributions and worst relevant incidents; an average can hide the delayed events that matter most.

Set separate expectations for provisional and finalized results where appropriate. An operations dashboard might accept a clearly labeled preliminary count, while a financial close needs a reconciled boundary and approved corrections.

Feature matrix: architectural trade-offs

These are design considerations, not inherent guarantees. A poorly designed batch job can be harder to recover than a well-operated stream, and an always-running system can still deliver stale data.

Delivery semantics and replay

A processing engine’s guarantee has a scope. Kafka’s delivery-semantics discussion distinguishes publication and consumption behavior and explains that external destinations need cooperation to obtain corresponding end-to-end guarantees. Do not infer that a broker transaction makes an unrelated external API call occur exactly once.

Spark’s documented foreachBatch behavior also requires explicit reasoning about custom output writes: its default write guarantee is at least once. A callback that retries can repeat a business effect unless the destination supports an appropriate deduplication or transactional design.

For each destination, define the accepted event key, ordering scope and update policy. Test a failure after the target write but before progress is recorded. If replay duplicates a transaction or sends a second external instruction, the implementation has not met an exactly-once business-effect requirement.

Late data needs its own contract. Decide how long results may change, where too-late records go and how a repair reaches downstream users. Watermark configuration is an implementation choice with consequences; it is not proof that all events beyond a threshold are invalid.

For batch processing, an idempotent rerun matters just as much. Re-extracting the same source range should not double target rows. Keep a durable run boundary and reconcile accepted, rejected and pending records.

A throughput and failure test plan

Use representative payload sizes, key distribution and transformations. Measure accepted records per second alongside event-to-visible latency, error rate, resource consumption and backlog. A high ingestion rate is not useful if the destination falls further behind.

1. Start with a bounded fixture and expected output totals.
2. Run a steady workload, then a realistic burst and skewed-key case.
3. Pause the destination and observe backpressure or backlog growth.
4. Restart from a checkpoint or batch boundary.
5. Inject duplicates, late records and a schema change.
6. Reconcile the destination and measure time to catch up.

No throughput benchmark is claimed here. Save runtime versions, resource configuration, fixture revision and background workload with actual test results.

Decision flow and practical examples

Define the latest useful decision time
        ↓
Can a scheduled, bounded load meet it reliably?
  Yes → start with batch and test recovery
  No  → test incremental micro-batch freshness
        ↓
Still insufficient, with a source that emits changes?
  Evaluate streaming + state + destination guarantees
        ↓
Verify correctness, recovery, operations and total cost

Finance reporting: a reconciled daily boundary can favor batch if it meets the reporting need. Preserve a repair path for backdated corrections.

Inventory visibility: frequent incremental updates can reduce stale availability, but the checkout system still needs its own authoritative reservation rules. An analytical stream should not be mistaken for transactional inventory control.

Equipment monitoring: event-driven processing may support time-sensitive alerts. Define sensor gaps, late readings, alert deduplication and human response; processing speed alone does not establish a safe operational decision.

Compare total cost over the same accepted workload: compute, transport, storage, state, retries, monitoring and support. Include engineering effort for replay and schema evolution. A hybrid design can retain raw events for replay, serve timely views and run periodic reconciliation.

Use the CDC guide for source-change boundaries and the quality framework comparison for validation and failure ownership.

Common questions

Choose the freshness your business can use

Bring your decision window, source-change behavior and recovery requirements to a review. Request a data engineering pipeline assessment to compare batch, micro-batch and streaming options against an explicit acceptance plan.