Back to blog

The System That Waits

Published
31 Oct 2025
Tags
  • batch processing
  • systems design
  • architecture

Every system starts full of optimism.
Then someone loads a million records, half fail silently, and suddenly your “real-time” dream looks like a pile of pending retries.

That’s when you start respecting the idea of a batch.

Batch is not nostalgia.
It’s the moment an architecture admits that chaos needs structure and time needs fences.

Why I Think in Batches

When I build or audit systems, I don’t start with APIs or tools.
I start with a calendar.

Not everything must happen now.
Some things should happen after verification, in groups, with a clear trail.
That’s how you keep large, unpredictable systems alive:
by giving them a rhythm instead of a heartbeat disorder.

Batch is time-boxed honesty.
It’s saying, “We’ll process this when we’re ready, and we’ll know what happened when it’s done.”

The Shape of a Batch

Forget the buzzwords. A functioning batch process usually has four moving parts:

  1. Collection — where data gathers and waits.
    It can be a file, a queue, or a database snapshot. Doesn’t matter. What matters is that it stops the flood.
  2. Validation — the gatekeeper. Everything wrong is caught here, logged, and set aside.
  3. Execution — the workers take over, chunk by chunk, predictable and boring. That’s the goal.
  4. Result — two piles: done and broken. Both are valuable. One feeds production, the other feeds improvement.

That’s it. The rest is detail — scheduling, retries, notifications — the usual plumbing of control.

A Well-Designed Batch System

A well-designed batch system isn’t just a loop of background tasks.
It’s a controlled workflow with defined stages and clear ownership of each phase.

Layer Responsibility
Input Collector Aggregates jobs (from database, API, message queue, file upload)
Batch Manager Groups requests into batches, assigns IDs, validates format and dependencies
Executor / Worker Pool Executes jobs asynchronously (local threads, serverless jobs, or remote API)
Status Tracker Monitors progress and maintains states like Pending, In Progress, Completed, Failed, Expired
Result Store Saves successful and failed results separately for review and downstream processing
Monitoring & Retry Logic Detects partial failures, supports re-run of specific jobs

This pattern scales from local Python scripts to global distributed job schedulers.
The structure doesn’t change — only the tools do.

Lifecycle and Status Model

Every batch job has a lifecycle. A robust status model keeps the system transparent and recoverable.

Status Meaning
validating Input file or payload is being checked
in_progress Tasks are currently running
completed All tasks finished successfully
failed Validation or execution error
partial_success Some jobs succeeded, others failed
expired Time window exceeded before completion
cancelled Batch manually stopped
archived Output saved and cleaned from active memory

In practice, you can implement this via a relational table, document DB, or even cloud storage metadata.
The main point is traceability — every job, every failure, every rerun.

Data Flow Example

Imagine we need to process 10,000 data records for enrichment or validation.

  1. Collect jobs → Generate a .jsonl or .csv file with unique IDs and payloads
  2. Register batch → Insert a record into batch_jobs table with metadata (source, created_at, status)
  3. Validate input → Check format, schema, duplicates
  4. Execute asynchronously → Split jobs across worker threads or serverless functions
  5. Monitor progress → Update job statuses periodically; aggregate metrics
  6. Consolidate results → Merge successful outputs and write to an output file or staging table
  7. Error handling → Capture failed tasks, store logs and input parameters for retry
  8. Completion → Notify downstream systems or users via callback / event

Output and Verification

After execution, every batch should produce three things:

  • Result file (or table): all successfully processed records
  • Error file: failed or invalid records with reasons
  • Audit log: technical trace of execution time, parameters, and system metrics

That triad allows you to debug, retry, and measure performance without manual inspection.

For large-scale data, output order shouldn’t be relied upon.
Instead, match results by unique job IDs — the same way professional batch APIs do.

Verification

The last step is always reflection in code:
compare input vs. output, count records, match IDs.
If numbers don’t balance, something’s lying.

Good batch systems don’t hide lies — they expose them in neat CSVs.

Why It Matters

Real-time everything is an expensive illusion if you can’t prove what actually ran.

Batch design brings back accountability.
It gives you checkpoints, human-readable evidence, and the chance to breathe before the next wave of data.