Quarantine Bad Data Instead of Dropping It

Introduction

Filtering bad records out at the door works — until the "bad" batch turns out to be a broken export from an upstream system that you needed to keep and fix, not silently discard. This post builds a self-sorting import: a batch is validated once, and aux4/jobs routes it — a clean batch loads into the database whole, a dirty one is quarantined to a dead-letter queue where you can review, fix, and re-submit it. Nothing is dropped without a trace.

It's the natural next step after Stop Bad Data at the Door.

Install the packages

aux4 aux4 pkger install aux4/jobs aux4/validator aux4/queue aux4/db-sqlite

Step 1: a table, rules, and two batches

Create the destination table and a config.yaml with the validation rules:

aux4 db sqlite execute --database signups.db \
  --query "CREATE TABLE signups (id INTEGER PRIMARY KEY, name TEXT, email TEXT)"

config:
  data:
    signup:
      name:  { path: $.name,  rule: required }
      email: { path: $.email, rule: "required|email" }

Then two incoming batches — one clean, one with a broken email:

// batch-ok.json
[ { "name": "Sally", "email": "sally@example.com" }, { "name": "Priya", "email": "priya@example.com" } ]

// batch-bad.json
[ { "name": "Marco", "email": "marco@example.com" }, { "name": "Chen", "email": "not-an-email" } ]

Step 2: validate once, let the job decide

The trick is to let aux4/jobs do the branching instead of validating twice. Run the validation as a job: validator validate --onlyValid prints the clean rows and exits 0 when the whole batch passes or 40 when anything fails. The job's exit code is the branch:

on success — the batch was clean, so insert the rows the job already captured (aux4 jobs output $AUX4_JOB_ID) — no second validation.
on failure — the batch had a bad record, so send it to a quarantine queue.

Bottle it into an aux4 import command. Create .aux4:

{
  "profiles": [
    {
      "name": "main",
      "commands": [
        {
          "name": "import",
          "execute": [
            "aux4 jobs run \"cat ${file} | aux4 validator validate --configFile config.yaml --config data --rules signup --onlyValid\" --onSuccess \"aux4 jobs output \\$$AUX4_JOB_ID | aux4 db sqlite execute --database signups.db --query 'INSERT INTO signups (name,email) VALUES (:name,:email)' --inputStream\" --onFailure \"aux4 json inline < ${file} | aux4 queue send --name quarantine\""
          ],
          "help": {
            "text": "Import a batch: load it if clean, quarantine it if not",
            "variables": [
              {
                "name": "file",
                "text": "Batch file to import",
                "arg": true
              }
            ]
          }
        }
      ]
    }
  ]
}

Two details worth knowing. This is an all-or-nothing gate — if any record in a batch is bad, the whole batch is quarantined and nothing is partially loaded, so you never end up with half an import. And \\$$AUX4_JOB_ID is escaped on purpose: it keeps the job-id variable literal through both aux4 and the shell, so the success hook resolves it at run time and reads that job's captured output.

Step 3: a worker for the quarantine queue

aux4/queue is pub/sub — a message is only delivered to a consumer connected right now — so the queue is the transport, and a worker gives the rejects a durable home. Start the queue and a consumer that appends each quarantined batch to a dead-letter log:

aux4 queue start
aux4 queue create --name quarantine

aux4 queue receive --name quarantine >> rejects.ndjson &

In production this worker could re-validate to attach the errors, alert a channel, or open a ticket. Here it just parks each rejected batch in rejects.ndjson for review.

Step 4: import the batches

With the worker listening, import the clean batch. The job runs, succeeds, and its success hook loads the rows:

aux4 import batch-ok.json

{ "id": "1", "command": "cat batch-ok.json | aux4 validator validate ... --onlyValid", "state": "RUNNING" }

Now the dirty one — the job fails validation, so its failure hook ships the batch to the quarantine queue instead:

aux4 import batch-bad.json

{ "id": "2", "command": "cat batch-bad.json | aux4 validator validate ... --onlyValid", "state": "RUNNING" }

Step 5: clean data landed, bad data parked

The table holds only the clean batch:

aux4 db sqlite execute --database signups.db --query "SELECT id,name,email FROM signups" | aux4 2table --table id,name,email

 id  name   email
  1  Sally  sally@example.com
  2  Priya  priya@example.com

And the broken batch is waiting in the dead-letter log — intact, ready to fix and re-import:

cat rejects.ndjson

[{"email":"marco@example.com","name":"Marco"},{"email":"not-an-email","name":"Chen"}]

Nothing was silently dropped. Marco's record wasn't half-loaded, and Chen's bad email didn't vanish — the whole batch is quarantined where a human (or another job) can correct not-an-email and run aux4 import again.

Conclusion

You built a self-sorting import from four small packages: aux4/validator checks the batch once, aux4/jobs turns its pass/fail into a branch — load or quarantine — aux4/queue carries the rejects to a worker, and aux4/db-sqlite stores the clean data. Bad data isn't dropped and it isn't left to break the load; it's set aside, intact, for another pass.