# Quarantine Bad Data Instead of Dropping It

![banner](https://articles.aux4.blog/data/posts/aux4/quarantine-bad-data-instead-of-dropping-it/banner.png)

## Introduction

Filtering bad records out at the door works — until the "bad" batch turns out to be a broken export from an upstream system that you needed to keep and fix, not silently discard. This post builds a self-sorting import: a batch is validated once, and `aux4/jobs` routes it — a clean batch loads into the database whole, a dirty one is quarantined to a dead-letter queue where you can review, fix, and re-submit it. Nothing is dropped without a trace.

It's the natural next step after [Stop Bad Data at the Door](/posts/aux4/stop-bad-data-at-the-door).

## Install the packages

```bash
aux4 aux4 pkger install aux4/jobs aux4/validator aux4/queue aux4/db-sqlite
```

## Step 1: a table, rules, and two batches

Create the destination table and a `config.yaml` with the validation rules:

```bash
aux4 db sqlite execute --database signups.db \
  --query "CREATE TABLE signups (id INTEGER PRIMARY KEY, name TEXT, email TEXT)"
```

```yaml
config:
  data:
    signup:
      name:  { path: $.name,  rule: required }
      email: { path: $.email, rule: "required|email" }
```

Then two incoming batches — one clean, one with a broken email:

```json
// batch-ok.json
[ { "name": "Sally", "email": "sally@example.com" }, { "name": "Priya", "email": "priya@example.com" } ]
```

```json
// batch-bad.json
[ { "name": "Marco", "email": "marco@example.com" }, { "name": "Chen", "email": "not-an-email" } ]
```

## Step 2: validate once, let the job decide

The trick is to let `aux4/jobs` do the branching instead of validating twice. Run the validation *as a job*: `validator validate --onlyValid` prints the clean rows **and** exits `0` when the whole batch passes or `40` when anything fails. The job's exit code is the branch:

- **on success** — the batch was clean, so insert the rows the job already captured (`aux4 jobs output $AUX4_JOB_ID`) — no second validation.
- **on failure** — the batch had a bad record, so send it to a `quarantine` queue.

Bottle it into an `aux4 import` command. Create `.aux4`:

```json
{
  "profiles": [
    {
      "name": "main",
      "commands": [
        {
          "name": "import",
          "execute": [
            "aux4 jobs run \"cat ${file} | aux4 validator validate --configFile config.yaml --config data --rules signup --onlyValid\" --onSuccess \"aux4 jobs output \\$$AUX4_JOB_ID | aux4 db sqlite execute --database signups.db --query 'INSERT INTO signups (name,email) VALUES (:name,:email)' --inputStream\" --onFailure \"aux4 json inline < ${file} | aux4 queue send --name quarantine\""
          ],
          "help": {
            "text": "Import a batch: load it if clean, quarantine it if not",
            "variables": [
              {
                "name": "file",
                "text": "Batch file to import",
                "arg": true
              }
            ]
          }
        }
      ]
    }
  ]
}
```

Two details worth knowing. This is an **all-or-nothing gate** — if any record in a batch is bad, the whole batch is quarantined and *nothing* is partially loaded, so you never end up with half an import. And `\\$$AUX4_JOB_ID` is escaped on purpose: it keeps the job-id variable literal through both aux4 and the shell, so the success hook resolves it at run time and reads that job's captured output.

## Step 3: a worker for the quarantine queue

`aux4/queue` is pub/sub — a message is only delivered to a consumer connected *right now* — so the queue is the transport, and a worker gives the rejects a durable home. Start the queue and a consumer that appends each quarantined batch to a dead-letter log:

```bash
aux4 queue start
aux4 queue create --name quarantine
```

```bash
aux4 queue receive --name quarantine >> rejects.ndjson &
```

In production this worker could re-validate to attach the errors, alert a channel, or open a ticket. Here it just parks each rejected batch in `rejects.ndjson` for review.

## Step 4: import the batches

With the worker listening, import the clean batch. The job runs, succeeds, and its success hook loads the rows:

```bash
aux4 import batch-ok.json
```

```json
{ "id": "1", "command": "cat batch-ok.json | aux4 validator validate ... --onlyValid", "state": "RUNNING" }
```

Now the dirty one — the job fails validation, so its failure hook ships the batch to the quarantine queue instead:

```bash
aux4 import batch-bad.json
```

```json
{ "id": "2", "command": "cat batch-bad.json | aux4 validator validate ... --onlyValid", "state": "RUNNING" }
```

## Step 5: clean data landed, bad data parked

The table holds only the clean batch:

```bash
aux4 db sqlite execute --database signups.db --query "SELECT id,name,email FROM signups" | aux4 2table --table id,name,email
```

```text
 id  name   email
  1  Sally  sally@example.com
  2  Priya  priya@example.com
```

And the broken batch is waiting in the dead-letter log — intact, ready to fix and re-import:

```bash
cat rejects.ndjson
```

```json
[{"email":"marco@example.com","name":"Marco"},{"email":"not-an-email","name":"Chen"}]
```

Nothing was silently dropped. Marco's record wasn't half-loaded, and Chen's bad email didn't vanish — the whole batch is quarantined where a human (or another job) can correct `not-an-email` and run `aux4 import` again.

## Conclusion

You built a self-sorting import from four small packages: `aux4/validator` checks the batch once, `aux4/jobs` turns its pass/fail into a branch — load or quarantine — `aux4/queue` carries the rejects to a worker, and `aux4/db-sqlite` stores the clean data. Bad data isn't dropped and it isn't left to break the load; it's set aside, intact, for another pass.

## See Also

- [aux4/jobs](https://hub.aux4.io/r/public/packages/aux4/jobs)
- [aux4/validator](https://hub.aux4.io/r/public/packages/aux4/validator)
- [aux4/queue](https://hub.aux4.io/r/public/packages/aux4/queue)
- [aux4/db-sqlite](https://hub.aux4.io/r/public/packages/aux4/db-sqlite)