# Build an In-Memory Job Queue with queue, jobs, and cron

![banner](https://articles.aux4.blog/data/posts/aux4/an-in-memory-job-queue/banner.png)

## Introduction

Background work usually means standing up a broker like Redis or RabbitMQ and a worker process to match. This post builds a working job pipeline from three small aux4 packages instead: `aux4/queue` carries the messages, `aux4/jobs` runs each one in the background with history and failure hooks, and `aux4/cron` schedules the drain. Our example: a queue of image-resize tasks.

## Install the packages

```bash
aux4 aux4 pkger install aux4/queue aux4/jobs aux4/cron
```

## Step 1: start the queue and publish messages

`aux4/queue` is a small in-memory pub/sub server. Start it, then create a named queue:

```bash
aux4 queue start
```

```text
queue server started on port 8420
```

```bash
aux4 queue create --name resize
```

```json
{ "name": "resize", "status": "CREATED" }
```

Now the one thing you must understand about this queue: **it's pub/sub, not a stored backlog.** A message is delivered to whoever is subscribed *right now* — if nobody is listening, it's gone. Watch it happen. Send two messages with no consumer connected:

```bash
printf '{"file":"a.png"}\n{"file":"b.png"}\n' | aux4 queue send --name resize
```

```json
{ "errors": 0, "sent": 2 }
```

`send` cheerfully reports `sent: 2`, but try to receive them a moment later and nothing comes back — they were never buffered. So the rule is **connect the consumer first, then produce.** Start a receiver (in another shell) that stops after three messages:

```bash
aux4 queue receive --name resize --limit 3
```

`receive` blocks, waiting for messages; `--limit 3` makes it exit cleanly after three. With it listening, publish:

```bash
printf '{"file":"a.png"}\n{"file":"b.png"}\n{"file":"c.png"}\n' | aux4 queue send --name resize
```

The receiver prints them in order and exits:

```json
{"file":"a.png"}
{"file":"b.png"}
{"file":"c.png"}
```

## Step 2: run work as background jobs

`aux4/jobs` runs a command in the background and remembers it. `jobs run` returns immediately:

```bash
aux4 jobs run "echo processing a.png && sleep 1 && echo done"
```

```json
{ "id": "1", "command": "echo processing a.png && sleep 1 && echo done", "state": "RUNNING" }
```

The real value is what happens on failure. `--onFailure` fires a hook when the command exits non-zero — an alert, a retry, a log line. Run one that fails:

```bash
aux4 jobs run "false" --onFailure "echo ALERT: resize failed >> alerts.log"
```

```json
{ "id": "2", "command": "false", "state": "RUNNING", "onFailure": "echo ALERT: resize failed >> alerts.log" }
```

Every run is recorded, so `jobs list` is your history — with final state and exit code:

```bash
aux4 jobs list
```

```json
[
  { "id": "1", "command": "echo processing a.png && sleep 1 && echo done", "state": "COMPLETED", "exitCode": 0 },
  { "id": "2", "command": "false", "state": "FAILED", "exitCode": 1 }
]
```

And the failure hook ran:

```bash
cat alerts.log
```

```text
ALERT: resize failed
```

`jobs output 1` replays a job's stdout — `processing a.png` / `done`. Failures never get lost silently; they're logged, flagged, and can trigger a hook.

## Step 3: drain the queue into jobs

Now wire the two together: a consumer that receives messages and dispatches a job per message. Save `drain.sh`:

```bash
#!/bin/sh
# receive N messages, dispatch a job to process each
aux4 queue receive --name resize --limit "${1:-3}" | while IFS= read -r msg; do
  file=$(echo "$msg" | jq -r '.file')
  aux4 jobs run "echo resized $file" --source drain >/dev/null
  echo "dispatched job for $file"
done
```

Start the drain (the consumer), then publish three messages to it:

```bash
sh drain.sh 3 &
printf '{"file":"x.png"}\n{"file":"y.png"}\n{"file":"z.png"}\n' | aux4 queue send --name resize
```

```text
dispatched job for x.png
dispatched job for y.png
dispatched job for z.png
```

The `--source drain` tag lets you isolate exactly the jobs this consumer created:

```bash
aux4 jobs list --source drain
```

```json
[
  { "id": "3", "command": "echo resized x.png", "state": "COMPLETED", "exitCode": 0, "source": "drain" },
  { "id": "4", "command": "echo resized y.png", "state": "COMPLETED", "exitCode": 0, "source": "drain" },
  { "id": "5", "command": "echo resized z.png", "state": "COMPLETED", "exitCode": 0, "source": "drain" }
]
```

Messages became tracked background jobs — each with output, state, and an exit code you can audit later.

## Step 4: schedule the drain

The last piece is running the drain on a schedule so the queue empties itself. `aux4/cron` is a small scheduler daemon — start it, then add an entry. The command to run goes in `--run`:

```bash
aux4 cron start
```

```text
cron scheduler started on port 8421
```

```bash
aux4 cron add --name drain-resize --every "1 min" --run "sh /tmp/blog-queue/drain.sh 3"
```

```json
{ "name": "drain-resize", "every": "1 min", "run": "sh /tmp/blog-queue/drain.sh 3", "state": "active" }
```

```bash
aux4 cron list
```

```json
[
  { "name": "drain-resize", "every": "1 min", "run": "sh /tmp/blog-queue/drain.sh 3", "state": "active" }
]
```

Cron runs the drain every minute; each firing itself lands in the jobs history, so you have a full audit trail of every scheduled run. Stop everything when you're done:

```bash
aux4 cron stop
aux4 queue stop
```

## Conclusion

Three small packages add up to a real background-work system: `aux4/queue` moves messages (pub/sub — connect the consumer before you produce), `aux4/jobs` runs each one with history and `--onFailure` hooks, and `aux4/cron` schedules the drain. No broker, no worker framework, no database — just composable commands you can inspect and audit at every step.

## See Also

- [aux4/queue](https://hub.aux4.io/r/public/packages/aux4/queue)
- [aux4/jobs](https://hub.aux4.io/r/public/packages/aux4/jobs)
- [aux4/cron](https://hub.aux4.io/r/public/packages/aux4/cron)