# A JSON and Table Pipeline for Any REST API

![banner](https://articles.aux4.blog/data/posts/aux4/a-json-and-table-pipeline-for-any-api/banner.png)

## Introduction

Exploring an API from the terminal usually means a `curl` command, a wall of JSON, and a lot of squinting. aux4 turns it into a tidy pipeline instead: fetch, cache, slice, and tabulate — each step a small package, connected by pipes.

We'll start with a public endpoint that needs no credentials so you can run every command as-is, add caching so you stop hammering the API while you experiment, and finish by upgrading the *same* pipeline to OAuth-authenticated requests. The shape never changes; only the fetch command does.

## Install the packages

```bash
aux4 aux4 pkger install aux4/curl aux4/json aux4/cache aux4/2table
```

## Step 1: fetch and tabulate

GitHub's REST API allows unauthenticated GET requests (rate-limited, but plenty for exploring), which makes it a perfect no-setup example. `aux4 curl request` prints the response body; `aux4 2table` renders a JSON array as a table:

```bash
aux4 curl request "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 2table --format md name,language,stargazers_count
```

| name | language | stargazers_count |
| --- | --- | ---: |
| aux4 | Go | 9 |
| playbook | JavaScript | 0 |
| dispatcher.js | JavaScript | 0 |
| use-handler | JavaScript | 0 |
| 2table | JavaScript | 2 |
| ... | ... | ... |

One pipe, and the JSON is a readable table. But that endpoint returns dozens of repos — let's get control over the volume.

## Step 2: slice the JSON with aux4/json

`aux4/json` is a streaming JSON toolkit. The handiest command here is `json page`, which takes an `--offset` and a `--limit` and returns just that slice of an array:

```bash
aux4 curl request "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 json page --offset 0 --limit 5 \
  | aux4 2table --format md name,language,stargazers_count
```

| name | language | stargazers_count |
| --- | --- | ---: |
| aux4 | Go | 9 |
| playbook | JavaScript | 0 |
| dispatcher.js | JavaScript | 0 |
| use-handler | JavaScript | 0 |
| 2table | JavaScript | 2 |

Now you get the first five rows. The package has more where that came from:

- `aux4 json get '$.path.to.field'` — pull a value out by JSONPath.
- `aux4 json count` — print how many items are in an array.
- `aux4 json collect` — combine an NDJSON stream back into a single array.
- `aux4 json pretty` / `aux4 json inline` — reformat expanded or compact.

For large responses, `json page --stream true` emits NDJSON one item per line so you never hold the whole array in memory; collect it back with `json collect` right before the table:

```bash
aux4 curl request "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 json page --limit 100 --stream true \
  | aux4 json collect \
  | aux4 2table name,language
```

```text
 name                       language
 aux4                       Go
 playbook                   JavaScript
 dispatcher.js              JavaScript
 use-handler                JavaScript
 2table                     JavaScript
 ...
```

## Step 3: stop hitting the API — cache it

While you're iterating on column lists and page sizes, refetching every time is wasteful (and on a rate-limited API, self-defeating). `aux4 cache` wraps *any* command and memoizes its output for a TTL. Just prefix your fetch:

```bash
aux4 cache --cacheDuration 600 curl request \
  "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 json page --offset 0 --limit 5 \
  | aux4 2table --format md name,language,stargazers_count
```

The first run hits GitHub (~0.2s); for the next 10 minutes (`600` seconds) the response comes straight from `.aux4.cache` (~0.04s). The cache key is a hash of the full command including the URL, so different endpoints cache independently.

Prefer rules over flags? Drive durations from config with prefix patterns:

```yaml
config:
  cache:
    defaultDuration: 300
    patterns:
      - match: "curl *"
        duration: 600
```

```bash
aux4 cache --configFile config.yaml curl request "https://api.github.com/orgs/aux4/repos"
```

Manage what's stored with `aux4 cache list` — it prints age and remaining TTL per entry as JSON — and clear it with `aux4 cache clear`:

```bash
aux4 cache list
```

```json
[
  {
    "command": "curl request https://api.github.com/orgs/aux4/repos?per_page=100",
    "age": 3,
    "duration": 600,
    "remaining": 597,
    "valid": true
  }
]
```

> Cache GETs, not mutations. The key is built from the command and URL, not from request bodies or auth tokens — so it's ideal for read-heavy endpoints and wrong for POSTs.

## Step 4: upgrade to authenticated calls

Most real APIs need a token. The elegant part is that you don't rewrite the pipeline — you swap `curl request` for `curl auth-request` and the rest is unchanged.

First, log in once. `aux4 curl oauth login` runs the full OAuth flow (with PKCE) in your browser and stores the token under `.oauth/<provider>.json`:

```bash
aux4 curl oauth login github \
  --clientId "$GH_CLIENT_ID" \
  --clientSecret "$GH_CLIENT_SECRET" \
  --authUrl https://github.com/login/oauth/authorize \
  --tokenUrl https://github.com/login/oauth/access_token \
  --scopes read:user
```

Then every request injects the `Authorization: Bearer` header automatically — and refreshes the token when it expires:

```bash
aux4 cache --cacheDuration 600 curl auth-request --provider github \
  "https://api.github.com/user/repos?per_page=100&visibility=private" \
  | aux4 json page --offset 0 --limit 5 \
  | aux4 2table --format md name,private,stargazers_count
```

Same `json page`, same `2table`, same `cache` — the only change is the fetch command. (Add `.oauth/` to your `.gitignore`; that's where the tokens live.)

If you'd rather keep provider endpoints in config instead of on the command line, `aux4/oauth` is a thin wrapper that resolves them from a `config.yaml` and writes to the same token files, so `curl auth-request --provider <name>` keeps working.

## Conclusion

A reusable recipe for working with any REST API from the shell:

- **curl** fetches (`request` for public, `auth-request` for OAuth),
- **cache** memoizes responses so you stop spamming the endpoint,
- **json** slices and reshapes the payload,
- **2table** renders it for human eyes.

Because the stages are independent pipes, the public and authenticated versions differ by exactly one command. Build the pipeline against a no-auth endpoint, then drop in OAuth when you're ready — your `json` and `2table` steps never know the difference.

## See Also

- [aux4/curl](https://hub.aux4.io/r/public/packages/aux4/curl)
- [aux4/json](https://hub.aux4.io/r/public/packages/aux4/json)
- [aux4/cache](https://hub.aux4.io/r/public/packages/aux4/cache)
- [aux4/2table](https://hub.aux4.io/r/public/packages/aux4/2table)
- [aux4/oauth](https://hub.aux4.io/r/public/packages/aux4/oauth)