A JSON and Table Pipeline for Any REST API

Introduction

Exploring an API from the terminal usually means a curl command, a wall of JSON, and a lot of squinting. aux4 turns it into a tidy pipeline instead: fetch, cache, slice, and tabulate — each step a small package, connected by pipes.

We'll start with a public endpoint that needs no credentials so you can run every command as-is, add caching so you stop hammering the API while you experiment, and finish by upgrading the same pipeline to OAuth-authenticated requests. The shape never changes; only the fetch command does.

Install the packages

aux4 aux4 pkger install aux4/curl aux4/json aux4/cache aux4/2table

Step 1: fetch and tabulate

GitHub's REST API allows unauthenticated GET requests (rate-limited, but plenty for exploring), which makes it a perfect no-setup example. aux4 curl request prints the response body; aux4 2table renders a JSON array as a table:

aux4 curl request "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 2table --format md name,language,stargazers_count

One pipe, and the JSON is a readable table. But that endpoint returns dozens of repos — let's get control over the volume.

Step 2: slice the JSON with aux4/json

aux4/json is a streaming JSON toolkit. The handiest command here is json page, which takes an --offset and a --limit and returns just that slice of an array:

aux4 curl request "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 json page --offset 0 --limit 5 \
  | aux4 2table --format md name,language,stargazers_count

Now you get the first five rows. The package has more where that came from:

• aux4 json get '$.path.to.field' — pull a value out by JSONPath.
• aux4 json count — print how many items are in an array.
• aux4 json collect — combine an NDJSON stream back into a single array.
• aux4 json pretty / aux4 json inline — reformat expanded or compact.

For large responses, json page --stream true emits NDJSON one item per line so you never hold the whole array in memory; collect it back with json collect right before the table:

aux4 curl request "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 json page --limit 100 --stream true \
  | aux4 json collect \
  | aux4 2table name,language

 name                       language
 aux4                       Go
 playbook                   JavaScript
 dispatcher.js              JavaScript
 use-handler                JavaScript
 2table                     JavaScript
 ...

Step 3: stop hitting the API — cache it

While you're iterating on column lists and page sizes, refetching every time is wasteful (and on a rate-limited API, self-defeating). aux4 cache wraps any command and memoizes its output for a TTL. Just prefix your fetch:

aux4 cache --cacheDuration 600 curl request \
  "https://api.github.com/orgs/aux4/repos?per_page=100" \
  | aux4 json page --offset 0 --limit 5 \
  | aux4 2table --format md name,language,stargazers_count

The first run hits GitHub (0.2s); for the next 10 minutes (600 seconds) the response comes straight from .aux4.cache (0.04s). The cache key is a hash of the full command including the URL, so different endpoints cache independently.

Prefer rules over flags? Drive durations from config with prefix patterns:

config:
  cache:
    defaultDuration: 300
    patterns:
      - match: "curl *"
        duration: 600

aux4 cache --configFile config.yaml curl request "https://api.github.com/orgs/aux4/repos"

Manage what's stored with aux4 cache list — it prints age and remaining TTL per entry as JSON — and clear it with aux4 cache clear:

aux4 cache list

[
  {
    "command": "curl request https://api.github.com/orgs/aux4/repos?per_page=100",
    "age": 3,
    "duration": 600,
    "remaining": 597,
    "valid": true
  }
]

Cache GETs, not mutations. The key is built from the command and URL, not from request bodies or auth tokens — so it's ideal for read-heavy endpoints and wrong for POSTs.

Step 4: upgrade to authenticated calls

Most real APIs need a token. The elegant part is that you don't rewrite the pipeline — you swap curl request for curl auth-request and the rest is unchanged.

First, log in once. aux4 curl oauth login runs the full OAuth flow (with PKCE) in your browser and stores the token under .oauth/<provider>.json:

aux4 curl oauth login github \
  --clientId "$GH_CLIENT_ID" \
  --clientSecret "$GH_CLIENT_SECRET" \
  --authUrl https://github.com/login/oauth/authorize \
  --tokenUrl https://github.com/login/oauth/access_token \
  --scopes read:user

Then every request injects the Authorization: Bearer header automatically — and refreshes the token when it expires:

aux4 cache --cacheDuration 600 curl auth-request --provider github \
  "https://api.github.com/user/repos?per_page=100&visibility=private" \
  | aux4 json page --offset 0 --limit 5 \
  | aux4 2table --format md name,private,stargazers_count

Same json page, same 2table, same cache — the only change is the fetch command. (Add .oauth/ to your .gitignore; that's where the tokens live.)

If you'd rather keep provider endpoints in config instead of on the command line, aux4/oauth is a thin wrapper that resolves them from a config.yaml and writes to the same token files, so curl auth-request --provider <name> keeps working.

Conclusion

A reusable recipe for working with any REST API from the shell:

• curl fetches (request for public, auth-request for OAuth),
• cache memoizes responses so you stop spamming the endpoint,
• json slices and reshapes the payload,
• 2table renders it for human eyes.

Because the stages are independent pipes, the public and authenticated versions differ by exactly one command. Build the pipeline against a no-auth endpoint, then drop in OAuth when you're ready — your json and 2table steps never know the difference.

See Also

• aux4/curl
• aux4/json
• aux4/cache
• aux4/2table
• aux4/oauth