06-reference/research

queue jsonl task distribution primitive spec

2026-07-02·research-brief·source: deep-research
multi-agenttask-queuefan-outcost-ceilingpipeline-skills

The queue.jsonl Primitive: A Pull-Based Task-Distribution Spec for RDCO pipeline-* Fan-Out

The question

What queue.jsonl / task-distribution primitive should RDCO standardize for pipeline-* fan-out — the per-run scratch dir + task-queue file that lets workers pull tasks instead of being explicitly invoked, plus per-task cost-ceiling enforcement? (Follow-up from the 2026-05-20 fan-out brief: Glasswing has a public 8-stage shape but no public queue.jsonl spec, so RDCO must spec its own. This is a DESIGN deliverable — a concrete spec, plus a candidate /deep-research adoption path.)

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

Design principle: queue.jsonl is the immutable task manifest; claim state and outcomes live beside it. The founder's naming ("queue.jsonl") is kept, but with correct semantics — the JSONL file is written once by the planner (one line per task) and only ever appended to (a Gapfill pass may add re-queued tasks). Task status is never mutated inside the JSONL; it is derived from filesystem state. This gives a clean answer to the "workers pull instead of being invoked" requirement: an orchestrator writes the queue and spawns a fixed pool of K generic worker sub-agents pointed only at run_dir — not one sub-agent per task. Each worker loops: find the highest-priority claimable task, atomically claim it, do the work under its cost ceiling, write its result, mark done, repeat until the queue drains. Worker count is now decoupled from task count (--count 12 becomes 12 tasks drained by 3 pullers, not 12 simultaneous sub-agents), which is the concrete scaling win.

(a) queue.jsonl record schema — one JSON object per line, immutable after write:

(b) per-run scratch-dir layout (extends the existing runs/<domain>-<timestamp>/ station convention):

~/rdco-vault/01-projects/skill-pipelines/runs/<domain>-<YYYYMMDDThhmmssZ>/
  run.json          # run manifest: run_id, domain, created_at, run_budget_usd, pool_size, status
  queue.jsonl       # immutable task list (planner writes; Gapfill may append)
  events.jsonl      # append-only audit log: claim / heartbeat / done / failed / killed-over-budget
  inputs/           # shared read-only inputs the planner drops for workers
  claims/<task_id>/lease.json   # atomic claim dir (mkdir = the lock) + lease + heartbeat
  results/<task_id>.<ext>       # per-task worker output (write BEFORE marking done)
  dead-letter/      # tasks that exhausted max_attempts

(c) pull / claim protocol (worker self-assigns):

  1. Read queue.jsonl; filter to claimable = no claims/<task_id>/ dir AND no results/<task_id> file AND all depends_on are DONE. Sort by priority, then seq.
  2. Attempt atomic claim on the first candidate: mkdir claims/<task_id> — mkdir fails EEXIST if another worker already claimed it (this is the atomic primitive; the mkdir-lock is chosen over CAS because it needs no version-read round-trip and is single-host-atomic on APFS). On failure, move to the next candidate.
  3. On success, write claims/<task_id>/lease.json = {worker_id, claimed_at, expires_at = now + lease_ttl, heartbeat_at} and append a claim event to events.jsonl.
  4. Do the work, touching heartbeat_at / extending expires_at periodically (long tasks) so a live worker's task is never reaped.
  5. On completion, write result_path first, then append the done event (CAF write-before-mark invariant — a crash between the two leaves an orphaned result the integrity check detects). Loop back to step 1. Exit when nothing is claimable.
  6. Lease reaper (orchestrator between dispatch waves, or a sweep): for any claims/<task_id> whose lease expired with no results/<task_id>, delete the claim dir, attempt += 1 (append a re-queued line if attempt < max_attempts, else move to dead-letter/), append a lease-expired event. This is work-stealing crash recovery — a died worker's task returns to the pool for another puller, which the current /deep-research "leave it In Progress for tomorrow" cannot do mid-run.

(d) where per-task cost-ceiling is enforced — three points, defense in depth: (1) Dispatch-time / static — the worker prompt injects cost_ceiling.max_tool_calls as hard caps (the enforceable gate today, extending the /deep-research 5+3+3 pattern) plus max_tokens and wall_clock_s; this is the parent→subtask hierarchical inheritance the web describes. (2) Runtime / self-metering — the worker tracks a running tally and, before any expensive call that would breach usd/max_tokens, stops, writes a partial result + killed-over-budget event, and exits gracefully (advisory until a metering proxy exists; say so). (3) Run-aggregate — run.json.run_budget_usd caps the whole run; the orchestrator sums events.jsonl spend before dispatching each wave and stops dispatching if the run cap is breached (the $47k/11-day and $1,200/night incidents are the reason this exists).

How /deep-research adopts it first. Only Step 4 changes. Instead of spawning one sub-agent per question, the orchestrator writes queue.jsonl to a fresh runs/deep-research-<ts>/ (one line per approved question, cost_ceiling = {usd:5, max_tool_calls:{qmd:5, websearch:3, webfetch:3}}, result_path = the brief file), then spawns K=3 generic worker sub-agents pointed only at run_dir with the pull loop above. Notion stays the approval/backlog system of record; queue.jsonl is the per-run execution substrate. The orchestrator reflects each DONE back to Notion (Status → Done, Brief Path). Net gains over today: bounded concurrency independent of question count, graceful over-budget stop per question, and mid-run crash recovery. The SOP for this belongs at ~/rdco-vault/02-sops/, with the pull-loop worker logic factored into a shared helper the other pipeline-* / station skills import rather than copy-paste.

Example queue.jsonl record:

{"task_id":"deep-research-20260702T0100Z-003","run_id":"deep-research-20260702T0100Z","task_class":"research-question","scope_hint":"queue.jsonl primitive for pipeline-* fan-out","payload":{"question":"What queue.jsonl primitive should RDCO standardize for pipeline-* fan-out?","category":"Trends","notion_page_id":"366f7d49-36d1-8129-b428-cd1f50fb6377","notes":"follow-up from 2026-05-20 fan-out brief"},"depends_on":[],"priority":20,"cost_ceiling":{"usd":5.00,"max_tokens":400000,"max_tool_calls":{"qmd":5,"websearch":3,"webfetch":3},"wall_clock_s":1200},"result_path":"results/deep-research-20260702T0100Z-003.md","attempt":0,"max_attempts":2,"created_at":"2026-07-02T01:00:04Z"}

Open follow-ups

Related

Sources

Vault:

Web: