"Pi Coding Agent Observability: HTML Specs with Gemini 3.5 Flash and GPT Image 2" — IndyDevDan

Why this is in the vault

This is the most direct external corroboration to date of two things RDCO is actively building: (1) agent observability as a control surface (every event/turn/tool-call/token streamed to a central server, persisted, replayed in a UI) and (2) spec-driven agent workflows where the format of the plan (markdown vs HTML vs visual-HTML) is a measurable lever on agent cost/speed/quality. Both map onto Ray's own thin-harness-fat-skills thesis and the pre-send verification gate the founder wants built. Dan's core claim — "if you don't measure your agents you're gambling with tokens" — is the exact argument for instrumenting Ray's own sub-agent dispatches, and his "all my plans are visual specs now" workflow is a concrete, testable upgrade to how Ray writes implementation specs for sub-agents.

Episode summary

Dan runs three Pi coding agents (all Gemini 3.5 Flash) against the same prompt with three different spec formats — markdown, HTML, and visual-HTML (HTML plus GPT-Image-2-generated interface mockups embedded inline) — and watches the whole thing live through a self-built Pi agent observability dashboard. The dashboard streams every event to a central server, persists to a DB, and replays in a UI with swim-lane, single-agent, and "race" (side-by-side turn alignment) views. The thesis: you cannot manage the performance/speed/cost "trade-off trifecta" of a product agent unless you can see its full trace — system prompt, every tool call, token counts, artifacts. He demos a "Steelman" product agent (generates a bear-case thesis against a stock pick, with generative-UI components) as the running example of a "product-focused agent" running at scale. Surprise finding: the markdown agent burned more tokens than the HTML agent on one run — variance he flags as exactly the kind of thing you'd never catch without observability, and the natural next step is to turn the comparison into a repeatable eval. Closing frame is "tokenomics" — the agentic value chain: use tokens → generate value → capture revenue, with observability as the thing that moves you up the chain.

Key arguments / segments

(00:00–01:41) Measure your agents to improve. Opens with the spec-format question (markdown / HTML / visual). Core mantra: "more useful tokens outperform fewer useful tokens" — the keyword is useful. Reframes the question from "what's better" to "what's the trade-off between performance, speed, and cost." 15+ years engineering; "if you don't measure, you won't improve" stated as an invariant rule.
(01:41–03:18) Pi agent observability architecture. "Dead simple" architecture: stream events to a centralized server → persist to DB → read back in a UI → closed-loop where the engineer ingests the info and improves the agent. Notes a first observed delta: markdown agent used more tokens / more turns (29) than HTML (17) — could be model variance or spec-writing differences. Stresses this matters most for out-of-the-loop product agents, not just terminal engineering agents.
(03:18–09:54) Product-agent observability with the Steelman demo. Introduces the distinction between classic engineering agents (terminal / software-factory / ADW) and product-focused agents that run thousands+ of times a day inside a product. Steelman = a counter-positional agent that builds the bear case against your thesis (demo: Apple as an underappreciated AI-distribution winner), generating UI components (quote, catalyst timeline, valuation gauge) plus 40 tool-call references. Argues default agents are sycophantic ("tell you what you want to hear"), so a deliberately adversarial agent is high-value. Praises Gemini 3.5 Flash on "cost per intelligence" over "cost per token" for product agents.
(09:54–11:07) Know your agent's system prompt. The boot/start event exposes the full rendered system prompt. His point: most engineers have never actually seen their agent's full system prompt, and loading lots of skills visibly bloats the context — only observable by inspecting the start event. Shows the Pi agent's loaded tools (Pi-to-Pi comms, sub-agent delegation, session-digging), tool snippets, prompt guidelines, and skills.
(11:07–20:49) Markdown vs HTML vs visual-HTML specs. The central experiment. Anchored to two external releases: Anthropic's "Unreasonable Effectiveness of HTML" post and OpenAI's GPT Image 2.

Markdown = token-efficient, text-focused. HTML = costs more tokens but communicates UI components more accurately (mocks out quote/timeline/valuation-gauge components visually). Visual-HTML (Vspec) = HTML plus GPT-Image-2 mockup images embedded, which multimodal models like Gemini read directly. Build prompt updated to "if there are any images inside the plan you must read them." On comparison, markdown unexpectedly used more events (170 vs 100) — flags this as either variance or better focus, and says the correct response is to convert it into a repeatable eval. Declares: "all my plans now are visual specs." Notes the observability tool missed the cost of generating the images (~$1–3 total) — an honest instrumentation gap.
(20:49–25:26) Tokenomics / the agentic value chain. Three-level economic funnel: (1) use more tokens → (2) generate value from them (automated assets) → (3) capture revenue (arbitrage). Observability is what moves you up the chain. "Running a fleet of AI coding agents is a great place to start and a terrible place to end." Frames Pi as a minimal, composable harness ("there are many agent harnesses, but this one is yours") that he extends one slice at a time, weekly. Forward view: multimodal specs (text+image+audio+video) will keep growing in value and Gemini holds the multimodal lead ("they have YouTube to train on").

Notable claims (specific techniques / tools)

Observability architecture (directly replicable): stream every agent event (turn, tool call, token count, system prompt, artifact) → central server → persist to DB → replay in a UI with three views: swim-lane (all agents at once), single-agent, and race mode (turns aligned left-to-right side-by-side across agents). Repo: github.com/disler/pi-agent-observability. Shipped as a single Pi extension + single UI.
Inspect the boot/start event to read the full rendered system prompt — surfaces skill-bloat (each loaded skill adds context) that is otherwise invisible. This is the single most transferable diagnostic in the video.
Visual specs (Vspecs): embed GPT-Image-2-generated interface mockups directly into the plan; multimodal agents (esp. Gemini 3.5 Flash) execute against them. Build prompt explicitly instructs the agent to read any images in the plan. "A picture is worth a thousand prompts."
Three spec tiers as a deliberate cost lever: markdown (cheapest, text) → HTML (more tokens, accurate UI communication, renderable proof-of-concept components) → visual-HTML (most expensive, richest, multimodal). Dan also mentions a "markdown-vspec" — markdown that references images — as a middle ground.
"Cost per intelligence" over "cost per token" as the model-selection frame for product agents.
Steelman pattern: a deliberately adversarial / counter-positional agent to fight default LLM sycophancy — generates the bear case with cited references and generative-UI components.
Honest instrumentation gap flagged: the observability tool did not capture image-generation cost (~$1–3/run). Good modeling of "measure, then notice what you're not measuring."
Pi as composable harness: stacks one extension per week (Pi-to-Pi flat comms, sub-agent delegation, session-digging, observability); "adapt Pi to your workflows, not the other way around."

Mapping against Ray Data Co

Strength: STRONG. This is one of the highest-fit external sources for RDCO's current build priority. Three direct connections:

Observability for Ray's own sub-agent dispatches. Ray already routes long artifacts and implementation work through sub-agents (CLAUDE.md hard rule 4; the implementation-notes pattern), but has no trace surface over what those sub-agents cost, how many turns they took, or what their rendered system prompt actually contained. Dan's "inspect the start event to see your full system prompt and skill-bloat" is directly actionable: Ray loads a large skill set, and the same context-bloat problem Dan demonstrates is exactly the "context rot" concern in the Thariq session-management guidance [[2026-04-15-thariq-claude-code-session-management-1m-context]]. Adoptable now: a lightweight per-dispatch trace log (model, turn count, token estimate, skills loaded, tool calls) written alongside the existing implementation-notes file would give the founder the cost/quality visibility he wants without new infra.
Visual specs for sub-agent dispatch. Ray's sub-agent dispatch prompts and implementation specs are currently pure markdown. For any visual build (landing pages via build-landing-page, PDF lead magnets, design surfaces), Dan's Vspec pattern — embed a GPT-Image-2 / reference mockup into the spec the sub-agent executes against — is a concrete upgrade that plugs straight into the existing design-sample workflow (02-sops/design-samples/ already holds visual ground-truth HTML+PDF). The HTML-spec tier specifically maps onto RDCO's existing HyperFrames / HTML-first design substrate. This corroborates the Anthropic "Unreasonable Effectiveness of HTML" thesis the vault already tracks via the harness corpus.
The pre-send verification gate. The founder wants a pre-send verification gate built (already partially codified: /verify-vault-write, /verify-strategic-output, /verify-dispatch, fresh-eyes critic pattern). Dan's Steelman = adversarial-agent-as-quality-gate is the same shape, and his observability layer is what would let Ray measure whether the verification gate is actually catching things vs burning tokens. The "turn the comparison into an eval" step he names is the natural maturation of RDCO's verify-* skills from one-shot checks into measured, repeatable gates.

Directly adoptable technique: the per-dispatch trace log + start-event system-prompt inspection. Low-cost, no new infra, addresses a real visibility gap, and aligns with the L5 north-star focus on "unhobbling the COO agent (toolset + visibility)" [[project_l5_north_star_strategic_direction]] — visibility is literally half that mandate. Caveat: Dan's full dashboard (central server + DB + UI) is over-built for a single-founder agent; adopt the diagnostic discipline (trace + system-prompt inspection + convert-to-eval), not the dashboard, until dispatch volume justifies it.

[[2026-05-18-indy-dev-dan-pi-to-pi-two-way-agent-orchestration]] — prior IndyDevDan Pi note; this video extends the same Pi-harness-as-composable-slices thesis with the observability extension.
[[2026-04-20-indydevdan-pi-agent-teams-harness-engineering]] — IndyDevDan's Pi agent-teams / harness-engineering framing that this builds directly on.
[[2026-04-11-garry-tan-thin-harness-fat-skills]] — the thin-harness-fat-skills thesis; Dan's "skills bloat your system prompt, observe it" is the measurement counterpart to fat-skills.
[[2026-04-15-thariq-claude-code-session-management-1m-context]] — context-rot / session-management; observability is how you'd detect the bloat that guidance warns about.
[[2026-05-18-agentway-harness-engineering-claude-code-design-guide]] — RDCO's harness-engineering reference; observability is the missing instrumentation layer in that design.