"Vibe Check: GPT-5.6 Sol Is Our Favorite Model to Collaborate With" — @katie.parrott12
Why this is in the vault
Every's multi-contributor Vibe Check team (Katie Parrott, Dan Shipper, Kieran Klaassen, Mike Taylor, Austin Tedesco, Arielle Shipper) evaluated GPT-5.6 Sol across writing, coding, research, and data analysis after it returned from a government review hiatus in late June 2026. The central finding: Sol is the best model for collaborative, iterative work with a human in the loop — but Fable (Claude) still dominates on autonomous assignments where the model must decide what to build without steering. This is a direct input to RDCO toolchain decisions about whether Sol warrants a slot in the agent fleet.
Note: this vault entry is based on the free email preview of a subscriber-only article. The full benchmarks, methodology details, and contributor anecdotes behind the paywall are not included here.
Issue contents
Benchmark results:
- Senior Engineer benchmark: Sol scored 56/100 vs Fable's 90/100 — gap explained largely by Sol producing 12,900 lines of unnecessary code
- Six-model writing benchmark: Sol finished last
Benchmark paradox — the collaboration caveat:
- Despite last-place in writing benchmark, Katie Parrott used Sol to move through 24 drafts of the vibe check article itself in 6–8 hours
- Sol is fast, unusually good at finding and using context, and easy to redirect mid-task
- The benchmark captures standalone output quality; it misses Sol's value as a real-time collaborator when a human is actively steering
Coding — Proof rebuild case (Kieran Klaassen):
- Klaassen rebuilt Proof with Sol in approximately one-third the time Fable required
- Despite the speed win, he preferred Fable's design output
- Speed advantage does not translate to quality advantage on design decisions
Data/research — CSV analysis case (Arielle Shipper):
- Sol found a buried email address, inspected 46 CSV files, and returned 7 useful clarifying questions
- Sol then made a calculation error serious enough to shake her trust
- Pattern: impressive context-retrieval, unreliable on computation
The headline distinction:
- Sol → collaborative work you want to shape as it goes (writing, iterative coding, research with human in the loop)
- Fable → broad assignments where the model decides what should be built; autonomous hand-off tasks
Model context:
- Sol went through government review late June; During the hiatus, Dan Shipper described returning to older models as "going back to the Stone Age"; Austin Tedesco compared GPT-5.5 to "shooting a basketball twice as heavy"
- Sol runs in OpenAI's new unified ChatGPT + Codex desktop app
Mapping against Ray Data Co
The Fable vs Sol distinction maps directly to the RDCO COO agent architecture: Fable is the active daily model precisely because the COO agent runs autonomous loops — tasks where it must "decide what should be built in the first place" without Ben steering in real time. Every's finding validates that choice; Sol's 56/100 Senior Engineer score and calculation errors in CSV analysis confirm it should not replace Fable in the autonomous fleet.
Where Sol could add value: Ben's high-volume iterative writing workflows (24 drafts in 6–8 hours) suggest Sol as a possible secondary drafting surface for Sanity Check article development, where human steering is continuous. Not an agent-fleet addition — a writing-session tool.
The calculation error with 46 CSV files is a hard signal: do not route any financial, modeling, or quantitative data tasks to Sol for autonomous execution. The trust-break pattern (impressive retrieval → computation failure) is exactly the failure mode that would cause silent errors in RDCO's investing pipeline or client deliverables.
Related
- [[2026-07-02-every-vibe-check-sonnet5]] — immediately prior vibe check by the same team; Sonnet 5 also failed to claim a clear default slot against any competitor
- [[2026-06-09-every-vibe-check-fable-5-best-coding-model]] — the Fable benchmark this note implicitly compares against; establishes Fable's coding baseline that Sol's 56/100 falls short of