Why this is in the vault
Founder: "something we should model out to experiment with our Shopify store, or other future Shopify stores." RDCO is launching a small Shopify coloring-book store ([[2026-10-06-shopify-launch-plan]]) and its scorecard asks whether AI shopping agents like Meta's Muse surface or sell the book. These three papers are Shopify's research on (a) simulating shoppers to predict A/B results, (b) benchmarking shopping agents, and (c) learning realistic buyer types. Note: the ShopifyEng post says COLM 2026 (Conference on Language Modeling), but none of the three arXiv PDFs names a venue; all are preprints.
SimGym (arXiv 2602.01443, Feb 2026)
- What: LLM shopper agents browse a control and a treatment version of a live storefront; the change in add-to-cart (A2C) rate predicts the real A/B result offline, with no real buyers exposed.
- Method: cluster the shop's own sessions into 5 groups; an LLM writes two-sentence shopping intents and personas (price sensitivity from browsed-vs-bought price gap, exploration depth, premium, performance and ethics focus). Agents read the accessibility tree (no screenshots), keep full session memory, and have loop/time guardrails. Model not named.
- Numbers: 20 shops, 600 agents per shop. Full system: 0.64 correlation with the human A2C change, 69% direction match. Without memory, correlation falls to 0.29 (55% direction match); with generic personas, to 0.27. Generic personas still got direction right 62% of the time but badly undershot effect size. Correlation stabilizes after ~300 agents per shop and both metrics plateau beyond ~500.
- Caveats (theirs, last point ours): small test set; the "experiments" are natural theme switches measured before/after, not randomized tests; visual-only changes excluded; A2C is the only metric; "state of the art" and "under an hour" are claimed but not benchmarked in the results.
- Data: per-shop production clickstream, enough traffic to cluster. The generic-persona ablation pools the top 10 highest-traffic donor shops in the same industry, region, price tier and catalog size.
ShopGym (arXiv 2605.16116, May 2026)
- What: generates realistic sandbox shops cloned from real storefronts (ShopArena) plus grounded tasks to run shopping agents against them (ShopGuru).
- Method: Claude Code agents (Opus 4.6) browse a seed store's public pages and catalog JSON, write a spec, and rebuild the shop feature by feature. Short tasks are rule-generated from the catalog (exact search, substitute search, browse, filter, shipping and returns lookup); long multi-step journeys are LLM-written from human-behavior examples. A validator rejects impossible tasks (filter value that doesn't exist, answer leaked in the prompt). A GPT-5 judge scores pass/fail; tasks stop at the cart.
- Numbers: 224 tasks, 6 shops. Long multi-step journeys: 48-63% success for GPT-5-mini, Gemini 3 Flash and GPT-5 on the BrowserGym harness (53-58% on Shopify's internal harness). Short tasks roughly 71-86% across both harnesses (read from bar-chart labels). Real vs cloned shop success is "positively correlated" with no coefficient given.
- Caveats: not a substitute for live testing (no personalization, apps, latency); no login, checkout, recommendations, cross-store comparison or multilingual coverage; matching success rates does not mean matching failure modes.
- Data: only public storefront pages and products/collections/pages JSON. Code release promised, no repo URL yet.
SimPersona (arXiv 2605.14205, May 2026)
- What: learns 256 discrete buyer "types" from raw clickstreams and gives each its own token in a fine-tuned Qwen3-14B browser agent, so you can sample a store's real buyer mix.
- Method: a 403-number vector per buyer (funnel rates, cart and order value, embeddings of viewed/bought products over 4 months) clustered by a VQ-VAE (vector-quantized variational autoencoder); traces generated with GPT-5 and Gemini, then two-stage fine-tuning on 16 H200 GPUs.
- Numbers: 42 held-out stores, 8.37M buyers. 78.7% action-rate alignment with the right persona token vs 57.4% with a wrong one; store funnel mix recovered well (mean Jensen-Shannon divergence 0.054). On checkout tasks it reached checkout more often than GPT-OSS-120B, a model about 8x larger (76.9% vs 53.1%), though its self-reported goal-reached rate was lower (77.7% vs 93.9%).
- Caveats: exploration behavior does not transfer; page-structure (DOM, Document Object Model) only, no visuals; no support for new stores without history. It argues prompt-only personas are brittle (cites 11.9% action reproduction for the best prompt-based agents).
- Data: multi-store event logs, catalog, search logs, product embeddings, GPU fine-tuning. Pipeline release promised.
Mapping against Ray Data Co
What a tiny store can actually replicate.
Cannot replicate: anything grounded in clickstreams. SimGym's calibrated personas and SimPersona's learned tokens need thousands of sessions per store (SimPersona's smallest eval store had under 1,000 buyers); a pre-launch store has zero. SimGym's own ablation says generic personas get direction but not size, which is the regime we'd be in.
Can build cheaply (all estimates):
- ShopGym-style agent audit (best first experiment). Write ~20-30 shopper prompts from our own products.json and policies ("religious Christmas coloring book for a 6-year-old under $20", "is the PDF refundable?", "does it ship before Dec 20?"). Run them with a browser agent (Playwright + Claude) against our storefront and 2-3 competitor stores, score with a URL/cart check plus an LLM judge, stop before checkout. Failures show where agents can't find, filter or understand our listings, which is the Muse question in miniature. Estimate: ~1 day to build, a few dollars of API per run, rerunnable after every theme change.
- A few hand-built personas (gift-buying grandparent, homeschool parent, Sunday-school teacher, price-checker), using SimGym's intent template and SimPersona's funnel tiers as structure. Treat as qualitative probes, not predictions; SimPersona explicitly argues these are the weak version. Estimate: ~2 hours.
- SimGym merchant preview, if our plan tier is eligible (launch plan already lists it). Expect generic shoppers until traffic arrives, so use it for gross layout/flow choices only. Note the paper itself names no merchant tool; the product claims come from press coverage (Modern Retail, linked in [[2026-10-06-shopify-launch-plan]]). Estimate: ~1 hour, $0.
Bias note
All three are Shopify-authored, Shopify-data, Shopify-platform papers that frame Shopify tooling as the answer; results are unreplicated, ShopGym's realism claim lacks a coefficient, and SimGym's validation is quasi-experimental. Useful as design patterns, not as proof the tools work for a small store.
Related
- [[2026-10-06-shopify-launch-plan]]
- [[2026-09-19-muse-connectors-agent-distribution]] (Scribble Works #333, agent distribution)
- [[2026-09-02-commerce-agents-takeaways]]
- [[2026-07-24-shopify-ucp-global-catalog-assessment]]
- [[2026-09-06-alphasignal-shopify-self-improving-pipeline]]
- [[2026-09-28-ark-invest-amazon-blocks-muse-agentic-commerce]]