06-reference

Shopify's simulated-shopper papers: SimGym, ShopGym, SimPersona

2026-10-07·reference·source: ShopifyEng research (shared by founder via a Sarah Catanzaro post quoting ShopifyEng, "COLM 2026")·by Shopify ML research (Castelo, Wang, Zanjani Foumani, Savadikar et al.)
shopifyagentic-commercebrowser-agentssimulationab-testingpersonascoloring-booksmuse

Why this is in the vault

Founder: "something we should model out to experiment with our Shopify store, or other future Shopify stores." RDCO is launching a small Shopify coloring-book store ([[2026-10-06-shopify-launch-plan]]) and its scorecard asks whether AI shopping agents like Meta's Muse surface or sell the book. These three papers are Shopify's research on (a) simulating shoppers to predict A/B results, (b) benchmarking shopping agents, and (c) learning realistic buyer types. Note: the ShopifyEng post says COLM 2026 (Conference on Language Modeling), but none of the three arXiv PDFs names a venue; all are preprints.

SimGym (arXiv 2602.01443, Feb 2026)

ShopGym (arXiv 2605.16116, May 2026)

SimPersona (arXiv 2605.14205, May 2026)

Mapping against Ray Data Co

What a tiny store can actually replicate.

Cannot replicate: anything grounded in clickstreams. SimGym's calibrated personas and SimPersona's learned tokens need thousands of sessions per store (SimPersona's smallest eval store had under 1,000 buyers); a pre-launch store has zero. SimGym's own ablation says generic personas get direction but not size, which is the regime we'd be in.

Can build cheaply (all estimates):

  1. ShopGym-style agent audit (best first experiment). Write ~20-30 shopper prompts from our own products.json and policies ("religious Christmas coloring book for a 6-year-old under $20", "is the PDF refundable?", "does it ship before Dec 20?"). Run them with a browser agent (Playwright + Claude) against our storefront and 2-3 competitor stores, score with a URL/cart check plus an LLM judge, stop before checkout. Failures show where agents can't find, filter or understand our listings, which is the Muse question in miniature. Estimate: ~1 day to build, a few dollars of API per run, rerunnable after every theme change.
  2. A few hand-built personas (gift-buying grandparent, homeschool parent, Sunday-school teacher, price-checker), using SimGym's intent template and SimPersona's funnel tiers as structure. Treat as qualitative probes, not predictions; SimPersona explicitly argues these are the weak version. Estimate: ~2 hours.
  3. SimGym merchant preview, if our plan tier is eligible (launch plan already lists it). Expect generic shoppers until traffic arrives, so use it for gross layout/flow choices only. Note the paper itself names no merchant tool; the product claims come from press coverage (Modern Retail, linked in [[2026-10-06-shopify-launch-plan]]). Estimate: ~1 hour, $0.

Bias note

All three are Shopify-authored, Shopify-data, Shopify-platform papers that frame Shopify tooling as the answer; results are unreplicated, ShopGym's realism claim lacks a coefficient, and SimGym's validation is quasi-experimental. Useful as design patterns, not as proof the tools work for a small store.

Related