Shipping in 48 Hours: Choosing an Architecture for an Amplitude-Style Product Loop
Amplitude's customer story with OpenAI describes a loop where a proactive product agent, Wave, identifies what to build, hands changes to coding agents, and then measures the impact — with improvements landing in one to two days. The interesting part isn't the model. It's that the same system closes the loop from signal to shipped change to measurement, which is the part most teams still do by hand across three tools and two meetings.
If you're trying to reproduce that, the real question isn't "which model?" It's "which architecture?" There are at least four credible ways to build an agentic product-improvement loop, and they differ wildly in cost, governance, and how fast you get to a first working cycle.

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: openai.com
What Amplitude actually built
Per the case study, Wave sits on top of OpenAI models and does three jobs:
- Diagnose — analyze product telemetry and user behavior to propose what to build next.
- Build — drive coding agents that produce the change, rather than just filing a ticket.
- Measure — evaluate whether the shipped change moved the metric, feeding the next cycle.
That third step is the one people skip. A loop that doesn't measure its own output is just a suggestion engine with a nice UI. Amplitude can do it because it is the analytics layer — the measurement surface is the same product the agent read from. That's an advantage you probably don't have, and it should shape your architecture choice more than model benchmarks do.
The four options
Option A — Self-hosted open-weights pipeline. You run an open model (or a fine-tune) for classification, summarization, and proposal generation; you own everything. Best data control, worst time-to-first-loop. Justifiable when you have GPU capacity and a hard residency requirement, or when your query volume makes per-token pricing absurd.
Option B — Frontier-hosted with orchestration (the Amplitude pattern). Route reasoning and planning to a frontier model, use coding agents for implementation, and orchestrate retrieval over your own telemetry. Fastest path to a working loop; quality on ambiguous "what should we build" questions is the best available. You inherit vendor dependency and a per-decision cost that scales with usage.
Option C — Buy the vendor copilot. Your analytics or product tool ships an AI assistant. Near-zero build cost, fastest to start, and usually the weakest option for end-to-end shipping because the copilot stops at recommendations and rarely touches your repo.
Option D — Hybrid routing. Frontier models for planning and code generation; small cheap models for high-volume classification, tagging, dedup, and routing; deterministic code for anything that must be auditable. More glue code, but it's how the economics usually survive contact with production.
Criteria that actually separate them
| Criterion | A: Self-hosted OSS | B: Frontier-hosted | C: Vendor copilot | D: Hybrid routing |
|---|---|---|---|---|
| Time to first working loop | Weeks–months | Days–weeks | Hours | Weeks |
| Quality on ambiguous "what to build" | Medium | High | Low–Medium | High |
| Cost at low volume | High (fixed) | Low | Lowest | Low |
| Cost at high volume | Lowest marginal | Scales linearly | Bundled | Lowest overall |
| Data residency control | Full | Contractual | Vendor-defined | Full for routed traffic |
| Eval & observability burden | On you | On you (but traceable) | Opaque | On you, by design |
| Lock-in risk | None | Medium–High | High | Low–Medium |
| Skills required | ML infra + fine-tuning | Prompt/orchestration + evals | None | Orchestration + routing logic |
| Closing the measure step | You build it | You build it | Usually absent | You build it |
The table's least popular column is the last row. Every option above still requires you to build the measurement half of the loop. No vendor sells you a closed cycle, because the cycle has to terminate in your own metrics, your own holdouts, and your own deploy pipeline.
Recommendation by scenario
You're a 5–50 person product team with an existing analytics stack. Go with B. Don't build infrastructure you'll throw away. Point a frontier model at your event data, generate proposals, and — critically — wire a human approval gate before any coding agent touches the repo. Ship the approval gate in week one; add autonomy later.
You're in a regulated environment (health, finance, EU public sector). Start with D, even though it's more work. Route only redacted, aggregated telemetry to a hosted frontier model, keep raw PII and per-user records on your own smaller model, and make every proposal traceable to a stored query. The orchestration overhead is cheaper than the compliance review you'd need for B.
You have very high query volume or a fixed GPU budget. A becomes rational once your monthly frontier spend exceeds roughly the fully-loaded cost of a small inference cluster plus one engineer. Below that threshold, self-hosting is a hobby with a pager.
You just want to know what to build next, not ship it. C is fine, and don't let anyone tell you otherwise. The Amplitude pattern's value is in the shipping and measuring, not the recommending.
You already have OpenAI in production. B, extended. Reuse your existing evals, logging, and key management. Amplitude's reported 1–2 day cycle depends more on organizational latency — who can approve a change — than on inference speed.
Three implementation notes before you start
Separate the model roles. Diagnosis, code generation, and measurement narration have different failure modes and different cost profiles. One frontier model doing all three is the expensive default, not the right one.
Build evals before you build autonomy. You need a labeled set of "good proposals" and "good diffs" before you let the loop run unattended. Without it, you cannot tell a quality regression from a busy week.
Design the measurement step first. Decide what counterfactual you'll use — holdout cohort, staged rollout, pre/post with a control — before the first agent-generated change ships. Otherwise you'll have velocity and no evidence, which is the exact failure mode this architecture is supposed to fix.

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: openai.com
Where the pattern breaks
Two honest caveats. First, the 1–2 day figure is a case-study claim from a company that owns its analytics primitives; your cycle time will be dominated by review, CI, and release process, not by the agent. Second, an agent that both proposes and implements changes creates a feedback loop with no adversarial check unless you deliberately build one. Assign a human owner per metric, and rotate who reviews agent-authored diffs.
The architecture is copyable. The discipline isn't.
Sources
- OpenAI Business, customer story: Amplitude — https://openai.com/business/customer-stories/amplitude
- Details on Wave's three-stage loop (diagnose, build with coding agents, measure) and the 1–2 day shipping figure are drawn from the above case study. Cost thresholds and architecture recommendations in this article are the author's analysis, not claims from the source.



