AI Engineering Pitfalls
Definition
A set of recurring mistakes teams make when building applications on foundation models, catalogued by Chip Huyen. Because everyone uses the same underlying models, the product — not the AI — is usually where projects succeed or fail.
Core Ideas
- Use gen AI when you don’t need it. Not everything is a nail. Many problems have cheaper, more reliable solutions (greedy scheduling, linear programming). Beware chasing the “we use gen AI” headline over “we solved the problem.”
- Confuse ‘bad product’ with ‘bad AI’. The AI is often the easy part; UX is the hard part. Users may want action items, not summaries; helpful answers, not merely correct ones; suggested prompts, not a blank box.
- Start too complex. Reaching for an agent framework, vector DB, or finetuning before trying direct API calls or simple term-based retrieval hides critical detail and adds bugs. Add abstractions only once best practices are proven.
- Over-index on early success. The demo-to-product gap is brutal: LinkedIn took 1 month to reach 80% quality and 4 more months for 95%. Hallucinations, latency/accuracy trade-offs, tool-calling confusion, API reliability, compliance, and safety all bite late.
- Forgo human evaluation. LLM-as-judge is useful but non-deterministic; validate it against daily human review of 30–1000 examples. “Staring at data for 15 minutes usually gives insight that saves hours.”
- Crowdsource use cases. Without a big-picture strategy you get a million Slack bots and text-to-SQL toys — low-impact work that wrongly concludes “gen AI has no ROI.”
The SLA a model API takes away
A pitfall that predates the list above and is easy to miss because it is contractual, not technical: an LLM API with no SLA becomes the SLA of everything it sits in.
Availability composes by multiplication. Two tiers at 90% each give a product at 81%. So if a provider publishes no availability guarantee and the model is on the core path, the honest number for the product is unknown — and any existing commitment is breached, not merely lowered. A service that promised 99.5% cannot keep promising it.
Three responses, each with a real cost:
- Bring your own API key. Moves the per-token cost to the end user and was the common pattern early on. But it breaks what makes something a SaaS — one bill, one login, cross-platform state, data persistence — and it hands the provider a credential to store and protect.
- Self-host and replicate across regions. Recovers the availability guarantee and cuts latency. Expensive, and the expense is mostly time: standing up and maintaining the infrastructure.
- Keep the model off the core path. The cheapest answer when it applies. A Facebook-Marketplace-style app should not depend on a model to extract fields from listings; make it an optional chat assistant instead, so core functionality is independent of an unguaranteed dependency.
The cost side is the sibling problem — early GPT-4 at $0.03 per 1K tokens against GPT-3.5-Turbo at $0.002 is a 15x spread that has to be transparent to users before it becomes a surprise on the bill.
Relationships
- Retrieval-Augmented Generation (RAG) — pitfall #3’s “don’t start with a vector DB” applies directly
- AI Model Routing — managing latency/accuracy/cost trade-offs (pitfall #4)
- OWASP Top 10 for LLM Applications — the safety/compliance risks that surface in production
- Product-Market Fit — “product is the hard part” (pitfalls #2, #6)
- SaaS — the delivery model an unguaranteed dependency undermines
- Observability — the alerting that turns a composed SLA into a number you can watch
- AI & Machine Learning
References
- Common pitfalls when building generative AI applications (Chip Huyen)