Choosing Your AI Stack: Opinionated Defaults and the Trade-offs That Matter

There's a failure mode common to almost every AI stack decision we see: teams optimize the wrong layer. They spend weeks benchmarking models and picking a framework, then ship a system that breaks on retrieval quality, has no fallback when the provider has an outage, and can't be re-audited when they want to swap the model six months later. The decisions that determine whether your AI works aren't the ones the internet argues about.

This guide is how we would choose a stack: opinionated where experience justifies it, honest about trade-offs, organized around the layers that matter rather than the model of the month. For an interactive version tailored to your constraints, the Tech Stack Picker does this in about two minutes without a sales call.

First principle: the model is the swappable part

The most consequential decision is architectural, not vendor. Treat the model (hosted LLM, managed ML platform, or your own trained models) as a replaceable component behind an interface. The provider landscape moves monthly, and the "best" model for your task will be different by the time you've integrated the current one.

If your application code is entangled with one provider's SDK and prompt format, every model change is a migration. Behind a thin interface, it is a config change. That one decision buys you the freedom to chase model improvements, negotiate on price, and route different tasks to different models. It costs nearly nothing on day one and a rebuild later. Everything defensible in your system lives in the layer around the model, so make that the layer you invest in.

The layers, and where to spend your decision-making

An AI stack is not one choice; it's a set of layers, each with its own trade-offs.

The model layer

Pick a sensible default and move on; your differentiation does not live here. A strong general-purpose model as the default, with the freedom (thanks to the swappable interface above) to route specific tasks elsewhere: a cheaper model for classification, a specialized one for a narrow domain, a self-hosted one where data-residency or cost demands it. Skip the extended bake-off; invest in the interface that lets you change your mind.

Retrieval and the vector store

If your application answers from your own data, and most do, retrieval is where quality is won or lost. The vector store choice is per-project:

• pgvector (Postgres) when you already run Postgres and are under roughly 10M vectors. Keeping embeddings and source of truth in one database is a major operational simplification; our take is on the Supabase / Postgres page.
• Pinecone for large hosted workloads where you'd rather not operate the store.
• Weaviate when you want hybrid search built in.
• Qdrant when self-hosting and performance-per-dollar matter.

The stack decision that matters more than which store you pick: budget for hybrid search plus a re-ranking layer regardless; that is what separates RAG that works from RAG that demos. And treat the vector store as a cache with your source of truth behind it, never the system of record. The RAG systems page goes deep on this.

Orchestration

There's a strong pull toward a heavy framework (LangChain, LlamaIndex) on day one. Our stance: use them selectively and lean toward custom orchestration for the core paths. Frameworks are excellent for prototyping and standard patterns; they become a liability when your control flow gets specific and you're fighting the abstraction. Prototype with a framework if it accelerates you, but don't let it own your production control flow if that flow is where your product's logic lives.

Data and integration

The AI stack plugs into your existing data sources, auth, and downstream systems, and that integration surface is where AI projects stall. Before committing to a stack, get honest about what you are integrating with: rate limits, data freshness, auth models, and the systems that have to consume the AI's output. The Integration Readiness Checker names the real risks here before they become mid-build surprises.

Evaluation and observability

This is the layer teams skip and regret. AI quality degrades silently: a model swap, a prompt change, or a data-source update can each regress your system without warning, and without evaluation you find out from your users. Budget for an eval harness (ground-truth datasets, regression tests on critical paths) and production observability from the start. In stack terms: treat evals as part of the stack, not a phase-two nicety.

Where different application shapes push the stack

The right stack depends on what you're building. Retrieval-heavy apps (knowledge bases, document search, support assistants) stand or fall on the retrieval layer; see RAG systems. Agentic systems that take actions add orchestration, tool-execution safety, and state as first-class concerns, often using retrieval as one tool among many; see Agentic AI. Classic ML tasks (scoring, classification, forecasting, vision) push you toward trained models, feature pipelines, and drift monitoring rather than LLM orchestration, covered on AI/ML.

Most real products are a blend, which is another argument for the swappable-core principle: you'll route different task shapes to different models and want to change those routes over time. Browse how we think about each on the technologies overview.

A sequence that works

1. Design the swappable interface first. Decide where the model boundary is before you pick a model: the highest-leverage, cheapest decision in the whole stack.
2. Choose the retrieval layer against your real data. Pick a vector store for your scale and infra, and commit to hybrid search plus re-ranking.
3. Keep orchestration light until your control flow demands more. Prototype fast; own the core paths when they become the product.
4. Get integration reality on the table early. Rate limits, auth, and data freshness sink more AI projects than model choice does.
5. Treat evals and observability as part of the stack, not a later phase.

Two cross-cutting trade-offs are worth making deliberately: hosted vs. self-hosted (data-sensitivity and volume decide it; do not self-host for ideology) and speed vs. optionality (many "fast" choices lock you in, whereas the swappable-core interface buys both). If you're at the whiteboard stage, a Discovery Sprint is a good container for locking these decisions. If you've already built and suspect you optimized the wrong layer, a Tech Audit diagnoses the stack you shipped.

The through-line

Choosing an AI stack well means investing your decision-making where the outcomes live, not picking the winning model. Make the model swappable, spend your attention on retrieval and integration reality, keep orchestration light until it needs to be heavy, and treat evaluation as part of the stack. Do that, and the model-of-the-month churn becomes a config change instead of a crisis.