Latency belongs in a job
Model calls are slow and flaky, which makes them the worst thing to do inside a request.
Workloads
The model call is the easy part. The rest is an application.
An AI feature is a queue, a vector index, a cache, a rate limit, a cost ledger, and a place to put the transcript. Stacks gives you all of those already wired to your models, so the interesting work is the prompt and the product rather than the plumbing around them.
What this always turns out to involve.
The work that arrives whether you planned for it or not.
Retrieval needs your data
RAG is only as good as the pipeline that keeps embeddings current with the rows they describe.
Cost is invisible until it is not
Tokens, retries, and runaway loops need metering and limits from the first release.
Providers change
Being locked to one vendor SDK makes evaluating a cheaper or better model a rewrite.
What is already in the box.
All of it in the same install, typed against the rest of your application.
One driver interface
Anthropic, OpenAI, and Ollama share one chat API, chosen in config/ai.ts, with Bedrock alongside as its own driver for AWS-hosted models.
Embeddings and RAG
Generate embeddings from model changes and query an in-memory vector index for retrieval, using the same events as everything else.
Queued generation
Long calls run as jobs with retries and timeouts, and results stream back over realtime channels.
MCP client
Talk to Model Context Protocol servers from the application, so tools and data sources are configuration.
Caching and limits
Cache-aside for repeated prompts, rate limits per user or team, and usage recorded like any other model.
Vision and images
Image analysis runs on Claude or GPT and image generation runs through DALL-E, both part of the same package rather than a second integration.
The layers it leans on.
Every one of these ships in the same install. Follow one to see what it covers.
From the command line.
Buddy scaffolds, runs, and ships this the same way it does the rest of the app.
buddy make:job SummariseTicket
buddy queue:work --queue=ai
buddy dev
Next door to this.
Projects of this shape usually grow into one of these.