In the past quarter, we gave our AI agent access to the app thousands of travel advisors use every day. The agent reaches into dozens of the product's domains through roughly 90 tools. Advisors ask it, in plain language, to build a trip, draft an email, price an itinerary, add activities, set up a cruise, or update a contact.
Throughout June 2026, the agent handled more than 40,000 chats and exchanged over 1 million messages. Along the way, it performed more than 139,000 accepted actions on advisors' behalf and processed over 15 billion tokens.
This post is about that agent specifically, though AI runs through far more of Tern: parsing commission statements from any supplier, multi-pass itinerary and activity parsing, importing through a Chrome extension, and more.
This isn't an architecture we designed up front and then executed. Most of the patterns below only became concrete by building them.
Our starting bets
We couldn't prove any of these going in. They were opinions we chose to build with, and so far they've held up.
Give the agent the app. It works inside the app we already had: the same ActiveRecord models, the same services, the same permissions.
Tool design is most of the game. The tools are the interface between the model and our domain: a tool's name, description, and parameter schema are literally what the model reads to decide what to do. The tool descriptions and params are the prompt.
Prompt less. It's tempting to write a giant system prompt that explains everything. Resist it. If a tool needs a paragraph of system-prompt instructions to be usable, that's a signal the tool's params or description need work, not that the prompt needs more words. Keep the system prompt as light framing and push the specifics into the tools.
Bet on the model. Don't over-engineer around it. Let the model make the decisions and give it good tools, rather than encoding all the control flow in a heavy harness. Models keep getting better and absorb more of what scaffolding used to do, so anything built to compensate for today's limitations is tomorrow's dead weight. Build for the model eating the harness.
Why we stayed in the monolith
The obvious counter-move is to build AI features as a separate service in a more "AI-native" stack: Python, a framework like LangChain. We seriously considered it, and decided against it.
The problem with splitting it out is the boundary. A separate app immediately needs an API between it and the main Rails app. Does it talk to the same database? Even if it does, it still doesn't have our ActiveRecord models, so it loses all the validation and business logic built around them. That leaves two bad options: rebuild that logic in the other stack, or build a thicket of two-way API contracts back to Rails, piling network overhead and complexity on top of the actual agent work.
So we kept it in the monolith, but RubyLLM wasn't where we started. We began fully hand-rolled, which worked fine for simple, one-shot calls to the model. Once we needed more, especially persisting multi-turn conversations, hand-rolling wasn't enough, so we tried a gem built for exactly that.
It turned out to be too opinionated: it reached too far up the abstraction stack, got obtrusive to work with, and reinvented ideas that didn't need reinventing. RubyLLM is where we landed instead. It sits closer to the right level: it handles the provider abstraction, gives us one shared API across models, and even carries some opinions on how conversations get persisted, but it stops well short of dictating how the rest of our stack has to look, so we still get to build what we need on top to best serve our advisors.
A tool call is a controller action; the only difference is that the agent is the caller instead of an advisor clicking through the UI. Same entry point, different caller. So the same discipline applies: keep tools (like controllers) thin, push the real work into shared service objects, and let both the web and the agent use the exact same business logic. Even authorization is the same: our tools run the same Pundit policies the controllers do. An agent calling a tool goes through the same checks an advisor would face when hitting the equivalent action.
Is Ruby as AI-native as Python? No. But one codebase means every engineer can more easily contribute to the agent, and our own AI dev tools understand it far better than they would a codebase split off. And we reuse the ActiveRecord models, the services, and the authorization we already had instead of re-plumbing them across a boundary. That reuse is the payoff that outweighs Ruby not being the "native" choice.
Speaking the advisor's language
The highest-leverage decision in tool design is naming.
A tool's parameters don't have to mirror the data model, and usually shouldn't. We name params (and shape the JSON a tool returns) to match the copy advisors actually see in the UI. That way, when the model asks a question or summarizes something back, it speaks in the words the advisor already knows. Map the raw data model straight through instead and the result is constant miscommunication: the model naming fields and entities the advisor has never seen. The tool schema is a translation layer into advisor language, not a mirror of the schema.
This also dissolves an apparent contradiction. The common advice is "fewer, larger tools," and we believe it. Yet we have roughly 90 tools and want more. Both are true because they're different axes. Fewer/bigger is about not fragmenting a single capability into five tiny tools (one capable create_activity, not a scatter of narrow ones). The count grows because the agent keeps gaining genuinely distinct capabilities across the product. The rule isn't "few tools total." It's "each tool as coarse as makes sense." The count still grows, because coverage does.
Prompt less
The "prompt less" instinct shows up most clearly in a common pattern, often called skills. A skill looks like a tool, but instead of performing an action it returns more prompt: a chunk of workflow-specific instructions the model then reads and follows. Most instructions only matter for one kind of task; loading all of them into the system prompt taxes every chat with tokens it doesn't need. So we lazy-load: the workflow-specific guidance lives behind a tool, and the model only pays for it when it actually reaches for that workflow.
The counterintuitive part of "prompt less" is what to do when the model misbehaves. The reflex is to add an instruction telling it not to. Often the real cause is that we were already over-prompting, sometimes somewhere far away. For example: we added a skill for writing in a particular voice, and inside it told the model to add no rich formatting, because we wanted plain text. Later, advisors drafting emails with links or bold found their formatting silently stripped. The reported bug was "emails can't send links." The actual cause was that stray "no formatting" instruction in the voice skill: the same skill gets reused across many contexts, and in this one, drafting an email with links and bold, it was simply wrong. It killed our markdown-to-rich-text conversion downstream. The intuitive fix looked like adding an instruction to the email tool telling it to preserve links. The actual fix was to delete the offending line from the voice skill's prompt: the cause wasn't near where the bug showed up.
The same instinct scales up from instructions to knowledge. We gave the agent a tool to search and read our help center. Crucially, we treat the help center as the source of truth rather than baking that knowledge into the agent. The prompt shouldn't hold what a help article already says; the article should, and the agent should reference it. It's a better abstraction for reasons that have nothing to do with AI: the same content serves advisors who browse the help center directly, our agent, and whatever other tools or AI sources come later: one canonical source, many consumers. And it keeps us from taxing every turn with knowledge the agent can simply look up when it needs it.
Less isn't always more. Before reaching for more prompting, the first move should be questioning whether something can be removed or simplified, or whether the real cause is prompting that's already wrong somewhere else. Sometimes, once that's ruled out, the model genuinely needs the guardrail.
Searching, not embedding
How does the agent find things? A common answer right now is RAG (Retrieval-Augmented Generation): embed everything into a vector index and let similarity search surface it. We doubled down on query-only instead. We emulated how Claude Code navigates a codebase with grep. We did not embed our data. All of our search tools are plain text. The bet is the same one from the top of this post: models are genuinely good at querying and finding, and staying query-based lets us skip the complexity of building and maintaining an embedding pipeline.
If the agent searches in plain text, our free-text search has to be good, and good free-text search is exactly what advisors already need, because it's how they use the product themselves. So, we invested in the search everyone was already using: a de-normalized, asynchronously-maintained search index. One investment, two consumers: the same reuse story, applied to search. There's still plenty of headroom here, but the shape is right: lean on what the model is already good at, and make the thing we had to build serve the humans too.
The honest wrinkle is that plain text can't always carry the meaning. Take the photos advisors upload to our media library: a photo has a filename and maybe a location or a supplier, and that's it. Even fully de-normalized, the odds that a picture of a plane contains the word "plane" anywhere in its metadata are basically zero. Here's a case where we do reach for a model instead of plain text: we run each photo through a small, fast tagging model on upload, producing a list of tags: plane, waterfall, beach, whatever's in the frame. Those tags become searchable text like everything else, so when an advisor (or the agent) looks for a beach photo, free-text search can actually find it.
Human-in-the-loop for writes
Reading changes nothing. Writing has consequences. So for any mutating tool call, we stop and require the advisor to confirm before it runs. The agent never silently creates or updates records. That's a deliberate product decision, human-in-the-loop for writes, and it's honestly a fair amount of engineering complexity: a whole prepare / confirm / accept machinery around what would otherwise be a single call.
The prepare step earns its keep by validating beyond the schema. Param schemas check types and shapes: an integer, an enum, a JSON structure. They don't check it against the world: does the advisor have permission, does this foreign key resolve, does the referenced record even exist. Prepare runs that heavier validation, and when it can prove up front that a call will fail, it fails it right there, so we never surface a confirmation card that would only fail after the advisor accepts it. And when every action in a turn pre-fails, there's nothing to confirm, so we don't stop and strand a row of dead cards; we let the turn continue.
What makes that continuation work is self-healing errors, which deserve their own rule: never return a generic error to the model. No "invalid input." Say exactly what was wrong and why, in the exact words from the UI. Then the error serves two audiences at once: the model, which can often correct itself and re-call the tool with no human round-trip; and the advisor, for the cases the model can't fix, who gets a clear explanation in language that matches the screen instead of a leaked internal error code.
Seeing what the advisor sees
A lot of good agent UX is just not making the model go rediscover what it can already see.
If an advisor is looking at a trip and starts a chat, the agent should see what they're looking at rather than go rediscover it through search and read tool calls. So when a chat starts, we infer the default entities from the page the advisor is on and hand the agent exactly what those reads would have returned. Open a chat on an activity and it already has the activity, its itinerary, and its trip: the same context the advisor is looking at, without a single tool call. That also happens to be faster, cheaper, and less error-prone, but the real point is that the agent starts from the same view the advisor already has.
Our ask-a-question tool came from the same instinct. We noticed what the model already does naturally and made it first-class. Left alone, the model would print options 1/2/3 or end a turn with "what would you like me to do?", forcing the advisor to type prose back. We gave it a dedicated tool to ask, turning a wall of text into proper selectable options. We borrowed a lot of this from proven patterns across the broader AI-product space rather than reinventing UX from scratch.
Making it durable and observable
An agent turn is a long-running, IO-bound task, and we run it as a background job: today on Sidekiq, though fibers are the better long-term fit for work that's mostly waiting on the model. Running it as a job creates one sharp problem: deploys kill jobs. On our platform a deploy gives a job about 25 seconds to wind down, and an agent turn can easily run past that, so an in-flight chat can get killed mid-turn.
But that forced us to build something we needed anyway. A turn can die for any reason (a crash, a timeout, a deploy) and leave the chat half-finished, which the model provider rejects on the next request, effectively bricking the conversation. So we repair interrupted turns on the way back in. Crash recovery isn't an edge case here; it's table stakes.
Running this in production surfaces two other realities worth naming: cost and visibility. Cost shows up first in the prompt itself: we split the system prompt into a static half (all the tool definitions and fixed instructions) and a dynamic half (the date, the advisor's context), in that order, so the static half caches and is shared across every advisor instead of each one paying to write their own cache. Visibility is the other half of being observable. We tried third-party tools first, but in Ruby the integration was tedious and, worse, they were blind to our domain data, so the traces were incomplete. We built our own: open any single chat and trace exactly what happened, with token and cost accounting per message and thumbs feedback that closes the loop back into improvement. Building it ourselves was cheaper than it used to be thanks to today's AI tooling, and it fits our data exactly.
Evals as the loop
We evaluate in two layers. The cheap layer is basically unit tests for tool plumbing: given this prompt, does the agent call the right tool at all? It confirms the wiring works: the model sees the tool and picks it when it should. The expensive layer, still being built out, grades full chat transcripts against a rubric: the quality of the whole conversation, not just whether one tool fired.
We're early here, with a lot more still to build.
Where we're headed
Where we're headed builds directly on the groundwork above. We want to lazy-load tool definitions the way we already lazy-load skills, so the model isn't shown every tool on every turn: pure token savings, and it matters more the more tools we have. We want to run more than one model, across providers, which means decoupling from any single model and then leaning on trusted evals to prove a faster or cheaper model doesn't cost us quality. And we're looking forward to RubyLLM v2, whose more complete API should let us delete a few of the workarounds we've had to build on the current version.
The agent gets the same tools and access the advisor already has, inside the same app. That also kept the work accessible to every engineer. The agent is a new caller into the app we already had, and most of what made it work was refusing to rebuild what already existed.