When a founder tells me they want to "add AI to the product," the first question I ask is: what do you mean by add? Because the word covers a lot of different operations, and most of them do not require touching the core of your system at all.
The thing that makes LLM integration tractable — or complicated — is usually not the model itself. It is the data pipeline around it.
What integration actually means
In practical terms, connecting an LLM to an existing stack usually involves three separate problems.
Getting the model access to relevant context. This means either passing structured data from your database as part of the prompt, or building some form of retrieval layer that fetches the right chunks at query time. Neither requires changing your database schema.
Getting the model's output back into your system. Sometimes this is a simple API call that returns a string. Sometimes the model needs to take actions — create a record, send a message, update a status. That second case requires some thought about permissions and error handling, but it is still just an API integration.
Deciding what the model should and should not do. This is the part teams underestimate. Prompting, guardrails, fallback behavior — this is where most of the design work actually lives.
The migration myth
I keep hearing "we need to refactor first" as a precondition for adding agents. Sometimes this is true. But more often it is a rationalization for delay, or a misunderstanding of what the integration actually touches.
We have built agents on top of monolith Rails apps from 2016, legacy Airtable setups, and HubSpot instances that had accumulated five years of field chaos. None of those required a rewrite before the agent could start doing useful work. They required some adapter code and a clear scope for what the agent was responsible for.
The adapter pattern is underused. You build a thin layer that translates between your existing data structures and the format the agent needs. The underlying system does not change. The agent sees a clean interface. Everyone is fine.
Where to start
Pick the narrowest possible first integration. One trigger, one data source, one action. Not a general-purpose assistant that can do anything — a specific agent that monitors one thing and does one thing when a condition is met.
The reason for this is not conservatism. It is that narrow scope makes evaluation possible. You can tell immediately whether the agent is working correctly. With a general-purpose agent, the surface area for failure is enormous and hard to instrument.
Once the narrow agent is running and trusted, you expand from there. Scope creep from the agent side is fine — scope creep from the requirements side, before the first agent is stable, is how projects fail.
Authentication and permissions
This comes up in almost every integration and is consistently handled late. If the agent is going to take actions on behalf of users — not just the system — you need a clear model for whose credentials it is using and what it is allowed to do.
The simplest approach is a dedicated service account with explicit, minimal permissions. The agent can only do what that account can do. Log everything. This is not exciting but it avoids a class of security and audit problems that are much harder to fix after the fact.
For EU companies, there is also a GDPR dimension: if the agent is processing personal data, that needs to be documented in your records of processing activities. Not something to sort out at launch.
When it gets complicated
Multi-step agents that depend on each other's outputs. Agents that need to maintain state across sessions. Agents that interact with users directly and need to handle ambiguous input gracefully. These are real engineering problems, not configuration work.
The honest answer is that the complexity curve is steep. Simple agents are genuinely simple. Complex agents require careful design, testing infrastructure, and ongoing maintenance. The jump between the two is not always obvious from the outside.
If you are scoping an integration and it feels like it will require state management, session continuity, or multi-agent coordination — budget accordingly. That is not a warning sign that the project is wrong. It is just a different category of work.
Most teams start with the simple category and get a lot of value from it. That is the right approach.