What Changes When Your Application Depends on AI
I've been thinking about some of the application dependencies that become more important once AI is introduced into an application. A lot of the infrastructure isn't actually new. We already use Redis, queues, workers, Kafka, databases, third party APIs, caching and monitoring in normal applications. AI just creates some interesting cases where you start depending on these much more heavily.
One major difference is that we're used to application code being mostly deterministic. Given the same inputs we generally expect our code to do the same thing. AI isn't really like that. It can be slow, responses can be different for the same input and sometimes it can give you something completely unintended.
And of course human input can be... creative. If you give people a textbox they'll eventually paste things into it you never expected.
Here are some of the dependencies I've been running into or thinking about while building AI functionality.
AI Provider Outages and Availability
Once you integrate OpenAI, Anthropic or another provider into a workflow, it's basically another external service that can go down. We've dealt with this forever with Stripe, Cloudflare, Elasticsearch and pretty much any service we don't completely control, but with AI there's a tendency to make the model part of the main application flow.
For example:
User Request
↓
Application API
↓
OpenAI
↓
Parse Response
↓
Continue Workflow
This works fine until OpenAI takes 45 seconds to return or returns a 500. If the application has a 30 second HTTP timeout then we now have a failed request even though technically the AI provider may eventually complete the work.
For anything that can take awhile, I've been leaning toward treating AI work more like a job than a normal request. I've built systems using Redis backed queues and workers for long running or scheduled work and the exact same pattern applies pretty naturally here. Put the work into a queue, let a worker process it and either poll for the result or push it back through a WebSocket when completed.
Retries also have to be handled carefully. Retrying a normal GET request is pretty harmless, but if the AI is calling another service that creates a payment, sends an email or modifies data then retrying the entire workflow can potentially execute the action twice. At that point things like idempotency become just as important as the AI call itself.
Prompts and Model Versions
Prompts are starting to feel like another form of application code. A pretty small change to a system prompt can completely change what comes back from the model.
For example:
Extract the information from this document.
and:
Extract the information from this document.
Only return information explicitly found in the document.
Do not guess missing values.
Return the result using this JSON schema...
are technically asking for the same thing but can have very different results.
Because of this I think prompts should be kept in source control and changes should be tested similar to other application changes. I probably wouldn't bury an important production prompt as a random string inside a React component anymore.
The model version becomes part of this dependency too. You can have the same application code and the same prompt but get different behavior after changing models. This makes upgrading the model a little different from upgrading most libraries.
Traditionally we expect something close to:
input + code = output
With an AI feature it's more like:
input + prompt + model + context = probably the output we want
AI models are non-deterministic, although there are settings that can make the output more consistent. The important thing from an application perspective is that I wouldn't assume the same prompt and input will always produce exactly the same response. If I'm changing a model in production I want to run our real use cases against it first.
Data, Context and Other APIs
The model is only one part of most useful AI applications. Unless you're asking general questions, it needs some way to know about your application's data.
I've worked on a RAG platform where documents went through ingestion, ETL, normalization, metadata enrichment and source prioritization before the information ever got to the model.
Something roughly like:
Documents
↓
Ingestion / ETL
↓
Normalize + Metadata
↓
Search / Retrieval
↓
Context
↓
AI
Once you build something like this it becomes pretty obvious that improving the model doesn't automatically improve the application. If search pulls the wrong document or your ingestion process has bad metadata, the AI is working with bad information.
Garbage in, garbage out still applies. AI just makes the garbage sound more convincing.
This gets more complicated with agents because now the model can call other APIs as part of the workflow. Taking a sports registration application as an example, imagine someone asking:
"Refund the registration for Team A"
The actual operation could become:
Find Team
↓
Find Registration
↓
Find Payment
↓
Stripe
↓
Update Registration
↓
Send Confirmation
At that point OpenAI is only one dependency. MongoDB, authentication, Stripe, your own APIs and whatever you're using for email all have to work correctly too.
I would also be very careful about what decisions the model is actually allowed to make. Finding a payment is one thing. Deciding whether the user has permission to refund it or how much they're allowed to refund should still come from the application.
Latency
AI can be slow. Most normal API endpoints I've worked with are measured in milliseconds and if one consistently takes multiple seconds I would normally consider that a performance problem. With an AI request multiple seconds can be completely normal, and once you add retrieval and tool calls the request can easily get much longer.
This is where queues, Redis, Kafka and workers become much more useful.
Instead of keeping this open:
POST /generate
...wait...
...wait...
200 OK
we can do something closer to:
POST /generate
202 Accepted
jobId: abc123
and let the worker handle:
retrieve context
→ AI
→ call API
→ AI again
→ validate
→ save
I've built distributed job systems using Redis and have also used Kafka for event driven systems, and AI workloads feel like a pretty natural extension of those patterns. Once an AI operation starts making multiple calls it's really not just a request anymore, it's a workflow.
There are of course cases where keeping the request open makes sense. Chat is an obvious one since the response can be streamed and the user expects to see it generated. I just wouldn't design every AI operation like chat.
Cost Is Part of the Application
Cost is another dependency that I didn't really think about in the same way before AI.
If one of my traditional APIs suddenly gets 100x the traffic, I'm looking at CPU, memory, database usage, bandwidth and potentially adding more servers. With AI every request can also have a direct inference cost.
Something as simple as sending unnecessary context can get expensive:
requests × tokens × model cost
If a feature is called 10,000 times and we're sending a huge document to the most expensive model every time, the feature can technically be working perfectly while still being a production problem.
This is one place where some pretty boring application optimizations become valuable again. Cache responses when it makes sense, don't send context you don't need, rate limit expensive functionality and don't use the most expensive model when a smaller model can do the job.
I've used Redis for years primarily to improve application and API performance. With AI the same cache can potentially improve response time and actually save money at the same time, which makes the tradeoff a little easier to justify.
Keep Deterministic Code Where It Matters
Even if AI is involved in a workflow I still want normal application code to own the important business rules.
For example, AI could extract:
{
"playerName": "John Smith",
"division": "Mens 4.0",
"payment": 150
}
from a registration form or uploaded document. That's a great use case because extracting inconsistent human input is something AI is good at.
I don't want the model deciding whether $150 is the correct price though. The registration system already knows the price and that should remain the source of truth.
I would handle it more like:
AI extracts information
↓
Validate schema
↓
Load application data
↓
Apply business rules
↓
Perform action
This is especially important around payments, permissions, authentication and destructive operations. I've worked with Stripe payments and ACL/permission systems and I wouldn't replace those rules with a prompt asking the model if a user is allowed to do something.
There's also another dependency here that isn't technical: the user.
Users misspell things, upload the wrong file, paste an entire email into an input, provide contradictory instructions and sometimes ask the AI to do something completely unrelated to the feature. With a normal form we can control most inputs with selects, checkboxes and validation. Once we give someone natural language input the number of possible inputs basically explodes.
The backend still needs to assume the input is wrong until it's validated.
Monitoring AI
Normal application monitoring tells me a lot about whether the application is healthy. I've used Sentry for production errors and have built monitoring around APIs, background jobs and application performance.
AI adds an unusual problem because everything can technically succeed and the feature can still fail.
For example:
OpenAI: 200
Latency: 2.4 seconds
Database: success
Exceptions: 0
Everything is green, but if the model tells the user the tournament is March 17 when it's actually March 18, the feature didn't work.
So on top of the normal metrics I think we need to start storing things like which model was used, prompt version, token usage, cost, latency, retries and validation failures. For important AI features it also makes sense to have a set of examples where we know what a good result looks like and run those again when prompts or models change.
We've always monitored whether a service is up. With AI we also need some way of determining whether it's actually doing a good job.
Conclusion
The funny part is most of the architecture around reliable AI applications isn't really new. Queues, Redis, Kafka, workers, retries, caching, rate limiting, monitoring, validation and fallback logic are all things we've already been using.
What's different is the thing in the middle is slower, non-deterministic, can be relatively expensive and accepts almost unlimited forms of user input. That puts more pressure on all of the boring infrastructure around it.
The AI model gets most of the attention, but I'm starting to think the reliability of an AI feature is going to depend much more on everything we build around the model.