← Taku Uechi

What Changes When Your Application Depends on AI

By Taku Uechi

Service and Application Dependencies with AI

AI adds a new layer of dependencies to modern applications. Some of these are familiar—queues, databases, APIs, monitoring—but AI changes how important they become.

A normal API call is usually expected to be fast, predictable, and deterministic. AI doesn't always work that way. Responses can take seconds or even longer, outputs can vary between identical requests, providers can have issues, and human input can be... unpredictable.

Because of that, building AI into an application isn't just about calling a model. You have to think about the entire system around it.

AI Provider Outages and Availability

If an application depends on OpenAI, Anthropic, Google, or another model provider, that provider effectively becomes another production dependency.

We already deal with this problem with services like Stripe, Cloudflare, Elasticsearch, Redis, or any other external/internal service. The difference is that AI functionality can sometimes become deeply embedded into a workflow.

For example:

User Request  
    ↓
Application API  
    ↓
OpenAI  
    ↓
Structured Response  
    ↓
Application continues workflow

What happens if OpenAI takes 45 seconds to respond? What happens if it returns a 500? What happens if the provider is completely unavailable?

The application needs to be designed with that possibility in mind.

Depending on the use case, that could mean:

  • retrying the request
  • moving the work into a background job
  • allowing the user to try again later
  • falling back to another model
  • falling back to a non-AI workflow

I've worked on systems where Redis-backed jobs and distributed workers handled scheduled or long-running workloads. AI makes this type of architecture even more useful because you don't necessarily want a web request sitting open waiting for an AI provider.

AI should generally be treated like any other external service dependency: assume at some point it will be slow or unavailable.

Prompt Dependencies and Model Version Dependencies

Prompts are starting to look a lot more like application code.

A small change to a system prompt can completely change how an AI feature behaves.

For example:

Extract the following information from this document.

might behave very differently from:

Extract the following information from this document.  
Return only fields explicitly present in the document.  
Do not infer missing values.  
Return the result using the following JSON schema...

That prompt is effectively business logic.

It should probably be:

  • version controlled
  • tested
  • reviewed
  • deployed
  • monitored

There is another dependency hiding underneath the prompt: the model itself.

Changing from one model version to another isn't necessarily the equivalent of upgrading a normal library from 1.2.1 to 1.2.2.

The exact same prompt can produce different results.

In a traditional application:

input + code = predictable output

With AI it's closer to:

input + prompt + model + context ≈ expected output

And yes, AI models are generally non-deterministic. Even with the same input and prompt, you can receive different outputs. You can reduce that variability through model settings and better prompting, but you should still design the application assuming the output isn't guaranteed to be identical every time.

That makes model upgrades something that should be tested against real application scenarios rather than blindly swapped into production.

Data, Context, Tools, and API Dependencies

An AI model by itself usually doesn't know enough about your application.

It needs context.

That can come from:

  • MongoDB
  • Elasticsearch
  • vector databases
  • internal documentation
  • application APIs
  • external APIs
  • third-party services

I've built an internal RAG platform where documents went through ingestion and ETL pipelines, normalization, metadata enrichment, and source prioritization before being provided to the model.

At that point, AI quality isn't only dependent on the model.

It's dependent on the entire pipeline:

Documents  
    ↓
Ingestion / ETL  
    ↓
Normalization  
    ↓
Metadata  
    ↓
Search / Retrieval  
    ↓
Context  
    ↓
AI Model

If retrieval returns bad information, the AI probably produces a bad answer.

Garbage in, garbage out still applies. AI just makes the garbage sound more convincing.

Agents make this even more interesting because they can call other services.

For example, imagine an AI feature in a sports registration platform:

User:  
"Refund the registration for Team A."

AI  
 ↓
Find Registration API  
 ↓
Find Payment  
 ↓
Stripe API  
 ↓
Update Registration  
 ↓
Send Email

Now the AI depends on your database, APIs, authentication, Stripe, and potentially your email provider.

The model might be the most visible part of the feature, but there could be five or ten other dependencies behind it.

Latency Dependencies

AI is slow.

Or at least it's slow compared to what we've traditionally expected from application APIs.

Most application endpoints I've worked on are measured in milliseconds.

With AI, a response taking several seconds isn't unusual. Complex reasoning, large context windows, tool calls, or agent workflows can take significantly longer.

This creates architectural problems.

If your HTTP infrastructure has something like a 30-second timeout, an AI request that takes 35 seconds isn't just slow—it fails.

That's where infrastructure we've been using for years becomes important again:

API  
 ↓
Redis / Queue  
 ↓
Worker  
 ↓
AI Provider  
 ↓
Database  
 ↓
WebSocket / Polling  
 ↓
Frontend

I've used Redis-backed distributed job platforms for scheduled and concurrent workloads, and I've worked with Kafka for event-driven systems.

Those patterns map extremely well to AI.

Instead of:

POST /generate  
(wait...)
(wait...)
(wait...)
(wait...)
200 OK

you can do:

POST /generate  
202 Accepted

jobId: abc123

Then process the AI request asynchronously.

This is especially important once AI starts performing multiple steps:

retrieve data  
→ call model
→ call external API
→ call model again
→ validate result
→ save result

A request that starts as a simple AI call can very quickly become a workflow engine.

Cost as a Runtime Dependency

This is one of the stranger differences between traditional application development and AI development.

Every request can have a meaningful variable cost.

If an API endpoint suddenly gets 100x more traffic, traditionally I'm mostly thinking about:

  • CPU
  • memory
  • database load
  • network traffic
  • autoscaling

With AI, there's another line item:

tokens × requests × model cost

Imagine a feature receiving 10,000 requests.

If each request sends a huge amount of context to an expensive model, that feature can technically work perfectly while still being a production problem.

This means cost becomes part of application architecture.

You may need to decide:

  • Does this request actually need the most powerful model?
  • Can we use a smaller model?
  • Can the result be cached?
  • Can we reduce the context?
  • Did we already generate this answer?
  • Should this user be allowed to make this request 1,000 times?

Redis caching becomes interesting here as well.

I've used Redis heavily for improving API performance and reducing repeated computation. With AI, caching can reduce latency and directly reduce inference cost.

The economics become part of the engineering.

Fallback and Deterministic Systems

AI is useful, but I wouldn't make every critical decision depend entirely on it.

There are plenty of cases where deterministic code is still the better option.

For example, imagine using AI to extract registration information:

{
  "playerName": "John Smith",
  "division": "Mens 4.0",
  "payment": 150
}

The AI can help extract the information.

But I probably don't want the model deciding whether $150 is actually the correct registration amount.

That should come from the application's business logic.

A better architecture is:

AI extracts data  
       ↓
Application validates data  
       ↓
Business rules verify data  
       ↓
Application performs action

AI proposes.

The application verifies.

This becomes especially important when dealing with things like:

  • payments
  • permissions
  • authentication
  • inventory
  • financial calculations
  • destructive actions

I've worked on systems involving Stripe payments, authorization and ACLs, and event-driven business workflows. Those are places where deterministic application logic should still be the source of truth.

AI can make the workflow better, but it shouldn't necessarily own the rules.

And then there's human input.

Human input can be... creative.

Users will misspell things, paste entire emails into a field, give contradictory instructions, upload the wrong document, try to manipulate prompts, or ask the AI to do something completely outside of what you expected.

The application has to assume that input won't always look like the happy-path examples from development.

Observability and AI-Specific Monitoring

Traditional monitoring usually asks questions like:

Did the request succeed?  
How long did it take?  
Did it throw an exception?  
What was the HTTP status?

I've used tools like Sentry and built systems around application monitoring, job processing, API performance, and production errors.

Those metrics are still important with AI.

But now we need another layer.

The request can return:

HTTP 200 OK

and still be completely wrong.

That's a weird problem.

From the application's perspective everything worked:

OpenAI request: 200  
Latency: 2.4 seconds  
Database write: success  
No exceptions

But the AI response might be:

The tournament is scheduled for March 17.

when the actual date is March 18.

Operationally, everything is green.

Functionally, the feature failed.

So AI observability needs to look at more than traditional infrastructure metrics.

You may want to track:

  • model used
  • prompt version
  • token usage
  • latency
  • cost
  • tool calls
  • retries
  • failed structured outputs
  • validation failures
  • user corrections
  • evaluation scores

This is where evaluations become extremely important.

Traditional monitoring tells us:

Did the system work?

AI monitoring also needs to tell us:

Did the system give a good answer?

That distinction is probably one of the biggest changes AI introduces to application reliability.

The interesting thing is that most of the infrastructure needed to build reliable AI applications isn't really new.

Queues, workers, Redis, Kafka, APIs, caching, validation, observability, rate limiting, retries, circuit breakers, and fallback systems have been around for years.

AI just makes a lot of them more important.

The model might be the exciting part of the architecture.

But in production, everything surrounding the model is usually what determines whether the feature is actually reliable.

Comments powered by Disqus