← Taku Uechi

When SaaS Becomes a Black Box for AI

By Taku Uechi

I've been thinking about another problem with AI coding tools that I don't hear talked about as much. A lot of modern applications don't really live entirely inside the codebase anymore.

Over the last 10+ years we've moved more and more functionality into managed platforms. Authentication goes into Clerk or Auth0. Payments go into Stripe. Search goes into Algolia. Commerce goes into Shopify. Feature flags go into LaunchDarkly.

From an engineering perspective this usually makes a lot of sense. I would much rather use Stripe than build a payment platform from scratch, and I definitely don't want to build authentication, password resets, MFA, OAuth and session management unless I absolutely have to.

The problem is that AI coding tools mostly understand what they can see, and a lot of the actual application is no longer in the repo.

Managed Application Platforms

I think managed application platforms is probably the best way to describe this category. They're more than just normal SaaS products because they're often part of the runtime behavior of your application. They own application state, configuration and sometimes a pretty significant amount of business logic.

Some obvious examples are:

Clerk  
Shopify  
Stripe  
Auth0 / Okta  
LaunchDarkly  
Algolia  
Contentful  
Sanity  
Salesforce  
ServiceNow

The application usually interacts with them through an SDK or API, but the SDK only shows part of what's actually happening.

You might have this in your code:

const user = await currentUser();

The AI can understand that line pretty easily. What it can't necessarily see is everything configured behind it:

organization roles  
permissions  
SSO configuration  
session settings  
MFA requirements  
dashboard settings

That configuration still affects the application. It just doesn't live in Git.

Clerk

Clerk is probably one of the clearest examples I've run into personally.

Your application might contain something like:

has({ permission: "org:admin" })

From the code this looks pretty simple. If an admin suddenly can't access something, an AI coding tool might search the repository, inspect the authorization check and conclude that the code looks correct.

The real problem could be that the permission was never assigned to the role in Clerk.

There may be absolutely nothing wrong with the application code. A human engineer who's been working on the system for a while probably knows to check Clerk. The AI might spend a bunch of time trying to "fix" code that isn't broken because that external configuration isn't part of the context it has.

Shopify

Shopify has the same problem but at a much larger scale. I've worked with Shopify where a lot of the behavior wasn't really owned by the React application at all.

Products, metafields, discounts, checkout behavior, themes, extensions, merchant configuration and installed apps can all change what happens.

The application might simply do:

const product = await shopify.getProduct(id);

but the behavior behind that product could depend on configuration that never appears anywhere in the repository.

This gets especially interesting when something breaks in production. The code didn't change, but Shopify configuration did.

If the AI only has access to Git, its view of the problem is incomplete before it even starts debugging.

Stripe

Stripe is another obvious one because a lot of payment logic looks like normal application logic from the outside:

await stripe.paymentIntents.create(...)

But Stripe owns a huge amount of state:

customers  
products  
prices  
subscriptions  
connected accounts  
invoices  
tax configuration  
webhook configuration  
payment methods

I've worked with Stripe Connect where the state of the connected account mattered just as much as anything happening in our own database. An AI looking at the code might see a connected account ID and understand that we're sending money somewhere, but it doesn't know who actually owns that account, whether payouts are enabled, what bank account is attached or what configuration was changed in the Stripe Dashboard.

Again, part of the application is outside the application.

Auth0 and Okta

Auth0 and Okta have the same issue as Clerk, especially once you get into larger organizations. You might have application code that validates a token and checks some claims, while the actual behavior depends on:

roles  
groups  
identity providers  
SSO  
MFA rules  
tenant configuration  
custom actions  
claims

A lot of this can change without a deployment, which I think is one of the more interesting parts of the problem.

We've always thought of source control as the history of how the application changed. That's not completely true anymore. The application can change while Git stays exactly the same.

LaunchDarkly

LaunchDarkly might actually be the purest example.

Your code could contain:

if (flags.newCheckout) {  
  return <NewCheckout />;
}

return <OldCheckout />;

The AI sees two code paths, but which one is production actually using?

Maybe both.

The answer could depend on:

environment  
user ID  
organization  
percentage rollout  
segment  
prerequisite flag  
manual override

None of that logic necessarily exists in your repository.

You could ask an AI:

Why does Customer A see the new checkout but Customer B doesn't?

and it could spend a lot of time analyzing React when the actual answer is just a LaunchDarkly targeting rule.

That's a pretty large blind spot.

Algolia

Search is another place where a surprising amount of behavior can live outside the code. I've worked with Algolia where the frontend really only knows how to send a query and display the results.

The actual search behavior can depend on:

ranking  
searchable attributes  
facets  
synonyms  
replicas  
merchandising  
index configuration

Suppose somebody says:

Search results are bad for "black dress."

The AI opens the frontend and finds:

index.search("black dress");

There's not much to fix there.

The problem could be that somebody changed ranking configuration or synonyms inside Algolia. A human who knows the application understands that search isn't just code. The AI has to somehow be given that context.

Contentful and Sanity

CMS platforms create a similar problem, although in a slightly different way.

Your React application might know about:

article.title  
article.heroImage  
article.body

but the actual content model, published entries, references, localization and editorial state live somewhere else.

This isn't necessarily traditional business logic, but it still affects application behavior. An AI can look at the component and decide that heroImage should always exist because the TypeScript interface says it's required, while an editor may have created 300 older articles before that field existed.

The production data is telling a different story than the code.

Sanity is a little better here because more of its schema can be represented as code, but the actual content and relationships still exist outside the application repository.

Salesforce and ServiceNow

Salesforce is where this problem can get pretty extreme. I've seen enterprise applications where Salesforce isn't just a database. It contains a huge amount of business behavior.

Things like:

validation rules  
flows  
approval processes  
permissions  
custom objects  
automation  
workflow rules

Imagine your application does this:

await salesforce.updateCustomer(customer);

Then something unexpected happens.

Maybe an approval process starts. Maybe another record gets updated. Maybe an email gets sent. Maybe the update is rejected because of a validation rule.

None of that behavior has to exist in your application's repository. You could give the AI every line of your code and it still wouldn't understand the complete workflow.

ServiceNow is very similar. A lot of enterprise workflows can live almost completely inside the platform through forms, business rules, approvals, permissions, integrations and automation. From the application side you might just see an API call, while behind that call there could be years of business logic.

This is probably where the term "black box" starts feeling pretty accurate.

The Repository Isn't the Application Anymore

This is the part I find most interesting.

For a lot of modern applications, the real system looks closer to:

Application Repository  
        +
MongoDB  
        +
Clerk  
        +
Stripe  
        +
Shopify  
        +
Algolia  
        +
Feature Flags  
        +
CMS  
        +
Production Configuration  
        =
The Real Application

But an AI coding agent might only see:

Application Repository

Maybe the database schema too, if you're lucky.

That's a huge difference in context, and AI is extremely dependent on context.

I think this also changes the idea of "give the AI the whole repo." Even if you give it every file in Git, you still may not be giving it the whole application.

Humans Have Institutional Context

A senior engineer working on an application for a few years picks up a lot of information that never gets written down.

You start knowing things like:

That permission is configured in Clerk.

That product price comes from Stripe.

Don't change that field, Shopify owns it.

That search issue is probably Algolia.

That customer is on a feature flag override.

None of those things are necessarily obvious from reading the code.

When another human joins the team, that knowledge gets transferred slowly through pull requests, Slack messages, documentation and somebody eventually saying, "oh yeah, that's weird because..."

AI doesn't really get that same onboarding. Every new coding session can effectively start with:

Here's the repository. Figure it out.

That's probably fine when most of the application lives in the repository. It's much harder when half the application lives in dashboards.

Vendor Lock-In vs Context Lock-Out

The traditional criticism of these platforms has always been vendor lock-in. You build heavily around Shopify and now moving away from Shopify is difficult. You build around Stripe and now Stripe is deeply embedded in your payment infrastructure.

That's still true.

But I think AI introduces another problem that I would call context lock-out.

Vendor lock-in is basically:

It's difficult to move away from this platform.

Context lock-out is:

It's difficult for an AI agent to understand the application because important parts of the application live inside this platform.

The application can still work perfectly fine and the developer can still understand it. The problem is that the AI has an incomplete model of the system.

I think that's becoming a real architectural cost.

This Doesn't Mean We Should Stop Using SaaS

I'm definitely not suggesting we start rebuilding all of this ourselves. That would probably be much worse.

I don't want our authentication implementation sitting in 30,000 lines of custom code just so an AI can read it. The benefits of these platforms are still huge:

less code  
faster development  
better security  
specialized infrastructure  
compliance  
scalability  
reliability

And generally fewer things we have to maintain.

The tradeoff is that we're moving application knowledge somewhere else. That was already a problem for developers, but AI makes the cost much more obvious because the quality of the output depends so much on the context the model actually has access to.

Maybe These Platforms Need to Become AI-Native Too

I don't think the obvious solution is putting everything back into the repository. It's probably giving AI access to more of the external context.

Imagine an AI coding agent being able to inspect:

Clerk roles and permissions  
Stripe products and subscriptions  
Shopify metafields and configuration  
LaunchDarkly targeting rules  
Algolia index configuration  
Contentful schemas  
Salesforce flows

Now when you ask:

Why doesn't this user have access?

the AI could inspect both:

application code  
+
Clerk configuration

That's a much more useful debugging environment.

Some of this is already possible through APIs, MCP servers, CLIs and other integrations, and I think that's going to become increasingly important.

It's probably not enough for AI coding tools to understand the repo.

Eventually they need to understand the application environment.

The Hidden Complexity Never Really Went Away

For years we've been moving functionality out of our applications and into managed platforms. For the most part, I still think that was the right decision.

We write less code, ship faster and let companies that specialize in authentication, payments, commerce or search handle the difficult parts.

AI changes the tradeoff a little.

The better these coding tools get, the more valuable complete context becomes. A modern application's context isn't necessarily in Git anymore. It's spread across the repository, database and probably five or ten different SaaS dashboards.

We've spent years making applications easier to build by hiding complexity behind APIs. AI is starting to make something else obvious: the complexity may be hidden, but it was still part of the application the whole time.

Comments powered by Disqus