When SaaS Becomes a Black Box for AI
I've been thinking about another problem with AI coding tools that I don't hear talked about as much.
A lot of modern applications don't really live entirely inside the codebase anymore.
Over the last 10+ years we've moved more and more functionality into managed platforms. Authentication goes into Clerk or Auth0. Payments go into Stripe. Search goes into Algolia. Commerce goes into Shopify. Feature flags go into LaunchDarkly.
From an engineering perspective this usually makes a lot of sense. I would much rather use Stripe than build a payment platform from scratch, and I definitely don't want to build authentication, password resets, MFA, OAuth and session management unless I absolutely have to.
The problem is that AI coding tools mostly understand what they can see.
And a lot of the actual application is no longer in the repo.
Managed Application Platforms
I think managed application platforms is probably the best way to describe this category.
They're more than just normal SaaS products because they're often part of the runtime behavior of your application. They own application state, configuration and sometimes a pretty significant amount of business logic.
Some examples:
Clerk
Shopify
Stripe
Auth0 / Okta
LaunchDarkly
Algolia
Contentful
Sanity
Salesforce
ServiceNow
The application usually interacts with them through an SDK or API, but the SDK only shows part of what's actually happening.
You might have this in your code:
const user = await currentUser();
The AI can understand that.
What it can't necessarily see is everything configured behind it:
organization roles
permissions
SSO configuration
session settings
MFA requirements
dashboard settings
That logic still affects the application. It just doesn't live in Git.
Clerk
Clerk is probably one of the clearest examples I've run into personally.
Your application might contain something like:
has({ permission: "org:admin" })
From the code this looks pretty simple.
If an admin suddenly can't access something, an AI coding tool might search the repository and decide the authorization check looks correct.
The actual problem could be that the permission wasn't assigned to the role in Clerk.
There may be absolutely nothing wrong with the application code.
A human engineer who's been working on the system probably knows to check Clerk.
The AI might spend a bunch of time trying to "fix" code that isn't broken.
Shopify
Shopify has the same problem but at a much larger scale.
I've worked with Shopify where a lot of the behavior wasn't really owned by the React application at all.
Products, metafields, discounts, checkout behavior, themes, extensions, merchant configuration and installed apps can all affect what happens.
The application might simply do:
const product = await shopify.getProduct(id);
but the behavior behind that product could depend on configuration that never appears anywhere in your repository.
This gets especially interesting when something breaks in production.
The code didn't change.
Shopify configuration did.
If the AI only has access to Git, its view of the problem is already incomplete.
Stripe
Stripe is another obvious one.
A lot of payment logic looks like normal application logic from the outside:
await stripe.paymentIntents.create(...)
But Stripe owns a huge amount of state:
customers
products
prices
subscriptions
connected accounts
invoices
tax configuration
webhook configuration
payment methods
I've worked with Stripe Connect where the state of the connected account mattered just as much as anything happening in our own database.
An AI looking at the code might see a connected account ID and understand that we're sending money somewhere.
It doesn't know who actually owns that account, whether payouts are enabled, what bank account is attached or what configuration was changed in the Stripe Dashboard.
Again, part of the application is outside the application.
Auth0 and Okta
Auth0 and Okta have the same issue as Clerk, especially once you get into larger organizations.
You might have application code that validates a token and checks some claims, while the actual behavior depends on:
roles
groups
identity providers
SSO
MFA rules
tenant configuration
custom actions
claims
A lot of this can change without a deployment.
That's the interesting part.
We've always thought of source control as the history of how the application changed.
That's not completely true anymore.
The application can change while Git stays exactly the same.
LaunchDarkly
LaunchDarkly might actually be the purest example of this problem.
Your code could contain:
if (flags.newCheckout) {
return <NewCheckout />;
}
return <OldCheckout />;
The AI sees two code paths.
Which one is production using?
Maybe both.
The answer could depend on:
environment
user ID
organization
percentage rollout
segment
prerequisite flag
manual override
None of that logic necessarily exists in your repository.
You could ask an AI:
Why does Customer A see the new checkout but Customer B doesn't?
and it could spend forever analyzing React when the answer is a LaunchDarkly targeting rule.
That's a pretty large blind spot.
Algolia
Search is another place where a surprising amount of behavior can live outside the code.
I've worked with Algolia where the frontend really only knows how to send a query and display the results.
The actual search behavior can depend on:
ranking
searchable attributes
facets
synonyms
replicas
merchandising
index configuration
Suppose someone says:
Search results are bad for "black dress."
The AI opens the frontend and sees:
index.search("black dress");
There's not much to fix.
The problem could be that somebody changed the ranking configuration or synonyms inside Algolia.
A human who knows the application understands that search isn't just code.
An AI has to be given that context somehow.
Contentful and Sanity
CMS platforms create a similar problem.
Your React application might know about:
article.title
article.heroImage
article.body
but the actual content model, published entries, references, localization and editorial state live somewhere else.
This isn't necessarily traditional "business logic," but it absolutely affects application behavior.
An AI can look at the component and decide heroImage should always exist because the TypeScript interface says it's required.
Meanwhile an editor created 300 old articles before that field existed.
The production data is telling a different story than the code.
Sanity is slightly better here because more of its schema can be represented as code, but the actual content and relationships still exist outside the application repository.
Salesforce
Salesforce is where this problem can get pretty extreme.
I've seen enterprise applications where Salesforce isn't just a database. It contains a huge amount of business behavior.
Things like:
validation rules
flows
approval processes
permissions
custom objects
automation
workflow rules
Imagine your application updates a customer:
await salesforce.updateCustomer(customer);
Then something unexpected happens.
Maybe an approval process starts. Maybe another record gets updated. Maybe an email gets sent. Maybe the update is rejected because of a validation rule.
None of that behavior has to exist in your application's repository.
You could give the AI every line of your code and it still wouldn't understand the complete workflow.
ServiceNow
ServiceNow is very similar.
A lot of enterprise workflows can live almost completely inside the platform:
forms
business rules
approval workflows
permissions
integrations
automation
From the application side you might just see an API call.
Behind that API call there could be years of business logic.
This is probably where the term "black box" starts feeling pretty accurate.
The Repository Isn't the Application Anymore
This is the part I find interesting.
For a lot of modern applications, the actual system looks closer to:
Application Repository
+
MongoDB
+
Clerk
+
Stripe
+
Shopify
+
Algolia
+
Feature Flags
+
CMS
+
Production Configuration
=
The Real Application
But an AI coding agent might only see:
Application Repository
Maybe the database schema if you're lucky.
That's a huge difference in context.
And AI is extremely dependent on context.
Humans Have Institutional Context
A senior engineer working on an application for a few years picks up a lot of information that never gets written down.
You start knowing things like:
That permission is configured in Clerk.
That product price comes from Stripe.
Don't change that field, Shopify owns it.
That search issue is probably Algolia.
That customer is on a feature flag override.
None of those things are necessarily obvious from reading the code.
When another human joins the team, that knowledge gets transferred slowly through pull requests, Slack messages, documentation and somebody saying "oh yeah, that's weird because..."
AI doesn't really get that same onboarding.
Every new coding session can effectively start with:
Here's the repository. Figure it out.
That's probably fine when most of the application lives in the repository.
It's much harder when half the application lives in dashboards.
Vendor Lock-In vs Context Lock-Out
The traditional criticism of these platforms has always been vendor lock-in.
You build heavily around Shopify and now moving away from Shopify is difficult.
You build around Stripe and now Stripe is deeply embedded in your payment infrastructure.
That's still true.
But I think AI introduces another problem.
I would call it context lock-out.
Vendor lock-in is:
It's difficult to move away from this platform.
Context lock-out is:
It's difficult for an AI agent to understand the application because important parts of the application live inside this platform.
The application still works perfectly fine.
The developer can still understand it.
But the AI has an incomplete model of the system.
This Doesn't Mean We Should Stop Using SaaS
I'm definitely not suggesting we start rebuilding all of this ourselves.
That would probably be much worse.
I don't want our authentication implementation sitting in 30,000 lines of custom code just so an AI can read it.
The benefits of these platforms are still huge.
They give us:
less code
faster development
better security
specialized infrastructure
compliance
scalability
reliability
And generally fewer things we have to maintain.
The tradeoff is that we're moving application knowledge somewhere else.
That was already a problem for developers.
AI just makes the cost much more obvious.
Maybe These Platforms Need to Become AI-Native Too
The obvious solution is probably not putting everything back into the repository.
It's giving AI access to more of the external context.
Imagine an AI coding agent being able to inspect:
Clerk roles and permissions
Stripe products and subscriptions
Shopify metafields and configuration
LaunchDarkly targeting rules
Algolia index configuration
Contentful schemas
Salesforce flows
Now when you ask:
Why doesn't this user have access?
the AI could inspect both:
application code
+
Clerk configuration
That's a much more useful debugging environment.
Some of this is already possible through APIs, MCP servers, CLIs and other integrations.
I think that's going to become increasingly important.
It's not enough for AI coding tools to understand the repo.
Eventually they need to understand the application environment.
Conclusion
For years we've been moving functionality out of our applications and into managed platforms.
That was mostly a good thing.
We write less code, ship faster and let companies that specialize in authentication, payments, commerce or search handle the difficult parts.
AI changes the tradeoff a little.
The better these coding tools get, the more valuable complete context becomes.
And a modern application's context isn't necessarily in Git anymore.
It's spread across your repository, database and probably five or ten different SaaS dashboards.
We've spent years making applications easier to build by hiding complexity behind APIs.
Now AI is making us realize that sometimes the hidden complexity was still part of the application all along.