For the past two years, the idea of a “modern AI stack” has been widely presented as a natural next step in enterprise architecture – structured, layered, and increasingly standardized. Most diagrams clearly separate data, models, orchestration, and governance. This creates the impression that organizations are moving step by step toward clearly defined AI systems.
This also helps explain a common frustration. Many organizations still can’t clearly show what they are getting out of AI. Without shared structures or well-defined KPIs, measuring impact is hard to pin down – even when AI tools are widely used.
In reality, things are rarely that structured. Most enterprises aren’t building anything close to a coherent AI stack. What’s actually happening is much more uneven. AI capabilities are introduced gradually, often through tools embedded in specific workflows, without a broader architectural plan guiding how everything should fit together.
In that sense, the “AI stack” isn’t really being designed – it’s emerging. It takes shape through local decisions, constraints, and experimentation rather than a top-down structure.
The real challenge for enterprises is not accessing AI capability, but turning this fragmented landscape into something that works at scale.
AI Adoption Starts in Workflows, Not Architecture
Despite all the emphasis on enterprise-wide AI strategies, adoption rarely starts there. In most organizations, it begins much closer to day-to-day work – inside individual teams trying to solve very specific problems.
In day-to-day work, developers plug AI coding assistants straight into their environments. QA teams start testing AI-driven automation alongside their usual workflows. Project managers use AI tools for meeting notes, summaries, or quick planning support. In some organizations, even compliance teams are beginning to test AI for document analysis – while still restricting how external models can be used.
On their own, these use cases often deliver clear benefits. Less repetitive work, quicker iterations, and small efficiency gains that add up over time.
Not all AI use cases scale equally. Broad, one-size-fits-all solutions – such as generic chatbots – often lose relevance quickly. More targeted tools, tied to specific workflows, tend to stick because they solve something concrete and integrate more naturally into existing processes.
This pattern is not limited to individual organizations. Across Europe, there’s still a noticeable gap between how people use AI privately and at work.
Fewer than 20% of employees report using AI tools at work, while personal use is much higher. That gap says a lot. AI is already part of everyday work – but it hasn’t really been absorbed into how organizations operate. And this is usually the point where things get messy.
Different teams not only choose different tools but also often operate under completely different assumptions. Some rely heavily on external AI services, while others avoid them entirely due to data security concerns. In some cases, even within the same organization, there is no consistent view on which tools are approved or how they should be used.
From an enterprise perspective, this creates an uneven landscape. AI is present in many places but rarely connected.
What looks like progress at team level often creates friction at enterprise level. Even within the same company, attitudes toward AI vary widely. Some teams rely on it daily, others avoid it altogether – often because of concerns around data security or trust. These differences are rarely resolved centrally and often reflect local leadership styles more than any company-wide strategy.
When Fragmentation Becomes Structural
As adoption spreads, fragmentation stops being a side effect and starts becoming a structural issue that teams actively struggle with.
What begins as local experimentation doesn’t stay local for long. At some point, tools need to interact, data needs to move across systems, and decisions made in one team start affecting others – often in ways no one planned for.
And in many cases, there simply isn’t a shared foundation to support that. Some teams rely on external APIs, others experiment with internally hosted models, and many operate in isolation with little coordination.
It’s quite common for multiple teams to work on similar AI solutions without realizing it. The result: duplicated effort, inconsistent approaches, and avoidable cost.
The challenge isn’t just technical. Another issue shows up quickly: visibility. Without a shared structure, it’s hard to see where AI is actually being used, which models are involved, or how data moves between systems.
The pattern is fairly consistent: usage grows fast, but oversight lags behind.
Constraints Shape the Architecture
This fragmentation isn’t just a technical issue – it is shaped by constraints that become visible as soon as organizations try to scale.
Most enterprises run into the same trade-off: build something in-house or rely on external services.
APIs make it easy to get started, but they come with questions around data security, long-term cost, and dependency. Cost usually becomes a concern once usage scales and bills start to reflect it.
Pricing models for AI services – whether external APIs or cloud infrastructure – are typically usage-based and not always transparent. As adoption grows, costs can increase quickly and unpredictably, especially when scaling across teams.
These trade-offs become more obvious over time. It’s not unusual for a tool that performs well for a dozen users to run into limits once hundreds – or thousands – of employees start relying on it simultaneously.
Hosting open-source models internally offers more control but introduces significant overhead. Infrastructure, GPU requirements, and ongoing maintenance quickly become limiting factors.
Governance adds another layer. In regulated industries such as finance or healthcare, requirements around data privacy, auditability, and compliance limit how AI can be deployed in practice.
In Europe, frameworks like the EU AI Act add another layer of complexity, forcing organizations to balance innovation with compliance (see related discussion on evaluating AI use cases in regulated environments).
And as use cases mature, another limitation becomes visible. General-purpose AI tools don’t perform equally well across all domains.
Tasks that require domain-specific knowledge, such as legal analysis, healthcare, or complex engineering workflows, often need additional fine-tuning or more specialized models.
As a result, architecture is not just designed. It is shaped by cost, regulation, and practical constraints.
The Missing Layer: Usability in Practice
Even when infrastructure and governance are in place, one issue keeps coming up: does any of this actually fit into how people work?
There is often a strong focus on automation – how much time can be saved, how many steps can be reduced.
But in practice, that’s not what determines whether an AI solution succeeds. At the end of the day, it’s simply: people either use the tool or they don’t.
If it doesn’t fit into existing workflows, or if it makes things more complicated, people stop using it. Even technically strong solutions can fail quietly when they don’t match how work actually happens.
This becomes obvious when tools are built without involving the people who are supposed to use them. Without that perspective, it’s easy to design something that looks efficient on paper but doesn’t hold up in reality.
At that point, usability stops being a design detail and becomes a deciding factor – one that determines whether capabilities are integrated into everyday work or remain isolated experiments.
What Starts to Matter as AI Scales
Once organizations move past early experimentation, a few things become clearer. Visibility starts to matter more than adding new capabilities.
Before scaling anything, enterprises need a clear picture of what’s already in use, where, and why. In many cases, this starts by asking teams directly what works, what doesn’t, and where AI can realistically support their workflows.
Alignment usually matters more than strict standardization. Forcing everyone onto the same tool rarely works. But leaving everything disconnected doesn’t scale either.
Progress often comes less from adding new tools and more from connecting what’s already there.
In more mature environments, you often see shared layers emerge – controlled access to models, internal APIs, or lightweight governance frameworks that bring some consistency without locking teams down.
Even then, the hard part isn’t generating output, it’s making sure it’s reliable and usable in real processes.
From Fragmented Tools to Operational AI
At some point, most organizations hit the same wall. What started as useful experimentation across teams is now creating friction at scale.
Tools don’t integrate, outputs aren’t consistent, and no one has a clear view of how AI is actually being used across the organization. This is usually the point where the conversation shifts – from “how do we use AI?” to “how do we manage what we’ve already introduced?”
In practice, this often leads to a second phase of adoption. Instead of adding more tools, companies begin to standardize selectively, introduce shared services, or define clearer usage frameworks.
Some organizations establish internal AI platforms or service layers to reconnect fragmented use cases and provide controlled access to models.
None of this is straightforward in practice. Different teams have already developed their own approaches to working with AI, and those choices are not easy to reverse. What made sense locally doesn’t always translate into something that works across the organization.
Moving from experimentation to operationalization is less about new technology and more about creating alignment.
What Enterprises Can Do Next
Enterprises don’t need to rebuild everything at once. Trying to enforce a fully unified AI architecture too early often backfires.
A more realistic starting point is visibility: understanding what’s already in use, where, and with what risks attached.
From there, organizations can begin to carefully create structure. That often means defining clear usage guidelines, introducing shared access layers for models, and aligning teams around a small number of supported approaches without removing flexibility entirely. In practice, the goal is not to eliminate experimentation. It is to contain it.
Some companies are experimenting with internal “AI hubs” – simple platforms where employees can share tools, use cases, or even small solutions they’ve built themselves. It’s a practical way to reduce duplication and make AI more visible without forcing central control.
There are organizations that are starting to move beyond this stage and treat AI more like shared infrastructure than a collection of tools. One example is the Bulgarian National Revenue Agency, which worked with INSAIT to develop a proprietary language model tailored to its needs.
Getting to that level of integration takes time, coordination, and real investment. For most enterprises, the more immediate challenge is making sense of what already exists before moving toward fully centralized setups.
Organizations that make progress tend to follow a few simple principles: make AI usage visible before scaling it, align teams before enforcing standardization, and invest in shared services that reduce fragmentation over time.
Conclusion: From Capability to Coherence
AI capability isn’t the bottleneck anymore. The tools are already there – and improving fast. The harder problem is turning that capability into something that works reliably inside an organization.
The new “AI stack” is not a clean architectural rebuild. It is an evolving system shaped by fragmentation, constraints, and continuous adaptation.
The next phase of enterprise AI will not be defined by better models or more tools. It will be defined by the ability to create coherence across systems, teams, and decisions that were never originally designed to work together.
Because the real challenge is no longer building AI. It is making it work within the organization.
