Enterprises spent 2023 and 2024 racing to adopt generative AI, convinced the hard part was choosing the right model. They discovered otherwise. The bottleneck was rarely the intelligence layer. Most often, it was the connection layer between AI applications and the databases holding their proprietary data.
Vinodkrishna Gopalan saw this problem early. As Head of Engineering for database connectivity and CDN infrastructure, he built the middleware systems that allow enterprises to actually deploy AI applications against relational databases without connection exhaustion, timeout storms, or security failures. His work enabled vector stores for retrieval augmented generation, optimized caching for LLM workloads, and developed connection management techniques that handle the chaotic, bursty access patterns AI applications generate.
While the industry fixated on model capabilities, Gopalan focused on the infrastructure underneath. That meant solving problems most teams discovered only after they reached production: how to handle hundreds of parallel embedding lookups hitting a database designed for orderly transactions, how to cache AI inference when exact-match strategies fail, and how to keep AI workloads from overwhelming the same databases running critical business operations.
His perspective offers a practical counterpoint to the hype cycle. He argues that most enterprises already have the data, security frameworks, and operational expertise they need. They lack the orchestration layer that translates between AI access patterns and traditional database infrastructure. That thin but critical middleware determines whether enterprise AI initiatives succeed or stall out at the proof-of-concept stage.
Gopalan walked us through what breaks first when enterprises try to deploy RAG applications, why the middleware layer matters more than most people realize, and where architectural decisions determine whether AI deployments thrive or collapse under production load.
Enterprises are discovering that their existing database infrastructure wasn’t designed for AI workloads. What’s the first bottleneck they hit when they try to deploy RAG or vector search applications?
The first thing that breaks is almost never the mode itself, it’s connection concurrency. RAG applications generate bursts of database queries that look nothing like traditional OLTP traffic. Traffic changes from predictable request-response patterns to hundreds of parallel embedding lookups and retrieval queries hitting the same database simultaneously.
Most enterprise database infrastructure are provisioned for steady-state transactional workloads. Their connection limits get exhausted almost immediately, and suddenly customer’s AI application is queuing behind their business workflows or worse AI Application has saturated their database and impacting critical business workflows.
The irony is that the database itself often has plenty of compute headroom; it’s the connection layer that becomes the chokepoint. Enterprises don’t realize this until they’re in production, because dev environments never expose the concurrency problem. The bottleneck is rarely intelligence, rather the plumbing.
There’s extensive coverage of AI models and applications, but the middleware layer that connects AI to enterprise data gets less attention. Why does that layer matter more than most people realize?
Everyone’s focused on what the model can do. Almost nobody’s asking how the model gets to the data. That middleware layer (connection management, query routing, caching, authentication) is where enterprise AI initiatives actually succeed or die.
It’s the translation layer between AI’s chaotic, bursty access patterns and databases that expect orderly, transactional behavior. Without it, you get timeout storms, connection exhaustion, and data access failures. These failures often incorrectly get lumped into the Model is not working category. The middleware is also where you enforce security, manage credentials, and control costs. Models are commoditizing fast. The differentiation for enterprises is going to be how efficiently and securely they connect AI to their proprietary data.
Customers have to solve middleware problems to really unlock their model’s potential.
Vector databases have become a popular solution for AI applications, but many enterprises already have relational databases full of critical data. What does it take to make traditional database infrastructure work for modern AI use cases?
Some customer’s initial instinct is to rip and replace and stand up a dedicated vector database and migrate. This is very expensive, operationally complex, and often unnecessary. Most enterprises already have the data they need sitting in relational databases with years of governance, backup, and compliance infrastructure built around it. The real work is adding a semantic access layer on top. That means enabling vector extensions or hybrid search capabilities within existing engines, then building middleware that can handle the mixed workload: transactional queries alongside similarity searches.
A simpler HTTP based access pattern, and then connection multiplexing becomes critical here because you’re now running two fundamentally different query patterns against the same infrastructure.
The enterprises that succeed treat this as an augmentation problem, not a replacement problem. “Keep the data where it is, change how you access it.”
Caching strategies that work for web content often fail when applied to AI inference workloads. What makes AI caching fundamentally different, and why does that difference matter at scale?
Traditional caching assumes exact-match lookups:same key, same response. AI inference workloads break that assumption completely. Two semantically identical questions with different phrasing produce different cache keys under conventional approaches, so you get near-zero hit rates.
You need semantic similarity-aware caching, where “close enough” queries can return cached results. But that introduces a precision-recall tradeoff that doesn’t exist in web caching: how similar is similar enough?
At scale, this matters enormously because LLM inference is orders of magnitude more expensive per request than a database query. A well-designed semantic cache can cut inference costs by 30-40%, but a poorly designed one returns stale or irrelevant results and erodes user trust. The caching layer for AI needs to be probabilistic, not deterministic, and that’s a paradigm shift most infrastructure teams haven’t made.
Connection management techniques like pooling and multiplexing have been around for decades. How do AI workloads change the requirements for these techniques in ways that traditional approaches can’t handle?
Traditional connection pooling assumes relatively uniform, short-lived transactions. AI workloads go against both these assumptions. You get long-running retrieval queries that hold connections for minutes instead of milliseconds, mixed with rapid-fire embedding lookups that need sub-millisecond connection acquisition. The variance in hold times is enormous. Classic pooling algorithms that size pools based on average hold time either over-provision (expensive) or under-provision (timeouts).
Connection multiplexing helps because you can interleave short queries on connections that would otherwise be idle during long-running operations, but the multiplexing logic needs to understand query characteristics and database connection state to route effectively. You also see connection storms during inference; a single user prompt can fan out into dozens of parallel database calls. The connection layer needs to absorb that burst without propagating it to the database. This allows for traffic shaping, not just connection sharing.
When you look at enterprise AI deployments that struggle or fail, what architectural decision usually caused the problem?
Tight coupling between the AI application and the database. Almost every time. Teams build their RAG pipeline with direct database connections, hardcoded connection strings, and application-level retry logic. It works in development. Then they hit production scale and discover they’ve created a brittle system where a database failover cascades into application-wide outages.
In practical terms, as customers scale the AI tier, their database connections too scale linearly overwhelming the underlying data architecture. The fix is always the same: introduce an abstraction layer between the AI application and the data. But retrofitting that into a production system is painful and risky. The teams that succeed architect for that separation from day one. The ones that fail treat the database as a direct dependency rather than a service behind an interface.
API design for AI-powered applications requires different thinking than traditional database access patterns. What shifts in how enterprises expose their data to AI applications?
Traditional database APIs are precise. A SQL query tells the database exactly what is wanted. AI applications need declarative, intent-based access. The application says “find me everything relevant to this concept” rather than “select from table where column equals value.”
That shift has massive implications for API design. You need APIs that accept natural language or vector representations as inputs, that can orchestrate hybrid queries combining structured filters with semantic search, and that return results with relevance scores rather than exact matches.
Pagination changes too; AI applications often need the top-K most relevant results, not sequential pages. The authentication model shifts because AI applications often need broader read access across more tables than any single traditional application would. Enterprises that try to force AI workloads through existing REST APIs designed for CRUD operations end up building complex orchestration layers that should have been in the data access tier from the start.
Most enterprises assume they need to build entirely new infrastructure for AI. How much of the solution already exists in their stack but needs to be configured or used differently?
More than most enterprises think. The relational databases already store the critical data. Most modern database engines support vector extensions or can be augmented with them.
Connection pooling and management infrastructure exists, it just needs to be reconfigured for AI traffic patterns. Caching layers are in place, they need semantic awareness added. The security and compliance frameworks are already built, but need more innovation to keep up with Agentic workflows accessing production databases.
What’s actually new is the orchestration, the middleware that understands AI access patterns. That is a thin but critical layer. The biggest waste I see is enterprises building parallel infrastructure stacks for AI when they should be building better plumbing to their existing stack. The data is already there. The governance is already there. The operational expertise is already there. Often customers just need a smarter middle layer, not a new foundation.
