When Not to Use LLMs: Lessons from Fintech and Healthcare Startups

Large language models have sparked innovation across nearly every industry. From automating complex tasks to generating human-like text and powering new user experiences, the technology is rapidly transforming the business landscape. However, as AI adoption accelerates, it’s crucial to recognise a less glamorous, but equally important truth: LLMs are not always the right solution. 

In this article, I’ll explore why and when organisations should not reach for LLMs, drawing from two anonymised cases of startups I’ve mentored in fintech and healthcare. But first, we need to understand what LLMs are, what they require, and where their limitations lie. 

What Is AI? What Are LLMs? 

Artificial intelligence (AI) broadly refers to systems that perform tasks requiring human-like intelligence, such as learning, reasoning, and decision-making. Machine learning (ML), a core branch of AI, enables systems to improve over time by learning from data, using algorithms like decision trees, regression, or neural networks. 

Large language models (LLMs) represent a specific type of AI designed to understand and generate human-like text. Trained on massive datasets, these models excel at tasks like summarisation, text generation, and question answering, making them incredibly versatile across industries, from legal contract analysis to customer support. 

What Do These Models Need? 

At their core, AI models are only as good as the data that fuels them. LLMs require: High-volume, high-quality text data for training; Robust fine-tuning datasets for specific tasks; and human feedback loops to guide alignment.

Key Limitations of LLMs 

Despite their power, LLMs come with major constraints: 

• Hallucination: LLMs sometimes produce outputs that are factually incorrect or fabricated, even when sounding confident and plausible. 

• Explainability: Unlike simpler machine learning models, LLMs behave like “black boxes.” It’s often difficult (or impossible) to explain why a model produced a specific result. 

• Dependence on Input Prompts: LLM outputs heavily rely on the quality and structure of the input prompt, adding unpredictability in high-stakes environments. 

• Data Mismatch: If the LLM hasn’t been trained on domain-specific data, or if sensitive or proprietary data can’t be shared, the model may underperform or produce unreliable results. 

These limitations are not merely technical, they can fundamentally undermine business outcomes. Two startups I’ve worked with recently learned this the hard way. 

The Healthcare Case: No AI Without Good Data 

One healthcare startup I mentored wanted to introduce AI-powered diagnostics into hospitals. Their goal was to assist medical professionals by flagging high-risk cases and suggesting treatment options based on patient history. Their ambition was admirable, but their biggest obstacle wasn’t the model or the algorithms. It was the data. 

The hospital’s records were riddled: incomplete patient histories, missing fields and unstructured notes, inconsistent data entry practices among medical staff and poor documentation of follow-up outcomes.

Without high-quality, reliable data, any machine learning solution was bound to fail. In this case, the most responsible and impactful advice was simple: “Don’t implement AI yet. First, invest in digitisation and data hygiene.” 

What this case highlights is critical: AI cannot compensate for poor data foundations. In industries like healthcare, the priority must first be: 

1. Structured, consistent data capture 
2. Training staff on the importance of proper documentation 
3. Developing data governance policies 

Once data quality improves, predictive models can be explored. Until then, AI efforts are likely to waste time, money, and resources. 

The Fintech Case: Know Your Tools 

Another startup I mentored, this time in the fintech sector, was building a platform to predict customer financial behaviour, such as default risk, churn probability, and spending patterns. 

The founders were excited about LLMs and planned to use them to generate these predictions. Their thinking was that LLMs’ ability to process unstructured text and complex correlations would give them an edge. However, they quickly ran into multiple challenges: 

• Unexplainability: In finance, regulatory and internal compliance standards require explainable models. LLM predictions couldn’t offer justifications or traceable reasoning, making them unusable for high-stakes decisions. 

• Inflexibility: LLMs struggled to incorporate structured financial data, such as transaction histories or credit scores, in a reliable way. These models excel at language tasks, but not at handling tabular numerical datasets. 

• Missed Opportunities: Traditional statistical methods were far more accurate and transparent for this problem. These models could be easily tuned and enriched with human input. 

Ultimately, I advised them to pivot back to traditional machine learning and statistical methods that offered better predictive performance, transparency, and compliance. The LLM approach not only underperformed, but it also added unnecessary complexity. 

Final Thoughts: LLMs Are Not a Silver Bullet 

LLMs and other advanced AI technologies are powerful tools, but they are not a universal solution. In many cases, the right answer may be: 

• Investing in data infrastructure first 

• Using simpler, well-understood models 

• Prioritising explainability over complexity

C-suite leaders, product managers, and founders must ask: “Is this a problem that actually requires an LLM—or am I using it because it’s trendy?” Responsible and effective AI isn’t just about what’s possible—it’s about what’s relevant.  

AUTHOR

Tomer is a tech leader in the UK. He is currently at Google Cloud, with previous experience at Microsoft and several startups. Tomer holds a BSc in Computer Science and an MBA from the University of Oxford.