Over 40 years ago, the open source software movement started as a push for transparency and developer freedom in a world dominated by closed platforms. It was seen as a way to reclaim control; to understand, modify, and improve the software developers relied on. This relentless pursuit of transparency and trust is what Ken Thompson highlighted in “Reflections on Trusting Trust” which shows how a system’s core can be compromised if you can’t see the underlying code. Over time, through community collaboration, standard-setting, and commercial investment, open source software became the backbone of modern digital infrastructure.
Today, open source AI is at a similar inflection point, only moving much faster. Meta’s open source models Llama and Mistral exploded in popularity by offering cheap, flexible alternatives to restricted proprietary solutions by OpenAI, Anthropic, and Google. DeepSeek’s launch this year was even called the industry’s own ‘Sputnik moment’.
But this momentum isn’t just about cost or flexibility. In a field changing at breakneck speed, trust and transparency are core concerns for developers and enterprises, and open source is seen as the way to get it. A 2025 Stack Overflow Developer survey found 61% of developers trust open-source AI for development work, compared to just 47% who trust proprietary models.
Yet, many of the safeguards that matured in traditional open source like software signing, provenance, and verifiable builds, are still catching up in AI. And that gap carries real risk.
Hidden risks of AI models
AI is already woven into our software, infrastructure, and even our decision-making processes. However, despite its business-critical value, AI can present a vast attack surface for businesses.
AI models are even more opaque than traditional software artifacts. To make matters worse, they’re much less understood by IT professionals so far. That combination makes them attractive targets for hackers and introduces a fundamentally different kind of risk altogether.
Picture a model that looks benign on the surface; it answers questions, writes code, translates text, and delivers every task you throw at it. But it has some embedded malicious behavior that only activates under specific conditions or prompts. It’s the AI equivalent of a “Manchurian Candidate” – an invisible backdoor just waiting to be triggered. With closed models, you’re left with having to trust the publisher, hoping they did the right thing. You cannot inspect the training data or the code, nor can you verify that the weights you downloaded haven’t been tampered with. It’s a risky position to be in when the stakes are high.
The real appeal of open source AI
On the contrary, open source AI models offer relative transparency into how the model was built. The source code, data, and training pipelines are all published, giving the community a chance to understand how the models reason and generate decisions. More importantly, in true open-source, users can reproduce equivalent artifacts themselves, compare outputs, and make sure what they’re running is what they think it is.
However, there lies a big gap in the data used for training. Many “open” models do not release their full training datasets due to issues with copyright, intellectual property, or personally identifiable information, meaning they are not truly open. This distinction is important, as a model can’t be fully rebuilt from its source unless we have the data too. This reproducibility is where we need to push the open-source movement in AI since it is the foundation of trust in any software system.
One of the big lessons from open source software development is that transparency counts. Most attacks against open source software don’t actually target the code itself but the opaque artifacts people publish, such as precompiled binaries, containers, or packages published without a verifiable link back to the source. This is why practices like software signing and reproducibility are important: a vendor will assert that they produced a particular artifact. But signing allows consumers to verify that what they’re running matches what was built from the source code, and who built it.
But let’s be clear: we’re still in the infancy stage of AI, more so for open source AI. The security practices like SBOMs, verifiable builds, and provenance metadata that have matured over decades to make traditional open source software secure aren’t widely established in the AI space yet. There is plenty of work ahead to make these models not only more powerful, but also verifiably safe for everyone depending on them. And given how fast AI is being deployed, we don’t have the luxury of waiting.
Making AI models secure
Models like Llama and Mistral are popular not just because they’re fast or low cost but because they’re available, inspectable, and, to a degree, reproducible. Users are gravitating towards them not just for flexibility and cost benefits, but primarily trust. They want knowledge of what they’re running or working with. They want to feel confident that their AI isn’t going to surprise them.
IT leaders have a huge opportunity and responsibility to apply the lessons of secure software to the world of AI. That means building the tooling, infrastructure, and standards to support open and secure models, holding ourselves and our vendors to higher standards. Open source AI gives us a chance to get this right. Not just to build more powerful models, but to build ones we can trust.
The open source community also plays a vital role here, not just in building better models, but in enabling the support, collaboration, and reproducibility techniques we need to make AI as safe and reliable as possible. The industry needs visibility into the processes, such as data sourcing, cleaning, code auditing and maintenance, and the governance of contributions. Reproducibility should not just be a goal, but an accepted norm and standard, just like it is for secure software builds today.
Trust is not something that can be bolted on at the end. It must be built in from day one. That is why the growing momentum behind open source AI is a step in the right direction. It’s a recognition that security, safety, and reliability all start with openness.
