The model achieves 95% accuracy on the test dataset. Can we consider the AI ready for production?
For Defense AI, the answer is: not necessarily.
In real-world conditions, the model faces not a carefully prepared benchmark, but smoke, dust, camouflage, changing lighting, new objects, limited computing resources, and system-wide latency.
This is where the gap between model performance and operational performance arises.
For Valkyria, Ukraine-based defense software company, this is not a theoretical problem. The company develops AI and machine learning software for defense systems, from computer vision and autonomous systems to embedded and edge AI, and tests its solutions in Ukraine, where models are exposed to real-world conditions.
Benchmarks are just the beginning.
An AI model can be trained and tested in a relatively controlled environment. But deployment changes the conditions.
For example, a computer vision model might be good at finding an object on a curated dataset, but it loses quality when the object is partially obscured, the camera angle changes, or smoke and dust appears. Valkyria highlights this gap as one of the key challenges of Defense AI: models can perform well on curated datasets but perform worse in real-world environments. Therefore, for defense systems, field data is not simply an additional source of information after development. It’s part of the product itself.
This problem is already visible at the level of American military programs.
The U.S. Government Accountability Office’s report on the development of AI for weapon systems states that the Department of Defense faces challenges specific to AI: the need to have sufficient usable data to train models, integrate trained AI into existing systems, and continuously monitor its performance. The GAO specifically emphasizes that AI differs from traditional software: it requires large volumes of data, computing resources, and continuous monitoring.
Therefore, a good test score is not proof of operational readiness.
What happens after the benchmark
In Defense AI, a model is only part of the system. A hypothetical pipeline looks like this:
Sensor → Data → Processing → AI Inference → Decision
If a model performs well but doesn’t meet latency requirements, overloads the edge device, or loses quality on real data, then the problem is not only with the ML algorithm.
For example, Valkyria provides a simple example: a tracker that delivers results in 180 ms may be useless for a specific scenario if this latency doesn’t meet the requirements of the entire system. Therefore, each module requires a specific latency budget for the target compute.
The same applies to computing resources
A model that runs perfectly on a powerful GPU system during development won’t run on edge hardware installed directly on the platform.
Valkyria faces this in practice: an FP32 model may not fit within the constraints of a 10 W edge board. Before deployment, it is necessary to use quantization and compiler-level optimization, and then re-check the performance. Therefore, the question for Defense AI is “Can the entire system perform the task on the hardware, in the environment, and at the speed required in reality?”
The American approach is also shifting towards continuous testing.
It’s not simply the position of individual Defense AI companies.
In its Data, Analytics, and Artificial Intelligence Adoption Strategy, the Department of Defense explicitly includes continuous experimentation and feedback between developers, users, and testing and evaluation specialists.
The DoD document specifically notes that different users work in completely different conditions: some have access to large volumes of data and high processing speeds, while others have limited bandwidth at the tactical edge. Therefore, such limitations must be considered in the development phase.
This is an important shift.
AI can no longer be developed according to the model:
build → benchmark → deploy.
A cycle is needed:
build → deploy → test → collect feedback → improve → deploy again. This is precisely the approach Valkyria uses in developing defense AI: operational feedback and field-driven validation become part of the process, not a final check of the finished product.
Field data is more important than a perfect benchmark
For Defense AI, one principle is especially important: real data must be fed back into the ML pipeline.
The model is trained on a specific dataset. After deployment, the system begins to see new conditions and new error types. This data can be used for the next iteration of training and testing.
It creates a closed loop:
Field data + Training + Model + Deployment + Field testing + Feedback + New data.
This is why Valkyria emphasizes field data collection and operator feedback. The company views operational feedback as part of the AI development process, along with machine learning, computer vision, MLOps, and embedded/edge AI.
This approach is especially important for systems that need to run directly on the platform.
If a model is intended for edge deployment, it needs to be tested not only on the developer’s server but also on the target hardware.
If AI needs to process a video stream in real time, it is necessary to measure not only the model’s inference time but also the latency of the entire system.
If the system needs to operate in a changing environment, real data that reflects that environment is needed.
A true benchmark is seen in the field
Will the system work when conditions no longer resemble the benchmark?
This is where the line is drawn between AI that performs well in tests and AI that is ready for operational deployment.
For Valkyria, this principle can be formulated simply:
The model must be tested not only on data but also on the hardware, in the system, and in the environment where it will actually operate.
Because in Defense AI, the benchmark demonstrates the capabilities of the model.
And the field demonstrates the capabilities of the system.