Identifying the use case and defining what success looks like
Any strategy for putting Artificial Intelligence (AI) to work to improve your business needs to start by establishing a collective understanding and agreement of what you are trying to achieve and who for.
Discovery phases often start inside customers’ data teams. In large enterprises this is likely to mean speaking with an organisation’s CDO (Chief Data Officer). However, to ensure systems are built to deliver meaningful business value, it makes sense to meet with a number of departmental heads of Product Delivery, Customer Service or Customer Success teams amongst others, to define the most pressing business requirement which AI might address.
The core goal of any system must be fully defined and agreed at different levels of the organisation from user right up to the c-suite exec signing off the budget.
Collecting, cleaning and preparing the data
Most AI initiatives lead from there naturally to an exploration of the underlying data needed. They might say, “we have a data warehouse full of these regulatorily-required records but we are not really using those records for anything (apart from holding them for the regulatory compliance purposes) right now”.
They might ask for thoughts on how to make all that data query-able and downloadable by their customers. In-house decision-makers might use intelligence from other queries to make better business decisions. So, an AI consultant’s right at the start of any engagement is to help shape a solution and work out how to organise that data so that it can be queried and thus form the basis of a valuable solution for employees and customers alike.
Any successful AI pilot will need to first explore what datasets need to be collected, extracted and checked for quality. Data gaps will need to be filled, duplications removed, mis-labelled data corrected and unit errors checked during initial Exploratory Data Analysis (EDA) work.
Unstructured data including PDFs must be logged, tagged and structured so that data can be extracted via a query. Very often, once the data has been cleaned, it is consolidated into a single database with a single version of the truth using uniform labelling.
Two of the more vital skills for ensuring data can be reliably turned into insights are AI data science and engineering. Key characteristics of any dataset are its structure, recency and depth.
You also need to consider the size of the dataset for a pilot. If you have too much data, it’s often advisable to select a sample of that data, extract and test it for completeness and accuracy, with a view to getting it ready for querying and extraction by an appropriate Large Language Model (LLM). The dataset must be large enough to ensure that the selected model being used is reliable.
Data processing and management
Beyond the initial collection and organising of data, it’s important to establish robust, sustainable data operations. The following questions need to be answered:
- Must the data support real-time querying or is batch processing sufficient?
- How will the data be secured, anonymised, and how will you control access?
- How will we classify data by sensitivity, particularly when mixing proprietary, personal, and public source data?
- How will we meet regulatory and compliance obligations, not just at point of use, but throughout the data’s lifecycle—including data disposal?
AI pilots are underpinned by sound, well-thought out data governance, as well as disciplined data science.
Data use and analysis
Ensuring AI systems provide accurate, up to date and meaningful data is a key part of our work during AI pilots. It’s important to be able to validate results against agreed ‘ground truths’. It is also important to ensure your underlying datasets are not biased or misleading.
You must also build in ‘human in the loop’ review cycles to prevent drift and ensure against ‘hallucinations’ which can emerge if queries are probing data which isn’t complete or isn’t present in the dataset. In addition, AI systems need to be built as living systems that require continuous learning, evaluation and recalibration as new data comes in.
Most of the AI initiatives we are shaping for customers right now have an intention to improve the efficiency around specific tasks and enable specific job types to be more productive. When this is the case, naturally the people who are invested in a new AI system making their life easier, will want to check its outputs very closely during pilots.
Explainability and observability are increasingly essential, enabling stakeholders to understand why an AI system produced a given output and ensuring that results remain traceable, defensible, and compliant with regulatory expectations. Above all, the analysis phase should be governed by clear KPIs linked to the original use case – ensuring that AI systems deliver measurable improvements.
AI consultants sometimes advise on which model to use according to the scale of the task and the type of data being analysed. Model selection is also an economic decision. For example, if you are only expecting a handful of in-house managers to query the system each month, an enterprise ChatGPT license is unlikely to be the most cost effective option. Alternatively, you may choose to deploy several different models to perform specific tasks.
Building workflows and enabling integrations
A good deal of the work we are doing is associated with remaining compliant. Our job often enables automation of legal or regulatory reporting. We could be doing the database building, the training of the models, or even offering an end-to-end outsourced, dashboard front-ended service.
For many companies, there is a nervousness around sharing company confidential data with a public model. AI consultants like us are increasingly involved in configuring what’s called ‘closed loop’ chatbots, so that new information can come into the chatbot but that underlying data cannot leak out of the model.
Often the systems we are building need to be integrated into existing enterprise systems. Indeed, if we are dealing with customer data we are likely to be integrated with an organisation’s CRM system. If the AI solution is linked to customers’ supply chain, we may need to integrate with their enterprise SCM (Supply Chain Management) system.
Summary
Right at the start of shaping any Proof of Concept it is important to try to establish with the customer what the desired outcome(s) is. What would good look like for them? What user experience are they looking to deliver? It is important to define goals which reach beyond efficiency into adding value to a key stakeholder, and add to the bottom line.
For if the AI pilot can evidence both value and the promise of additional revenue, then the company is not only going to invest to put that AI system into production but be willing to continue investing to nurture, enhance and improve it, and expand the number of users which derive value from it. That can set a good precedent for tackling the next challenge for which AI may offer a solution.
