Big data is getting bigger, as business leaders now commonly deal with data in petabytes or even exabytes. The World Bank estimates that by the end of 2022, annual total internet traffic will have increased 50% from levels seen in 2020, reaching 4.8 zettabytes. So how can business leaders make sense of the vast amounts of data at their fingerprints?
The solution lies with data experts – who are in exceedingly high demand. Research by the Royal Society in the UK found that demand for data specialists had tripled in the space of five years, rising by 231% during that time. Further, one in ten job adverts now mentions the need for data skills, and the struggle to keep up with the supply of data experts, who are highly-qualified professionals with backgrounds in computer science, or machine learning. For business leaders, this necessitates new ways of thinking about and dealing with data.
In response, businesses are already responding to the challenge and embracing new technologies, job roles, and ways of thinking. Forward-thinking organisations are slowly moving away from the old model where data is dealt with by a central team of data experts. The central team processes data into dashboards and reports before delivering it to management, who use these insights to make decisions. In short, the journey always starts with the organisation.
Data experts or data citizens?
In a world where data informs every decision, this model is no longer agile enough. Data is now central to our society, driving everything from traffic management to healthcare to fraud detection. Business leaders need to simplify and modernise their data stack, training employees to embrace the possibilities of data, and delivering it directly to the workers who will actually use it, rather than hoarding it within one hyper-specialised team of data experts. New technologies and new job roles offer businesses a chance to rise above the talent shortage, and make better use of the data at their fingerprints.
The first step is prioritisation. Businesses which fail to invest in their own internal talent and upskill will face challenges of being left behind, as their rivals use data more effectively and derive actionable business insights faster. Gartner warns that most CDOs (Chief Data Officers) will struggle to foster the data literacy within their organisations required to achieve data-driven goals until 2025, or perhaps even later. Businesses need to make data a priority, and that starts with training staff to deal with it.
In terms of staffing, companies need data leaders who understand the full picture, on the business and technology sides. Data leaders can help with data discoverability, training existing employees with the data skills needed. Further, data literacy is becoming part of the course at many major companies. Companies like Bloomberg and Adobe are going further, with in-house digital academies where workers are schooled in how to use data. Analytics engineers are one example, who transform data for business users and collaborate with teams to analyse the data, ensuring that the business can use the newly-discovered insights that their work generates. Working alongside wider data teams, these engineers are the key to setting up a modern data stack.
New ideas, new technologies
Alongside the mindset to understand data talent, the right use of technology also helps in supporting internal data experts. Low-code and no-code solutions are already making data more accessible to workers outside specialised data teams, with these ‘data citizens’ already able to extract business value from data, without relying on over-stretched data specialists.
Reverse ETL will also be powerful. It reverses the traditional ‘Extract, Transform, Load’ process by which data is loaded into a data warehouse. Instead, Reverse ETL delivers data directly into the software used by frontline teams such as sales or marketing. With data transformed in the data warehouse and fed directly into software such as ERP (Enterprise Resource Planning) or CRM (Customer Relationship Management), teams have data at their fingertips, in software which they use daily. For example, this can be used to deliver personalised offers to repeat customers. These technologies will be key to breaking down the silos around data.
Engaging with the ‘data mesh’
This ties into a new organisational approach to data: the decentralised ‘data mesh’. Basically, a ‘data mesh’ is a ‘self-serve’ model which frees companies from their reliance on one, central team of data experts. Instead, access to data is democratised. Data is queried where it lives within the organisation by the business users themselves.
The significance of the data mesh is that it distributes data ownership across the organisation, empowering teams to access the correct data they need when they need it.
By introducing a universal interoperability layer or platform that facilitates the connection of domains and associated data assets within it, companies can operationalise the data mesh effectively and manage the entire operation.
It’s no longer enough to simply be aware of data and its business potential. Data meshes will be key to enabling the idea of ‘data as a product’, applying the product life cycle to data deliverables. A data mesh approach ensures the discoverability, explorability and security of datasets are retained, allowing business leaders to apply product thinking to datasets.
The shift towards the ‘data mesh’ has already begun: it won’t be instant, but companies can’t afford to ignore the change. Increasingly, companies will compete on the speed with which they can derive business value from the information available to them. In the coming years, embracing the ‘data mesh’ will become a business imperative.
Unlocking more potential with data
Business leaders will increasingly realise that it’s essential that teams can access data directly, within the systems and processes they are already using. Users need to be provided with the skills and tools to self-serve data easily, across the business. By using data operations such as Reverse ETL, companies can prevent their central data team becoming a bottleneck, allowing teams to act on data in real time and make key decisions.
Being early to embrace the ‘data mesh’ and the right data tools will offer companies a key advantage, in a future where every worker is a data citizen, and the true business potential of data is unleashed. That time for movement is now, not later.
