After many years of admiration for the data wizards congregating in their ivory towers, some senior execs finally gather the courage to admit: “We don’t know what they are doing over there”.<\/em><\/strong> <\/p>\n\n\n\n

And while they might have cut a cheque in the good times – will they still do it in the era of massive layoffs and the global economy contracting?<\/p>\n\n\n\n

To understand what is going on, we need to look back at how it started in the first place.<\/p>\n\n\n\n

Creation of Data Scientist<\/strong>s<\/h2>\n\n\n\n

\u201cWeeks of Data Engineering, 1-day training Machine Learning models. Welcome to Data Science.\u201d<\/em><\/p>\n\n\n\n

The key skill needed to create a Predictive Model is the ability to train Machine Learning (ML) models.<\/p>\n\n\n\n

ML learns based on a large pool of examples and without them it\u2019s useless.<\/p>\n\n\n\n

And here is the crux of the issue – the real-life data is very rarely laid out as examples. <\/p>\n\n\n\n

It’s more of long-winded stories<\/strong> recorded in the database introducing the characters, developing them through time, sometimes jump over plot-holes, just to end in the least expected moment.<\/p>\n\n\n\n

ML can’t take any of it.<\/p>\n\n\n\n

Hence our ML specialists spend their lives on of single-handedly transforming the databases into an ML-friendly format. <\/p>\n\n\n\n

They improvise, hacking their way through, desperately trying to reshape the data by any means available. <\/p>\n\n\n\n

People on the outside are not even trying to understand what is going on. <\/p>\n\n\n\n

Over time it may look like this person \u2013 called a Data Scientist now<\/strong>, is the only one able to start with the raw data and work out their way to a Model. <\/p>\n\n\n\n

A unicorn.<\/em><\/strong><\/p>\n\n\n\n

It seems though that everybody lost the sight of the fact that there have been plenty of professionals around, working on reshaping the data. <\/p>\n\n\n\n

They are called Data Engineers.<\/strong><\/p>\n\n\n\n

Weeks of Data Engineering plus 1 day of training Machine Learning models. <\/p>\n\n\n\n

Welcome to Data Science.<\/p>\n\n\n\n

<\/figure>\n\n\n\n

Data Scientists are Machine Learning specialists forced to do ad-hoc Data Engineering. <\/p>\n\n\n\n

It\u2019s that simple.<\/p>\n\n\n\n

And it\u2019s bad for several reasons.<\/p>\n\n\n\n

Machine Learning specialists are not trained in best practice in Data Engineering. <\/p>\n\n\n\n

As such, typically they are not very good at it. <\/p>\n\n\n\n

They also don\u2019t like it.<\/p>\n\n\n\n

They resent that work and often feel it\u2019s beneath them.<\/p>\n\n\n\n

Reality Check<\/strong><\/h2>\n\n\n\n

“The slowness of Data Scientists created the impression that the world needs a lot more Data Scientists. But it doesn\u2019t.<\/em>“<\/p>\n\n\n\n

This is a good time to mention that over the last few years almost anybody who was tasked with independently delivering something, anything<\/em><\/strong> of business value from a messy data started calling themselves a Data Scientist, making the popular understanding of the role extremely muddied.<\/p>\n\n\n\n

How does that link to the “fight Covid-19” fiasco?<\/p>\n\n\n\n

Firstly – most of the Data Scientists declaring the “fight with the virus” did not fully realize that there is no data to work on. <\/p>\n\n\n\n

It’s the same hubris that promised to change the business with AI and later delivers a logarithmic chart.<\/p>\n\n\n\n

Secondly – those who got access to the real data<\/strong>, meaning – patient-level data with a medical history, will spend weeks if not months<\/strong> on engineering this mess into a format ready for analysis.<\/p>\n\n\n\n

Because data engineering executed by Data Scientists is very, very slow.<\/p>\n\n\n\n

<\/figure>\n\n\n\n

This slowness created the impression that the world needs a lot more Data Scientists to do the data science.<\/p>\n\n\n\n

But it doesn\u2019t.<\/p>\n\n\n\n

It\u2019s been three decades \u2013 how do we fix this?<\/strong><\/h2>\n\n\n\n

\u201cExtracting insights from the data should be a part of the fabric in the organisation \u2013 a part of a process, not a project.\u201d<\/em><\/p>\n\n\n\n

This inefficient, expensive, over-promised, and misguided execution of analytics has been in place for over 3 decades. <\/p>\n\n\n\n

In 2018\/2019 though there has been a growing realisation that the ROI on slow analytics is not there.<\/strong><\/p>\n\n\n\n

Add the pandemic to the mix and the decision-makers wake up every day thinking – how yesterday’s news will impact us going forward? What changed in our business in the last few days? Are our months-old models any good right now?<\/p>\n\n\n\n

So how do we fix this?<\/p>\n\n\n\n

Our take via people-process-technology lenses:<\/strong><\/h2>\n\n\n\n

People<\/strong><\/h3>\n\n\n\n

Business Stakeholders<\/strong><\/h4>\n\n\n\n

Business Stakeholders<\/strong><\/strong> need to pull their heads out of the sand and take a hard look at the data they want to use in the decision making. <\/p>\n\n\n\n

If their data infrastructure needs radical improvement (it does) then they need to put the money where their mouth is and sponsor this.<\/p>\n\n\n\n

Data Scientists<\/strong><\/h4>\n\n\n\n

Data Scientists<\/strong> need to accept the reality that in 99% of the cases right now the \u201cdata\u201d is not ready for \u201cscience\u201d. <\/p>\n\n\n\n

It actually is many miles away from it. <\/p>\n\n\n\n

Get down from your ivory towers and start delivering FAST using automated data engineering platforms instead of competing on Kaggle or your ground will be shaky soon. <\/p>\n\n\n\n

Some of you got hit already.<\/p>\n\n\n\n

Data Engineers<\/strong><\/h4>\n\n\n\n

Data Engineers<\/strong> can be the second to get some cheese by quickly upskilling themselves in Machine Learning and jumping on the already slowing Data Science bandwagon. <\/p>\n\n\n\n

It\u2019s not that hard and you will kick ass with your data engineering experience.<\/p>\n\n\n\n

Processes<\/strong><\/h3>\n\n\n\n

Delivering Data Science has been traditionally project-based. <\/p>\n\n\n\n

In the current day and age this design is obsolete. <\/p>\n\n\n\n

The need for fresh insights and models just jumped up a notch – while the need for analysis older than 2 months evaporated.<\/p>\n\n\n\n

The rapidly changing economy monitoring changes and extracting insights from the data should be a part of the fabric in the organisation \u2013 a part of a process, not a project.<\/p>\n\n\n\n

Technology<\/strong><\/h3>\n\n\n\n

In the rampant progress of automation – talking about the world needing thousands of new Data Scientists is madness.<\/p>\n\n\n\n

If the data side of the house is fixed and the right technologies are put in place then an average organisation only needs one Data Scientist for their data science efforts. <\/p>\n\n\n\n

It\u2019s a C-level role when a senior exec is responsible for organising the infrastructure to allow Business Users to interact with the data without the layers of people in-between.<\/p>\n\n\n\n

And this is what the new world will expect – being connected directly to the data. <\/p>\n\n\n\n

In real-time. <\/p>\n\n\n\n

With ease.<\/p>\n","protected":false},"excerpt":{"rendered":"

We took a look at how reality may change for the cosy world of data science and the Data Scientists working in it. “Despite thousands of bodies throwing themselves at …<\/p>\n","protected":false},"author":44,"featured_media":1606,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_eb_attr":"","_glsr_average":0,"_glsr_ranking":0,"_glsr_reviews":0,"footnotes":""},"categories":[39],"tags":[],"class_list":["post-1598","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data"],"_links":{"self":[{"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/posts\/1598","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/users\/44"}],"replies":[{"embeddable":true,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/comments?post=1598"}],"version-history":[{"count":4,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/posts\/1598\/revisions"}],"predecessor-version":[{"id":1628,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/posts\/1598\/revisions\/1628"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/media\/1606"}],"wp:attachment":[{"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/media?parent=1598"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/categories?post=1598"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aijourn.com\/wp-json\/wp\/v2\/tags?post=1598"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}