Post #45: The data deluge, revisited

What do Kafka, fabric and catalogues have in common? It’s nothing to do either with literature or with clothing. Rather, all three terms were used with liberal abandon at the Big Data LDN event we attended earlier this week. If you’re a geek, then they relate to a tool for streaming data, an architecture for providing consistent data capabilities and a metadata management service, respectively. Alternatively, if you’re a generalist like this author, then the most overwhelming conclusion we drew from the event was that we all know data is growing; the challenge lies in how to manage it to be relevant.

Big Data LDN is an annual event (we also attended in 2018) where over 8,000 attendees spent two days meeting with more than130 different businesses all of whom are seeking to provide answers to (varying aspects of) the above challenge. There were also a series of keynote speeches from the likes of Google, Microsoft and IBM (Google’s, on intelligent data science, was the most interesting). In terms of what we learned:

“We’re not even halfway there.” This was a quote from an executive whom we met referring to the data journey which most businesses are undergoing. Historically, companies have used software to improve outcomes; now, they are ‘becoming’ software. Think about the paradigm change underway, say, in ride hailing (Uber, Lyft etc) or in getting access to a loan. For this journey to reach fruition, the prevailing view from the event is that data needs to become more dynamic; or the tools used to leverage data need to work harder. Machine learning and artificial intelligence should not be seen as ends in themselves; more as means to an end. There was a strong and consistent view that (human) analysis and interpretation remain crucial as much as anything to avoid unconscious bias. 

Against this background, many participants with whom we spoke highlighted a profound shortage of skilled data personnel. IBM noted that some 500,000 IT jobs in the US are unfilled simply because candidates lack the right skill sets. Training, therefore, remains paramount. One other topic that emerged in many of our discussions at the event related to data sovereignty. With increasing regulatory scrutiny (think GDPR in Europe from one perspective, or suggestions that the mega-cap tech companies in the US should be broken up from another), every business needs to take more responsibility in terms of data management. Securing data effectively and ensuring its privacy remain paramount.