Tapping into dark data for efficiency, innovation, and income

Competing with data is nothing new. Startups and industry incumbents have long known that harnessing the power of proprietary data sets can be a sustaining source of competitive advantage. However, as many data sources become commoditized and the tools to manipulate data at scale become cheaper and more accessible, the rules of competition are shifting. Companies will need to become more creative in where they source data from and how they use it. Dark data presents an exciting opportunity for early movers to secure a competitive advantage over slower rivals.

A tsunami of data

Research commissioned by Seagate and conducted by IDC finds that enterprise data will grow at a 42% annual growth rate over the next 2 years. However, only 32% of this data is being used effectively, with more than two-thirds sitting idle in silos around the organization. Gartner defines this “dark data” as information assets collected as a byproduct of normal business activities but that firms fail to use for other, potentially income-generating activities. Much of this data is stored for compliance purposes with the cost of secure storage creating an expensive and unproductive overhead.

A sizeable proportion of this emerging data challenge will come from the proliferation of enterprise IoT initiatives. IDC predicts that by 2025 there will be 55.7 billion connected devices worldwide. Seventy-five percent of these will be connected to an IoT platform, generating 73.1 ZB of data annually, up from 18.3 ZB today. Usually collected for specific applications within the firm, IoT data could also be leveraged for other purposes.

Raising efficiency with dark data

With no shortage of data to work with, the challenge for data-savvy enterprises is finding ways to use it effectively. Initially, this might be by improving existing products and services as well as streamlining business processes. For example, through analysis of staff data relating to demographics, salaries and transportation options, airline catering provider, Gate Gourmet realized that distance from employees’ homes to the airport and their transportation options were major factors in staff turnover. A tighter focus on these factors in the hiring process improved staff attrition rate by 27%.

In another example of leveraging internal dark data, a large European chain of 27 hotels comprising 13,000 rooms analyzed usage data from their guest Wi-Fi networks to improve customer service. The data had been collected for several years but not been evaluated for potential business value. By tracking the data over time for variables such as total devices connected in each hotel, total devices per area such as restaurants or swimming pools and average time spent in each area, considerable improvements were made. A problem with excessive waits in check-in and check-out areas was found and rectified as well as better allocation of staff to sections of the hotel at key times.

Leveraging dark data with data exchanges

While opening up silos of dark data within an organisation can surface valuable and unused assets, it is often necessary to bring in external data to maximise returns. Alternatively, while the organisation concerned may find no use for the dark data it holds, others might. To realise these benefits, data exchanges have a vital role to play. Accenture believes that data exchanges in the IoT space will unlock $3.6 trillion of value by 2030.

This could include data from sensors in offshore oil rigs that could be purchased by meteorological agencies to improve predictive weather models. Data from air-conditioning units in large buildings could be sold to real estate developers trying to secure a competitive advantage for their new development projects. High-definition surveillance videos in busy town centers can be analyzed at speed and scale through AI to help designers spot emerging fashion trends before they become mainstream.

Data exchanges can provide the liquidity for enterprises and start-ups to combine previously disparate and undervalued data streams to create innovative new products and services. The winners will be firms with an open mindset and a willingness to experiment. Before Google launched its AdWords service in 2000, the data exhaust from web searches had little perceived value. By placing contextually relevant ads around search results, a $135 billion business was born and the advertising sector was changed for ever.

First steps in harnessing dark data

Although much easier said than done, creating a culture of data within the enterprise is the most important step in profiting from dark data. According to the Harvard Business Review, this must start from the top with managers establishing the principle that decisions must be anchored in data. Firms with a data science team need to ensure they are not hidden from the core operations of the business but integrated across departments. This will help educate teams in some basic principles of data management but also will give the data scientists a better view of where data silos may be hidden away.

Ultimately, companies need to think beyond their traditional lines of business and backoffice processes if they are to harness underused data assets. Those that apply an open and creative mindset to finding uses for dark data are likely to find some very profitable diamonds in the rough.