Midsize business leaders are right to be excited about the opportunities to harness value in their large data sets. But data in midsize businesses tends to be messy: spreadsheets and plain text files, many in different formats, are difficult (if not impossible) to integrate. It takes a lot of time and money to clean it up and make it useful. Disintegrated, poor-quality data can sabotage even the best of initiatives, including AI designed to increase value and efficiency. HdL Companies, a government services company based in Brea, California, used their data strategically and saw significant efficiency gains. The author offers three lessons that leaders need to consider when starting to automate data analytics.
As midsize companies grow, they develop data streams and data lakes (repositories for structured and unstructured data) that are too large to be manipulated and used effectively by a single person, or even a team. And even if a company is currently deriving value from its data, the people doing the work could move on, leaving the company to find, attract and hire expensive data analysts in a hurry.
Having a capable and up-to-date Enterprise Resource Planning System (ERP) will not solve the problem or relieve the pressure. Most midsize companies start with finance-centric ERPs and end up installing systems to store other data, such as customer activity and production throughput, which is more operational than strategic.
As a result, automating data analysis as your business grows is a lot Very good idea. Automation is often where programmers write algorithms that perform previously manual tasks as directed. Doing so quickly pays dividends, drives innovation and greater growth, and paves the way for the implementation of artificial intelligence, which makes practically everything simpler, more efficient and cheaper. AI is coded to learn how to perform a task, in a sense by inventing and writing its own algorithms.
But data in midsize companies tends to be messy. Spreadsheets and plain text files, many in different formats, are difficult if not impossible to integrate. It takes a lot of time and money to clean them up and make them useful. Disintegrated, poor-quality data can sabotage even the best of initiatives, including AI designed to increase value and efficiency.
As Joe Pucciarelli, Group VP and IT executive advisor to market research firm International Data Corporation (IDC) said in a recent Channel Company webinar, “The datasets of most organizations are not in great condition. We talk about data and analytics as a strategy and a priority, but the data isn’t ready to support it … Most organizations, when they try to solve a problem, the analyst who is working on it typically spends over 75% of the time … simply prepare the data.
As you can imagine, the ROI on the time spent doing it is not good. Let’s take a look at how a midsize business leveraged the value of their data and explore the three steps midsize business leaders can take to do the same.
How a midsize company handled its data
One of my clients, HdL Companies, a government services company based in Brea, California, is commissioned by the municipalities of California, Texas, and other states, to analyze the distribution of sales tax revenue in their respective states to ensure that their city or town is receiving its fair share. HdL looks for misallocations and discrepancies that councils may indicate when petitioning the state for redress. The heart of this work is to compare different databases to expose discrepancies that affect who should get the sales tax revenue. For example, in one database a company might be listed in Dublin, California, but in two other databases it might be listed in nearby Pleasanton. This makes a tax allocation error very likely; HdL’s task is to find it.
California’s 40 million residents purchase taxable products from 5.9 million licensed retailers, creating a massive dataset of nearly 46 million tax records in 2020. For years, HdL has employed analysts to review that data every quarter, looking for of errors. HdL’s IT group has created software to help, but over the years its analytics team has adopted many idiosyncratic manual techniques, and the IT group has had a long backlog of work to continue building the code base to include those techniques. . Coping with the backlog was delaying HdL’s automation projects and the development of new techniques to bring out tax discrepancies more efficiently. At the same time, the state of California was making its own improvements, leaving fewer discrepancies noticeable using old HdL tools. “Our team is always looking for new analytical techniques to identify hard-to-find misallocations,” says Matt Hinderliter, director of audit services at HdL. “However, we relied heavily on manual exports and data manipulation in Excel, as well as the need for senior analysts to manually review spreadsheets that often exceed 70,000 or 80,000 rows of data.”
To address both external (California improvements) and internal (IT department overloaded with HdL and laborious manual analysis) stressors, HdL, a mid-sized company with a mid-sized budget, hired a talented intern who was earning a full-time master’s degree in data analytics. She was able to transform some of the analytical processes used by team members to identify potential misallocations into algorithms that could generate more tax revenue reallocation opportunities in a fraction of the time.
Given this increase in efficiency, it could be assumed that HdL is considering layoffs. Instead, its audit department is preparing staff to pursue any opportunities that have emerged from the automated analysis. And HdL has moved closer to focusing on AI implementation and rollout.
Improving operational efficiency is almost always a top priority for midsize businesses. In a Channel Company survey of mid-market IT leaders, 75% of whose companies have revenue between $ 50 million and $ 1 billion, 58% of respondents said their top priority was improving efficiency. operational. This far exceeded their second priority, increasing new revenue (36%). Both objectives can be supported by automating data analysis, as was the case in HDL.
Medium-sized businesses cannot seize every opportunity. Their budgets, workforce and frenzy of daily operations will not allow it. (I’m not Google, after all.) So midsize companies should start automating their analytics processes by focusing on areas where critical operations are inefficient or too dependent on one person or a few people. Before automation, HDL predicted that 15 people spend a significant portion of their time doing what algorithms are doing today.
HdL was already doing the work on the data; many companies – printers, plumbing suppliers, and so on – are not. But those companies are still accumulating data and can benefit from its strategic use. It is important to start with a solid foundation. Here are three things leaders need to consider when they start automating data analytics.
Prioritize cleanliness. Data in a midsize business is typically messy and needs a lot of reordering before it can become useful. Another key activity is identifying which data is important and then deleting it. This can be slow work at first and isn’t inexpensive, so find areas where the company can recover a refund in the first year. This will turn the skeptics into believers.
Hire the right people. Executives are not analysts. They lack the time, patience, and skills to perform data analytics as an addition to their daily activities. Business analysts are partly programmers, partly businessmen. HdL started with an intern and hired her as a full-time business analyst.
Prepare the data. Only when your data has been thoroughly prepared can you start thinking about AI. AI creates its own logic from an analysis of the patterns it discovers in the data. While AI and machine learning are useful and exciting, both technologies need large data sets to train on, with confirmed positive and negative results. After proper data cleansing and some algorithm-based analytics, most midsize companies will have a sufficiently large and useful dataset on which to train an AI model.
Midsize business leaders are right to be excited about the opportunities to harness value in large data sets. Now is the time to begin this multi-year journey and commit to hiring the right talent, taking incremental steps to produce value from data automation and other types of advanced analytics.