The Worst Data Project I Ever Saw Was Perfectly Well Run...
There are three ways a mid-market business usually goes about doing more with data. Here's how each plays out, what it really costs, and what to watch for whichever route you pick.
By Graham PerryEveryone did their job properly, but that was the problem I think…
A few years back I sat with an MD of a facilities business turning over somewhere near £40m. It was a very good business, growing nicely tbh. He'd spent eighteen months and a genuinely eye-watering sum on a ‘data programme’ as he called it (my guess is that crept in from the consultants), and he walked me through it like a man showing you round a house he'd just realised he couldn't afford 🫠. The warehouse was built, the pipelines ran overnight (mostly) without falling over. There was also a data engineer, an analyst, and a BI developer who'd been hired six months in when it turned out the first two couldn't do everything.
Then he showed me the dashboards, I think there were forty-something of them.
I asked which one he opened on a Monday morning and he laughed at me, and it wasn't a happy laugh ‘none of them, I still ring Karen for an update’.
That's the bit that stayed with me. Nobody had failed at all; the engineers had built what they were asked to build, the consultants had delivered against the statement of work and every milestone had been signed off by somebody senior but… at the end of it, the man paying for it all still picked up the phone to Karen, because Karen actually knew what was going on.
I've since come to think there are basically three ways a mid-market business can start this journey, two of them we see go wrong regularly & the third is the one we ended up building a company around, which I'll admit is not the most impartial position to argue from 😂 so take that with whatever size pinch of salt you think it deserves.
Approach one: go big
Find the money, hire the team, buy the software, and back yourself.
The appeal is obvious for this one, you own it and it’s the way the big boys do it. Tried tested and often very effective. Nobody's holding your data hostage, you can build exactly what your business needs rather than what a vendor thinks businesses like yours need, and if you're honest, there's something in it for the ego too 👀 - a data function on the org chart looks like a company going places.
Here's what I'd want anyone considering it to understand. The software is the cheap bit. A data engineer, an analyst and a BI developer, on-costs included, will be comfortably pushing you past £150k a year before anyone's licensed anything. And the day one of them leaves, and one of them always leaves btw, you don't just lose a person, you lose the only human being who understood why the overtime figure was calculated the way it was. I've watched that happen too many times, the knowledge walks out the door and eight months of work quietly stops being trustworthy.
The deeper problem is what happened to my MD with the forty-odd dashboards. When you hire a data team, they will build things, that is what they are for and that is how they demonstrate value. If nobody has told them precisely which four or five numbers change how the business is run, they'll build everything, because building everything is safer than guessing wrong (strange logic but true, I promise). The result looks impressive in a board pack and gets used by… you guessed it, nobody.
For a business over about £250m with the volume to justify a permanent function, going big can be the right call. Below that, are you buying a full Formula One team to do the school run? Maybe.
Approach two: buy it as a project
Bring in a services firm to get from A to B. Buy the software and hire a junior analyst to keep the lights on afterwards, best of both worlds, right? … guys?... right?
I understand the logic completely and the first half often works for a while. A decent consultancy will come in, do good work, connect your systems and leave you with something that genuinely wasn't there before. I've seen it done well, no question, but the problem has never really been the project, the problem is day 91.
Because the consultants go home, they have to of course, that's the model. But, what you're left with is a system built by people who understood it deeply, now in the hands of one person, usually early in their career, almost always the cheapest hire you could justify because the budget went on the build (or even cheaper, repurposing ‘spreadsheet Tom’ to being your ‘analyst’). That person is bright and willing, in many ways capable and… completely unsupported. They're being asked to maintain a data architecture, respond to change requests from directors, and somehow keep pace with a business that reorganises itself every eighteen months.
Then your payroll provider changes their export format for fun, or you win a contract with a client who wants their own reporting cut slightly different to the current way it’s done, or the FD asks a question the model was never designed to answer. Any one of those could be a two-week job for someone experienced and a two-month job for someone who's never done it before. Meanwhile the reports drift slightly out of true, and the thing about slightly-wrong numbers: nobody actually announces they've stopped trusting them. They just quietly drift back to Karen.
I met an operations director last year who described her situation better than I could. "We spent forty grand and now we've got a really expensive spreadsheet that one guy understands."
Approach three: Subscribe to the whole department, not the tool
There's a third route that's become viable in the last few years, and it starts from a different question.
Instead of asking what you need to build, ask what you'd need to have permanently. You need people who know data, you need the platforms connected and you need a process that keeps working when the business changes. What you don't need is to employ all of that, own all of that, or carry the risk of all of that on your own balance sheet.
So you subscribe to it -People, platforms and process, on an ongoing basis, for a fraction of what the team alone would cost. Someone else carries the risk of the analyst leaving. Someone else deals with the payroll export changing format. When you restructure your contract portfolio, the reporting follows you, because the people who built it are still there and still on the hook.
That's what we do at Datore, so yeah, of course I’m biased, but I'd make the same argument to someone who never spoke to us, because the concept matters more than who provides it:
All the benefits of doing it in-house
retained knowledge, right skills for right job, modern and evolving technology stack, on-going support for a changing business landscape.
Without the headaches of doing it in-house
recruitment, technology selection, salaries, lengthy projects, massive budgets.
HTS Group (Property & Environment) went down this route, and this is how it turned out for them:
"It simplified the analysis for us. Rather than looking at loads of numbers, it gives a quick snapshot without needing to know what the data should look like." ~ Shakeel Khan, Head of Environmental Services
Four things I'd watch for whichever route you pick.
Start with the business, not the systems.
The first conversation should be about where you're trying to get to, not what's in your finance system that we can use. We run a workshop before we look at a single system, and it's a pure business conversation. The number of times a leadership team discovers, out loud, in the room, that they don't agree on what good looks like, or on what their top 5 strategic objectives are - that's worth the morning on its own.
Insist on a small number of metrics.
Four to eight. If someone offers you a dashboard with all your data on it, they've misunderstood the assignment. Forty dashboards is not forty times better than four, it's a filing cabinet nobody opens.
Make somebody accountable for the answer, not the build.
Delivery against a spec you defined last year is not the same as the MD actually opening something on a Monday morning for the next 3 years. If nobody's contractually on the hook for the second thing (actually making sure that the dashboards get adopted and become valuable) you'll get the project delivery and a bill (a big bill)
Look hard at the ongoing part.
Everyone's proposal looks good for the first ninety days. Ask what happens in month fourteen when your operations director leaves and her replacement wants to see the business differently. The honest answers to that question are the ones worth listening to.
I still think about that MD and his forty dashboards. He wasn't unlucky and he hadn't been mis-sold. He'd made a reasonable decision with the options in front of him, and the options in front of him weren't good enough.
That's changed. You already know your business needs better data, or you'd have stopped reading four hundred words ago. The question worth sitting with is whether the way you're about to go and get it will still be standing up in three years, or whether you'll be the one ringing Karen.