The Cost of Accuracy

The other day I was on a call with Finance and another VP, on the subject of trying to find a better way to report headcount numbers to Finance. It seems that every month Finance was asking this VP to give them the headcount of people that worked under him. This was not just some number of people employed under him. They needed him to account for who was out on PTO or FMLA or maybe just started working a few days so they didn't really count.

The VP admittedly was not using any rigorous means to get this number. He more or less just did an educated guess. He had called this meeting specifically because he figured there had to be a better way to get it. When the call started, the VP and his team pulled up this dashboard they built, which showed the number of new files that had been assigned during the month by employee. The idea was that employees who received new files meant they were in rotation and thus were working during that time.

Well the Finance team listened and kind of stalled here and there on whether this would be a suitable proxy for the numbers the VP had been giving them. There were some individuals on the list that should be removed because of some reasons and so on.

Being new to this topic, I asked the question to the Finance team. "What do you need this number for?" Pretty quickly, they pulled up their "dashboard" which in Finance speak means highly formatted Excel spreadsheet. To summarize, each month the Finance team is gathering individual metrics like file counts (from one of our reports) and headcount (from our VP) and pasting them into this spreadsheet. Then generating ratios and charting them. And there is a lot of different metrics on this wall of numbers presented in typical Finance fashion.

It's pretty common to see them present numbers horizontally. With a bunch of metrics listed down the left, and an ever growing number of columns for each month stretching out to the right. It makes little sense from a data management perspective to constantly add new columns, but I guess it looks better on a wide screen.

Anyway, they sounded pretty confident about this tool they built that they manually update every month by copy/pasting/keying individual metrics. So I asked them them a follow up question. "What do you use this ratio for?" (The ratio built from the headcount numbers the VP was providing). Their response was that they are just tracking it right now, it's not really used for anything yet.

...

So not only are they painstakingly compiling all these numbers one by one into this spreadsheet every month, but they are making this VP spend hours each month to give this number to them. A number that they are not even really using for anything yet. An unfortunate waste of time on both sides.

I probed a bit more to better understand what exactly they were trying to track, which turned out to be assessing some measure of average workload or efficiency (the conversation kind of went back and forth on that). I proposed instead that we could very easily give them an automated report/dashboard from one of our existing datasets that showed average pending workload per employee. But again, the Finance team said they needed it to be more accurate; pick out certain individuals or cases where the person wasn't really working the whole month.

When I asked them how many days they needed to work in order to count, they couldn't really give a clear answer. More or less it was just based on the VP's gut feel.

When you work in data long enough, you observe this behavior. The notion that touching/massaging the data somehow makes it more "accurate". Picking out the odd cases, trimming the outliers, adding in the extra bit of personal knowledge that is not in the data. It gives a sense of perfection. I should know, I've been guilty of it myself.

The reality is that this perfection comes at a cost. Besides the human cost of having to employ someone to manually update and apply these changes every single time you run the numbers, there's also the fact that it will be literally impossible for anyone to independently verify your results. These numbers will inevitably move up the chain to the C-suite and they won't reconcile to anything. People will question whose number is the right number.

In the particular case of this ratio, putting aside the fact that the value is inherently wrong anyway, since it is dependent on the VP's educated guess for the denominator, the impact of adding/subtracting a few people had no material change to the final ratio. In truth, the value of the ratio itself didn't really matter. All that really mattered was what direction the ratio was changing. If the ratio is trending down, that's good, our staff are not overworked and we are more efficient. If the ratio is going up, then workloads are increasing and we might need to add staff.

Those are the kind of decisions that are trying to be made here. It doesn't require a number that is 100% accurate.

Which I guarantee even if it wasn't being plucked from the mind of the VP, it still wouldn't be 100%. Even the cleanest data is messy and noisy.

All that is required is a precise number. A value that is determined consistently and reliably. Then you can get directional trends much easier and cleaner. You don't have to manually enter them into a spreadsheet, they can easily be automated. You're saving yourself time, the time of others, and your results can be replicated.