I am not – and never have been – a fan of metrics. I understand their potential utility; I’ve been happy to gather, analyse, and leverage data when and where it made sense to do so. That said, I’m firmly opposed to useless metrics. Most data crunching in the modern enterprise, I believe, is just that: useless. I suspect that most data is compiled, solely for the sake of having it even though it doesn’t serve a realistic purpose for the work centres generating it.
I chalk this up to the Demingite Cult from the end of the last century. If you haven’t heard of them, let’s recap: Dr. W Edwards Deming was an American engineer who became famous for his work in post-war Japan where he evangelised quality control techniques that greatly increased Japanese manufacturers’ efficiency and profitability. Deming became a celebrity in American business circles after Ford hired him in 1981 to turn around their failing business; after five years of following Deming’s advice, Ford became the most profitable car manufacturer in the USA.
After the “miracle at Ford Motor,” it seemed like every MBA in America was indoctrinated into the glorious revelations of Saint Deming. His treatises on quality control were everywhere, even crossing into popular culture. Best of all was his most famous commandment: “If you can’t measure it, you can’t manage it.” I have yet to serve under a manager, supervisor, or executive who didn’t quote that scripture to me with messianic fervour.
Deming’s aphorism is meant to convey the idea that managers must measure all aspects of their work processes and then scrutinize that data to find evidence of flaws and inefficiencies. The act of studying one’s work objectively will allow a leader to dispassionately improve quality by lowering preventable defects. It will also give managers the ability to predict problems by noting trends. This idea, I allege, gave American business its obsession with collecting and publishing metrics.
There are only two problems with the Holy Words of Saint Deming. First, he never said that! Deming’s actual quote from his 2000 book The New Economics for Industry, Government, Education was “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” The Deming Institute itself wrote a clarifying essay on this misinterpretation in 2015, saying:
“…when people see a quote emphasizing the importance of data attributed to W. Edwards Deming, it seems sensible that he said it. And it is likely shared so often because people notice that their organization is flailing away when they would benefit from using data to improve their management of the situation. So the quote appeals to people who think that their organization fails to use data when they should be using it.
“Dr. Deming did very much believe in the value of using data to help improve the management of the organization. But he also knew that just measuring things and looking at data wasn’t close to enough. There are many things that cannot be measured and still must be managed. And there are many things that cannot be measured and managers must still make decisions about.”
Second, I argue that metrics are employed in far too many situations where they serve no practical purpose. The collection and “analysis” of valueless data wastes time and drags down productivity even while management liturgy promises exactly the opposite.
As a practical example of this, I offer the tale of Division Manager Bob. [1] Bob was one of five division managers in a medium-sized organisation. He supervised more departments than any of his peers and – as the support division – had a wider range of functions than did all of his peers put together. This made it challenging for Bob to burnish his brand with the senior executives.
To cope, Bob launched a formal metrics program. All his departments were required to collect data on their core processes, convert the data into graphs, and brief their findings monthly. Bob then cherrypicked certain visually appealing graphs and presented them as “proof” of his outstanding leadership. Note that neither his peers nor the executives ever asked Bob for this.
The joke was the data being collected was largely irrelevant. Sure, the personnel department could reliably collect numbers on how many new employee ID cards they’d printed while the physical security department could publish how many fingerprint cards they’d processed for background checks. Those figures didn’t mean anything. If the data showed 20 cards in February and 30 in March, who cared? Nothing would change as a result of knowing “the numbers.”
Then came us. Bob demanded that the IT department track data to show how “productive” we were, so we gave him our server uptime: 100%. That wasn’t good enough, as it didn’t show variation. Bob demanded that we track Help Desk trouble tickets. That didn’t work either, as we didn’t record trouble tickets for a job that could be solved immediately; we only used tickets for knowledge transfer between techs. Frustrated, Bob created his own metric: customer contacts. All IT personnel would record every customer interaction we had, every day, no matter the context.
The way it worked was that each worker had a worksheet. Every time they spoke with a user about anything related to IT services, they’d add a tally mark in a table with “work type” along the X axis and “contact type” on the Y axis. So, a Help Desk tech might place a dozen marks for “PC support” in the “face-to-face” cells for walk-up customers … as would the phone tech who was asked a random question by the bloke at the next urinal (after he washed his hands and returned to his office, hopefully). Literally any technical question “counted” as a “contact.”
This data helped us create some complex graphs but didn’t serve any purpose. Bob insisted that we would, eventually, be able to see trends form through extrapolation. I countered that we already knew what caused surges in tech support calls: new PC replacements, software patches, power outages, and users taking long vacations between Thanksgiving and New Years. Bob countered that with “enough” analysis, we could gain insights into heretofore unimaginable “secret” trends. By watching how the lines on the graph undulated over time, we might realize that an unexpected surge in support needs was coming … so we could prepare for it!
I acidly countered that even if our “Predictions of Operational Performance” (or P.O.O.P. if you’re into acronyms) afforded us some mystic insight, that foreknowledge wouldn’t make a meaningful difference. We could process wheelbarrow loads of P.O.O.P. but we couldn’t change our staffing authorizations. We had 15 employees working in 8 different specialities; the organisation didn’t allow us to hire more. We already had overtime and comp time systems for surge management. We already had the authority to cancel vacation time requests and bring in off-duty workers for emergency operations. What, exactly, was the P.O.O.P. doing to do for us?
Bob sputtered and raged, but he lacked an answer. In his MBA worldview, data was valuable and meaningful simply by virtue of existing. Bob believed in a sort of Schrödinger’s Parthenogenesis, where raw data would spontaneously birth meaning and value the moment that a manager looked at it … which completely missed the point of everything Dr Demming had written.
Simply collecting data and analysing it alone doesn’t create new cosmic understanding. Predicting problems such that you can act pre-emptively to mitigate requires a deep understanding of your processes, an unbiased understanding of past mistakes, and a grasp of how both internal and external forces affect your functions. “Professional experience” by any other name.
Sure, there are valuable metrics. I’d be a damned fool to deny it. Our IT department tracked CPU load on our hosts, network saturation on our switches, and data storage stats on our SAN to decide when to upgrade or replace our equipment. We used other departments’ operations plans and work forecasts to decide when to schedule downtime for maintenance. We used repair costs by device type to decide when to cut our losses and chuck out a bad model of PC. Those metrics made sense for us, but weren’t useful for Bob because he didn’t understand what they meant.
Why? Two reasons: first, Bob wasn’t qualified to make decisions based on that data. He knew nothing about IT (let alone enterprise IT), so his “input” could only harm us, never help. Second, those metrics didn’t serve his overall objective which was to paint a picture for his superiors about what a great job he was doing. The IT department managed IT technologies and services, while Bob managed … budgets, paperwork exercises, and graph making. The parts of our day that Bob controlled had nothing to do with the operation and efficiency of IT services. By design.
Bob wasn’t having it. He needed processed data to tell his story via colourful graphics and back his claims up with ostensibly ‘objective” evidence. So, we dutifully filled out our personal “customer interaction” tally sheets for about two years and processed the data to give Bob his monthly bucket of P.O.O.P.. He went away happy to play with his graphs while we got back to work.
And that’s the core problem, isn’t it? I’ve found that most actionable metrics systems are the ones created by a process owner to measure their own performance. When you understand your job, you can measure it to confirm or refute your suspicions. That said, most of the useless metrics systems I’ve encountered have been imposed by an outside entity onto the process owner. Bob’s example of higher management demanding meaningless stats is a great example. So too are “performance figures” required by Six Sigma teams, marketing groups, bean counters, auditors … really, anyone that doesn’t understand a process and tries to get a handle on it using their own frame of reference so they can then judge and criticize it. Never to improve or support it.
I’ve seen this trend manifest in every organisation I’ve served in since the 1990s. Every business has its Deminginte true believers who divine the future from the eldritch signs and portents they interpret in holy spreadsheets. I can sort of empathize. It would be cool to accurately predict the future. The inescapable flaw in these zealots’ belief system is that meaning doesn’t cohere from data no matter what style of numerology one applies to a bit bucket. Most data is garbage, therefore most metrics are garbage too. Actionable meaning comes from a nuanced understanding of one’s business processes, operating environment, and key supplier relationships.
In my experience, folks like Division Manager Bob have reflexively and violently rejected that heretical view because it implied that they couldn’t manage their subordinate functions without first understanding how to perform those functions … which is 100% correct. You can’t directly manage something as complicated as IT without experience doing IT. You can certainly make decisions, sure, but that requires you to trust your IT staff to know what they’re doing. A fellow like Bob couldn’t accept that. He demanded that we provide him the ability to make decisions for the IT department about how to improve IT functions without knowing anything about IT … and he felt he could do just that thanks to the sorcery made possible by large collections of numbers.
It was no use arguing with the man. We let him play with his P.O.O.P. He wasn’t any smarter or more useful, but he went away happy. To be fair, though, Bob’s constant recitation of not-Deming’s aphorism was accurate: if he couldn’t measure it, he couldn’t manage it. Of course, even when he could measure it, he still couldn’t manage it … but we politely kept that to ourselves and got on with the job.
[1] Not his real name, as per tradition.
Pop Culture Allusion: none this week
