Most IT solutions fail at some point due to unforeseen complications. Business Technology’s resident U.S. blogger Keil Hubert argues that it’s irrational to expect IT people to be able to accurately predict the future when it comes to technology.
I ran into one of my former network managers the other day. I asked about his new position, and he told me that he was worried about one of his storage arrays; a couple of drives had failed, and he suspected that the entire shelf was going to fry. He couldn’t get it replaced, though, because he couldn’t convince upper management to release funds for a replacement unit. After all, the logic went, it hasn’t failed yet, so why should we pre-emptively pay to replace it before it fails? How do you know that it’s dying now? The tech and I shook our heads and I clapped him on the shoulder. We both know that he’ll take all the blame for the systems outage when it inevitably happens. IT can be a rough business, in large part because we’re irrationally expected to predict the future with perfect accuracy.
As much as we try to market IT services to the rest of the enterprise as a stable utility, it isn’t. IT solutions are comprised of hundreds of interdependent, man-made components. Try as you might, perfect stability is a goal, not an achievable state. Stuff breaks. Stuff always breaks. The only question worth asking is ‘when?’
That IT is critical to the functioning of the modern business is no longer in question; what remains open to debate is whether the IT team has any bloody idea what in blazes is actually going on inside their data centre from day to day. Modern, enterprise-grade systems are growing so complicated and queerly interdependent that new systems rollouts are often based more in irrational faith than they are in engineering. [1]
That’s not a failure of staff; it’s a reflection of what they have to work with. Hardware and software components develop haphazardly. As any given product reaches the market, it will usually solve one thorny problem – and at the same time introduce a bunch of new (and unforeseen) conflicts with other systems. What worked fine for one combination of hardware, software, and settings might totally come undone after a change to any random component. Manufacturers and developers routinely introduce new features and ideas that the market isn’t ready for and for which no agreed-upon standard exists. Gambling on the deployment of cutting-edge IT solutions can be like gambling on the thickness of interior walls by sprinting into them headfirst. You couldachieve a remarkable breakthrough … but it’s hard to predict just how traumatic it’ll be getting there.
As IT professionals, we’re supposed to solve our companies’ business objectives with information systems that achieve meaningful results for the company at an acceptable cost in blood and treasure. We’re supposed to make problems go away, not introduce new problems that require ever-more complicated (and expensive!) ‘fixes.’ Unfortunately, tech deployments that achieve all of their desired objectives on-time and under-budget are heartbreakingly rare. Tech deployments that remain relevant and functional over the life of a five year plan are ever more rare. That’s because we simply cannot predict how any given component will actually perform over time in an operational environment. When we tell upper management that “everything related to solution X will work out fine,’ it’s more often a desperate prayer than a reliable statement of reassurance.
To reiterate: we can only guess at how things will play out.
Knowing this, it’s no wonder that non-IT executives are often leery of committing funds to major IT projects. Once the IT team gets a business process correctly automated, upper management’s inclination is to leave it well enough alone – in local parlance, ‘If it ain’t broke, don’t fix it.’ The more conservative a business unit is the more likely it is that a minimally-acceptable production system will be required to soldier on unmodified forever. The business teams loathe the threat of an upgrade-induced production outage. Honestly, so do we. If we could get away with leaving a working solution running undisturbed, we would.
Unfortunately, components go obsolete every day. Hardware makers change designs, stop manufacturing spare parts, and simply go bankrupt. Operating systems reach the end of their life. Applications stop receiving upgrades, security patches, and support. Developers introduce changes to their code to make their products work on new generations of hardware, only to have the changes break their code on older hardware. The data centre is never static; elements of it keep changing, much to our (and to our users’) endless frustration.
What’s hugely frustrating is that all of this deterioration happens on its own, before you start talking about exploits and hacks. Patching has to happen to remediate vulnerabilities, and every patch threatens to break something new. As one of my techs complained, the IT profession is like trying to overhaul an airplane’s engine while the plane is in flight … and is taking fire from enemy fighters.
So, drama. Unless the systems architect can accurately predict the future, there’s no way to guarantee that IT operations team will be able to keep her solution running for any length of time. The good architects will try like hell using every clue and scrap of experience that they have at their disposal, but they’re not superhuman.
This is why IT leaders tend to guzzle antacids straight from the bottle. We’re expected to be able to accurately predict the unknowable future, and we simply can’t. It’s maddening.
We’re expected to guess how long our business units will depend on a given solution. We’re expected to know how long every single component of an integrated solution will function before breaking down. We have to know how long each element will be supported by its manufacturer. We’re expected to predict how many and what sorts of failures our equipment will experience so that we can buy exactly the right quantity and type of emergency spares. We’re supposed to know which direction the industry will take, and to design our solutions to be ‘future proof’ against ‘standards’ that can be superseded or abandoned on a whim.
Being the head of IT is often a no-win predicament: you do the best that you can to guess how the world will change, hedging your bets all the while. You get chastised for building time and financial buffers into your deployment plans and support budgets to mitigate potential emergencies, and then you get blamed for not having laid in adequate resources to recover from a failure that nobody could have predicted. You cannot win, but it’s not for a lack of trying.
It should come as no surprise to anyone with more than a fortnight’s experience in the IT trenches that non-IT-savvy executives don’t want to hear any of these points. When it comes to the Top Desk, the business discovers a problem and the powers-that-be impatiently want IT to solve it for them. The executives demand from us an accurate estimate of what our proposed solution will cost to deploy, how cost-effective it will be to maintain, and how soon it can be put into production. When the head of IT says ‘it depends,’ she’s not being obtuse or hostile; that’s simply the unpleasant truth. Any confident answer the she gives will almost certainly be crap and vapour. The best that a responsible IT lead can give you is a reasonable prediction based on her past experience and shrewd guesswork.
The core takeaway of this message is that we IT folks do our utmost to satisfy business needs and we strive earnestly to control for future environmental changes, but we’re not infallible prophets. If we could accurately predict the future, we never would have wasted company resources on MySpace pages, BlackBerry handsets, or OS/2 “Warp”. Many IT solutions seemed great when they were new and original, but they failed to hold up over time … and now we don’t want to see or hear any trace of them in the office. That’s not a negative reflection on the solution designer; it’s simply a sign that the future is largely unpredictable.
All that being said … If you’re a CXO, please cut your IT team some slack. They’re trying like hell to do right by you and by the business, but they’re only human.
[1] If our ‘cloud-enabled global post-modernist’ applications continue to evolve technically the way we’ve seen them evolve over the last decade, we’ll likely bring the Warhammer 40,000 vision of IT to life long before we’ll need tax discs for our flying cars. Imagine, if you can, robed sysadmins chanting prayers (in distribution-specific command-line syntax) and swinging incense censers in front of the cooling intake vents of racked servers that are so old and mysterious that no one dares replace a blown power supply for fear that the moon will implode.
