In the ten years Anitian has been working incident response and digital forensic cases, our analysts have observed many ingenious ways to break into systems and applications. However, they rarely encounter an ingenious root cause for the vulnerabilities. After all the analysis and disk imaging is complete, the root cause is almost always the same: laziness. More specifically, laziness on the part of the IT team to properly manage, maintain and patch systems.
IT laziness does not happen overnight. Like most risks, it builds slowly over time. Which is why it is easy to ignore until there is an incident. Lazy IT teams, like the problems they precipitate, have their genesis in another slow-acting, pervasive problem: bad IT leadership.
Bad leadership begets lazy IT teams. Lazy IT teams beget poorly managed environments. Poorly managed environments beget bad security. And we all know what bad security begets.
A few years ago, I watched the CIO of a fairly large company dress down one of the IT administrators because a hard drive had failed in a RAID array. He chastised him for taking the system offline during the day and jeopardizing their business. I kept quiet as I watched this incident, but I felt bad for the IT guy. Why is it his fault that a piece of equipment failed? And why is the manager so focused on who he can blame and not how to solve the problem?
This was a typical bad manager move. This CIO, like many other bad leaders, did not want to be bothered with reality, facts, and circumstance. He just needed somebody to blame. Moreover, his obsession with uptime and cost cutting had won him praise among upper management, but subsequently created a dysfunctional internal culture. His staff had become conditioned to fear being blamed for every tiny problem. As such, they were unwilling to do anything to improve the environment, because it came with the risk of being blamed for downtime. They simply focused on doing the bare minimum to keep their job and keep things running.
Over the years, Anitian has observed thousands of horrible managers. Bad leadership can transform a proactive, dynamic and responsive organization into a lazy, ineffectual and insecure business in very short order. A common feature of bad leaders is their obsession with blame at the expense of solving problems.
Effective security depends on an IT team that does not wait for a problem. Rather, the IT team has the tools, training and practices to identify, define and respond to issues before they grow into an incident. System patching is a clear example of this principle. IT teams need to proactively patch systems to avoid breaches.
However, the flip side of this issue is “the business” that needs IT systems to run in order to fulfill the company mission (and make money.) If the business becomes intolerant to any downtime risk, and subsequently does not plan for it, it can lead to an intolerance of action on the part of IT administrators. This ultimately creates a poorly managed environment that is susceptible to exploitation.
If an organization is so fragile that it cannot handle the routine maintenance of systems and applications, then it is destined for failure. Lazy administrators running ramshackle servers, piled up with poorly engineered applications that are never maintained is the back-story of every breach.
IT management needs to make resilience and IT operations a priority. Moreover, management needs to treat every failure or mistake as an opportunity to grow, learn and improve. Furthermore, management needs to empower IT people to act and think proactively.
Far too many IT executives judge the success of their department exclusively on uptime. It is not enough that systems run. They need to run well. That means providing the time and tools for IT people to effectively manage the systems. If 100% uptime is required, then there should be extensive high availability. And there should be practices and systems to test patches before being rolled out to production hosts.
The irony of bad leadership is that it is a self-fulfilling prophecy of sorts. Bad leaders never see their demise coming. Slowly, over time, the resentment accumulates among the staff. The culture of the organization disintegrates and quality people resign. What is left are the people who wholly embrace the sick culture and revel in laziness. They too will always have an excuse for why something cannot be done.
This is why organizational culture is so important to security. While many security leaders obsess over the performance statistics of the latest “next generation” gadget, the culture of their organization is awful. A good CIO or CISO should make sure that all managers receive training as “people” managers. Moreover, the “blame game” must be removed from the culture. Management must focus their attention on solving problems, not blaming people. However, this applies to IT staff as well. They need to stop creating excuses for why they cannot fix something. There is a way to fix everything.
Bad leadership and the subsequent laziness of an IT team are truly a “Cultural Zero-Day.” There are persistent vulnerabilities that are easily overlooked, ignored and pushed aside in favor of uptime, certifications and performance throughput. However, these Zero-Day vulnerabilities are lurking and creating an environment that will breed the next weakness an attacker will exploit.
I have been that IT guy before. You are absolutely correct that bad leadership can be very demotivating. It made me so resentful and unwilling to do anything extra. I was fortunate that I figured this out before I became a bitter cynic. My new job, while less money, is much more satisfying. I get the time to innovate and improve, and my boss does not flip out over every problem.
Excellent insights as usual, Mr. Plato.