Last week’s global technology outage was traced to a bug in the quality control system of US cybersecurity firm CrowdStrike.
The impact of the outage was widespread, affecting around 8.5 million Windows devices and disrupting banks, emergency call centers and airlines. Fortune 500 companies, excluding Microsoft, face an estimated $5.4 billion in losses due to the outage, according to insurer Parametrix. Meanwhile, hackers are using the outage as an opportunity to target CrowdStrike customers.
“The fact that proper analysis wasn’t done created this huge cascading problem that companies are still dealing with today,” said Scott White, associate professor and director of the Cybersecurity Program and Cyber Academy at George Washington University in Washington, DC.
What caused the IT outage?
CrowdStrike released a routine software update early Friday morning to monitor for possible new threats, but the company’s postmortem found that the update was “problematic,” causing memory issues and resulting in Windows “blue screens of death.” Mac and Linux hosts were not affected.
According to Dominic Sellitto, clinical assistant professor of management science and systems at the University at Buffalo’s School of Management, the software “was trying to do something that Windows couldn’t handle, which resulted in a system crash.”
CrowdStrike said it has a “content validator” that inspects software updates before they are released, but that a bug caused the program to miss problematic content in the update.
“We deeply apologize for the inconvenience our customers experienced on Friday,” CrowdStrike Chief Security Officer Sean Henry said in a LinkedIn post on Monday, adding that “thousands of team members are working 24/7 to fully restore our customers’ systems.”
The company told USA Today that it sent Uber Eats gift cards to teammates and partners who worked with customers. TechCrunch reported that some recipients were unable to access the gifts, and CrowdStrike confirmed that Uber had flagged the gift cards as fraudulent due to “high redemption.”
What does the future hold for CrowdStrike?
CrowdStrike said it plans to improve its testing, give customers more control over when they install updates, and phase future software updates in line with “Rapid Response” content.
Gregory Falco, an assistant professor of engineering at Cornell University in New York, called the measures “good software deployment and engineering practice.”Some cybersecurity experts question why certain safeguards weren’t put in place before the technology failure.
“It’s easy to be an armchair expert, but we’re seeing best practices here that probably should have been implemented sooner,” Sellitto said, adding that he appreciated how quickly CrowdStrike responded to the outage.
Nicholas Behar, an adjunct cybersecurity professor at the University of San Diego, said he was surprised to learn the outage was linked to CrowdStrike, “one of the best, if not the best, cybersecurity companies in the country.”
“They were talking about doing more testing to make sure this doesn’t happen again, but they should have been doing the testing already in the first place,” Behar said.
The House Homeland Security Committee sent a letter to CrowdStrike CEO George Kurtz requesting him to testify about the outage.
“We cannot ignore the significance of what some are claiming to be the largest IT outage in history,” the letter said, adding that “Americans will undoubtedly feel the lasting real-world impacts of this incident” and “you deserve to know more about how this incident happened and what mitigating measures CrowdStrike is taking.”
A painful warning:What’s next for CrowdStrike and Microsoft after the update outage?
CrowdStrike said it plans to release a full analysis of what caused Friday’s outage once its investigation is complete. Experts who spoke to USA Today said they hope the upcoming report will shed more light on the decision-making process that led to the bug affecting millions of devices.
“I hope the manufacturer did their due diligence and will wait to hear their explanation,” George Washington University’s White said. “Whether you found the defect or not is irrelevant. My question is why the defect was put on the market, and that seems to be missing here.”
Reuters contributed to this report.
