Software maintenance mistake at center of major FAA computer meltdown: Official

(NEW YORK) — The ground stop and Federal Aviation Administration systems failures Wednesday morning that impacted thousands of flights across the U.S. appear to have been the result of a mistake that occurred during routine scheduled systems maintenance, according to a senior official briefed on the internal review.

An engineer “replaced one file with another,” the official said, not realizing the mistake was being made. As the systems began showing problems and ultimately failed, FAA staff feverishly tried to figure out what had gone wrong. The engineer who made the error did not realize what had happened.

“It was an honest mistake that cost the country millions,” the official said.

Earlier Wednesday, the FAA said normal operations were “resuming gradually” after ordering a nationwide pause on all domestic departures until 9 a.m. on Wednesday morning following a computer failure that has delayed and canceled flights around the country.

“The ground stop has been lifted,” officials said at about 8:50 a.m. ET. “We continue to look into the cause of the initial problem[.]”

Departures were resuming at about 8:15 a.m. ET at two of the nation’s busiest hubs — Newark, New Jersey, and Atlanta — FAA officials said on Twitter, adding, “We expect departures to resume at other airports at 9 a.m. ET.”

The affected Notice To all Air Missions, or NOTAM, system is responsible for sending out flight hazards and real time restrictions to pilots, administration officials said earlier.

“The FAA is still working to fully restore the Notice to Air Missions system following an outage,” said the FAA announcing the temporary grounding of all planes nationwide. “The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information.”

Had the FAA’s new NOTAM system been in place, redundancies would likely have stopped the cascading failures. With the antiquated system in place, there was nothing to stop the outages, the official told ABC News.

“At this time, there is no evidence of a cyberattack. The FAA is working diligently to further pinpoint the causes of this issue and take all needed steps to prevent this kind of disruption from happening again,” the FAA said in a statement Wednesday night.

There were still more than 7,300 delays and 1,100 cancellations midday, according to tracking website Flight Aware.

Failures likely due to ‘glitch’

Transportation Secretary Pete Buttigieg said a full investigation is necessary to prevent any future mishaps.

“When there’s an issue in the FAA that needs to be looked at, we’re gonna own it, same way we asked the airlines to own their companies and operations,” Buttigieg said during an appearance on CNN Wednesday.

Congressional hearings are expected as is a possible speed-up of system replacement.

On what caused the system meltdown, Buttigieg said that overnight there “was an issue with irregularities in the messages that were going out” — though more needs to be learned on what led to the widespread failure.

“Now we have to understand how this could have happened in the first place. Why the usual redundancies that would stop it from being that disrupted, did not stop it from being disrupted this time, and what the original source of the errors or the corrupted files would have been,” he said.

A senior official briefed on the FAA computer problems told ABC News the software issue developed late last night and led to a “cascading” series of IT failures culminating in this morning’s disruption. As has been reported, the disruption is confined to the commercial side of aviation.

As of now, the assessment is the failures are the result of a “glitch” and not something intentional. All possibilities are being looked at to ensure that the FAA systems were not breached.

The FAA first reported the system failure on Tuesday, according to an internal memo from the Cybersecurity and Infrastructure Security Agency obtained by ABC News.

Notably, the FAA system that failed is overdue for replacement.

The official compared the current outage to the crisis that crippled Southwest Airlines during the holidays: antiquated software overdue for replacement inside a critical IT network. If one thing goes down, the system can become paralyzed.

White House press secretary Karine Jean-Pierre confirmed in a tweet that President Joe Biden had been briefed and said “there is no evidence of a cyberattack at this point.”

“I just spoke with Buttigieg,” President Joe Biden said addressing the media on the South Lawn of the White House. “They don’t know what the cause is. But I was on the phone with him about 10 minutes. I told him report directly to me when they find out. Aircrafts can still land safely, just not take off right now. We don’t know what the cause of it is expected to be able to — in a couple hours we’ll have a good sense of what caused it. And we’ll respond at that time.”

Buttigieg also said there’s currently no indication of a cyberattack, but “we also are not going to rule that out until we have a clearer and better understanding of what’s taking place.”

Republican Sen. Ted Cruz criticized the FAA’s management of the system and called for an explanation of issues.

“The flying public deserves safety in the sky,” Cruz said in a statement. “The FAA’s inability to keep an important safety system up and running is completely unacceptable and just the latest example of dysfunction within the Department of Transportation. The administration needs to explain to Congress what happened, and Congress should enact reforms in this year’s FAA reauthorization legislation. This incident also highlights why the public needs a competent, proven leader with substantive aviation experience leading the FAA.”

Airlines hit with delays, cancellations

In the wake of the temporary nationwide pause, several airlines confirmed they were experiencing delays and canceling or diverting flights due to the NOTAM system outage.

“We are closely monitoring the situation, which impacts all airlines, and working with the FAA to minimize disruption to our operation and customers,” American Airlines said in a statement. “We encourage customers to check aa.com for the latest flight information.”

The airline had canceled nearly 400 flights and 850 flights have been delayed as of late Wednesday morning due to the FAA system issue, it said.

United Airlines said it has “temporarily delayed all domestic flights and will issue an update when we learn more from the FAA.”

Southwest Airlines also confirmed it was suffering through delays due to the technical issues at the FAA.

“Please check your flight status in the Southwest app or website to watch for any flight status changes,” Southwest Airlines said in a statement released early Wednesday morning. “If your flight status changes substantially we will message the day of travel contact listed on your reservation by their preferred contact method.”

Delta said it “will continue to work with the FAA on strategic delay programs at certain airports throughout the day,” with 130 flights canceled and another 30 diverted.

“Potential for additional delays and cancels continue,” Delta said in a statement.

Several airlines are waiving fees to rebook flights due to the FAA issues, including American, Delta and United.

ABC News’ Sarah Kolinovsky contributed to this report.

Copyright © 2023, ABC Audio. All rights reserved.