Thousands of vulnerabilities in Seattle's IT network attributed to siloed approach to cybersecurity

Staff told leadership the vulnerabilities had been patched, but in fact they had not.

Early morning at the Seattle Municipal Tower on June 12, 2019. (Photo by Paul Christian Gordon for Crosscut)

Last May, Seattle’s head of information security flagged a problem within the city’s technology department: Because of a process breakdown, employees were indicating that they had fixed vulnerabilities in the department’s computer network when, in fact, they had not been fixed.

“It has been discovered that there are currently over 21,000 known critical and high vulnerabilities on systems throughout Seattle IT,” Andrew Whitaker, then the department’s chief information security officer, wrote in a May 22 email to technology leadership. “Tickets have been closed out, claiming to have vulnerabilities remediated, but upon follow-up review they were, with a few exceptions, not remediated.”

The result was that the servers, desktops and applications within the newly consolidated Department of Information Technology — which now handles the vast majority of the city of Seattle's technology functions, from utilities to the fire department — contained open miniportals that could be accessed by would-be hackers.

When left unremediated, vulnerabilities provide possible paths for hackers to plant spyware, ransomware, viruses and other malicious software that can be immensely harmful to an organization, especially one that provides critical services. Cities are often particularly open to an attack and the effect can be devastating, as recent ransom attacks in Baltimore and Atlanta have shown.

Saad Bashir, Seattle’s new head of the Department of Information Technology, said in an interview that he believes the vulnerabilities are manageable. He said Seattle is at risk, as are all organizations, but, in general, not abnormally so.  

However, Bashir acknowledged the process breakdown was indicative of a broader problem he has been attempting to address within the organization since taking his position earlier this year. “What I observed very early was that there was a siloed approach in how cybersecurity was being practiced in the world of IT,” he said.

Because of a disconnect between teams, Bashir said, some part of the security process would get completed, but would not be properly handed off to the next team. “If you're not clear, then you may not know whether that particular vulnerability management work has been completed the way it’s supposed to be completed,” Bashir said.

In an effort to improve the processes within the department, Bashir began a major reorganization of the relatively new department — including his firing of 14 directors and managers — just two days before Whitaker’s message. The reorganization was not motivated solely by security weaknesses, he said, but was intended to create a smoother structure that would better catch possible entry points. When asked if the city was safer from an attack since he took over, Bashir said, “Absolutely.”

Every organization contains some number of vulnerabilities. The trick is to continually identify and address them as they arise — an e-windshield wiper of sorts, where the vulnerabilities are the raindrops.

Experts say hackers are increasingly less likely to gain access through a vulnerability than they are through a phishing expedition. In such cases, a deceiving email message persuades employees to provide passwords or a malware-infected USB drive is left in a parking lot in hopes that someone finds it and plugs it in to their computer.

But addressing vulnerabilities in the city’s systems continues to be an important function of its IT department.

“If I were a serious bad guy I’d be looking at the most vulnerable place,” said Dr. Barbara Endicott-Popovsky, executive director of the Center for Information Assurance & Cybersecurity at the University of Washington. “I’d be looking at cities and I’d be looking at universities, because they’re open and they can’t afford the latest and greatest. It’s kind of like, ‘Open sesame.’ ”

Mike Hamilton, founder of CI Security and Seattle’s chief information security officer from 2006 to 2013, said there are a number of reasons cities struggle to stay ahead of cyberattacks.

For one, the number of qualified security experts is down across the country, he said. And of those who are on the market, cities can’t match the pay of large companies like Amazon or Microsoft.

“The ones that are good are in short supply, which means that local governments cannot compete for those resources,” he said.

Additionally, cities are responsible for the security of all their departments, each of which may require vastly different things. “Because government is a federation of agencies, that makes it a little difficult to have policies in place that apply to [for example] the regulated industry of human resources without raising the ire of unions,” he said.

Hamilton also said the biennial budgeting of local government makes keeping up challenging. “Technology moves a whole lot freaking faster,” he said.

All of this, Hamilton said, is in the context of extremely high stakes. Compared with for-profit companies, “the potential impact [of an attack on government] is so much greater and government can’t afford it,” said Hamilton. “We know something needs to be fixed, and we don’t fix it until something blows up.”

Bashir said the new processes he’s put into place has made him “confident that we no longer have any glaring process gaps.” He couldn’t say exactly how many vulnerabilities are still open on city systems, but that it was less than 21,000. The ideal number, Bashir said, is zero, but that’s also extremely unlikely, which makes it hard to identify what a “good” number is.

“I worry about all of them,” said Andrew Cushman, the city’s new chief security officer. “Whether that number is 21,000 or whether that number is 10 depends on the attacker and how skilled that attacker is and how motivated that attacker is. So I don't worry more because that number is 21,000, then I do if that number is 10.”

Going forward, Bashir said he wants “to create a high level of security awareness mindset across the organization.” The city could have zero vulnerabilities and it wouldn’t matter if one employee plugs in the wrong USB to a work computer.

Hamilton said there are several easy things cities can do that, while not offering total protection, would make it so they are no longer “the slowest gnu in the herd getting picked off.” For one, mandate zero personal use of city equipment, something Singapore implemented in 2017.

Phishing attacks remain the easiest entry point for hackers and so that’s where the bulk of the city’s attention should focus, Hamilton said. Because no matter how many protections are put into place, “There is not now, nor will there ever be, a firewall for stupid.”

Please support independent local news for all.

We rely on donations from readers like you to sustain Crosscut's in-depth reporting on issues critical to the PNW.

Donate

About the Authors & Contributors

David Kroman

David Kroman

David Kroman is formerly a reporter at Crosscut, where he covered city politics.