After crashes, D.C. officials say new vaccine site will be better

Washington, D.C., officials said Thursday a new website that will replace the city’s current system of scheduling COVID-19 vaccination appointments as soon as next week is being subjected to more rigorous user and capacity testing ahead of its launch, potentially providing relief to residents who were stymied last week by coding malfunctions, server outages and other issues.

For three days last week, tens of thousands of residents opened their computers just before 9 a.m. to participate in what’s become a new ritual in D.C. since the development of the vaccines: trying to sign up for one of about 4,300 appointments being released those days. The process has been compared to gamers hoping to nab an elusive PlayStation 5 console and fashion junkies scoring a rare pair of sneakers.

But the D.C. vaccine website quickly buckled under intense demand when the Feb. 25 allotment — reserved for residents of majority-Black and low-income ZIP codes that’ve seen the highest rates of coronavirus infections — opened up. Some users struggled to get past the CAPTCHA user verification, others were inadvertently redirected to a login page for city government workers and still more could not load the site at all. D.C. Council members tried to share troubleshooting tips over Twitter, such as clearing browser caches and trying multiple devices.

And despite assurances from Mayor Muriel Bowser’s office and Chief Technology Officer Lindsey Parker that Microsoft, the vendor that built the vaccine website, was addressing the issues, they persisted when more appointments were offered Friday and Saturday.

On Thursday, Parker acknowledged that last week’s appointment windows — the first since D.C. opened vaccine eligibility to all adults with a range of qualifying health conditions — crashed after interest in the site was many times higher than in previous weeks.

“We had a 27,000% increase in traffic,” she said. “We’ve seen 62 users peak in the past. Twenty thousand all trying to do something at the same time caught the Microsoft platform a little off-guard.”

‘We weren’t terribly surprised’

A batch of about 5,000 vaccine appointments released at 9 a.m. Thursday went more smoothly — though all were claimed within seven minutes — after Parker’s office made several modifications to the current site, including the creation of a waiting room and the removal of the CAPTCHA layer. But one expert said D.C. still has ways to go in correcting what has been the rough rollout of a crucial digital service.

“It’s an incredibly frustrating experience when the public wants vaccines and aren’t able to get it,” said Raphael Lee, a cofounder of the U.S. Digital Response, a nonprofit group of civic technologists who’ve been helping state and local governments develop tools during the pandemic.

Lee said he does not envy the position governments like D.C.’s have been placed in.

“State and local health agencies have been given a daunting task in the fastest vaccine rollout in history,” he told StateScoop. “When we saw the news about Microsoft system not working, we weren’t terribly surprised.”

Lee said that of the three major vendors to offer end-to-end software that manages vaccine eligibility and scheduling — a group that also includes Salesforce and PrepMod, an online clinic management platform developed by a Maryland nonprofit — none “are in a complete state of being.” (PrepMod has been blamed for a troubled vaccine rollout in Massachusetts, one of more than two dozen states using it.)

And the review that often accompanies the launch of a new digital government product has been uneven as vaccine portals have been rushed into service. That includes testing both a site’s user experience and its ability to withstand demand surges. While the tens of thousands who rushed to the D.C. site last week were a record at the time, Lee said it should’ve been able to weather the rush.

“No website of this scale should fall over when 37,000 users are trying to access it,” he said. “Those numbers are not that big.”

But Lee added that these quality checks are often “one of the first things to go under a tight deadline.” He also said that government agencies can only pin digital service failures on their vendors for so long.

“I don’t think you can keep kicking that can down the road forever,” he said.

In both emailed questions and questions asked directly to Parker, the District’s IT agency has not said if the current site was thoroughly vetted before its launch earlier this year. And when pressed during Thursday’s press conference by a reporter from Washingtonian magazine, Bowser’s reply was short.

“The system didn’t work the way it should have,” the mayor said. “I don’t really have anything to add.”

‘This website will get hammered’

Parker did say the new site is undergoing rigorous user and load testing before its planned launch next week. The new site will also include a pre-registration questionnaire, which will allow D.C. residents to be notified when they are eligible to make an actual appointment, depending on their employment status, neighborhood of residency and other factors. (The city’s health director, Dr. LaQuandra Nesbitt, said the chronological order in which people sign up will be the least important factor in determining their later eligibility.)

Lee, the U.S. Digital Response technologist, said load testing is about more than just front-end capacity. Many crashes, he said, are the result of too many concurrent connections creating bottlenecks on databases. And he predicted that D.C.’s new pre-registration site will be popular.

“They have to assume this website will get hammered from 100,000 people at the same time,” he said. “It can’t fall over under those circumstances. I would also advise them to do a round of UX testing, particularly with the groups they most want to get vaccinated. If it’s not usable to them, what are we even doing?”

On Thursday, Parker said her office set up a “sandbox” environment to test the usability of the new site. “We have a number of community members who’ve made themselves available for that,” she said. District officials have not given a specific date when the site will be live. John Falchicchio, Bowser’s chief of staff, said the city expects to launch it within the next week, but later Thursday, Parker told the D.C. Council her office was still deciding whether to keep Microsoft as the vendor or use a site developed by Accenture; city officials confirmed Friday they are sticking with Microsoft.

Parker also told reporters she also expects the pre-registration feature to “flatten the traffic” over time, as it’s expected to end the madcap races to book appointments. City officials also expect the pace of vaccinations to pick up with with increased allotments from the federal government including a new one-shot formula from Johnson & Johnson that joins the existing two-shot vaccines developed by Pfizer and Moderna. The new site will also be accompanied by a call center that can also take pre-registrations and a public awareness campaign, officials said.

Lee said the vaccination hiccups faced by D.C. and other cities and states are not simply about broken websites.

“This is not a story about technology,” he said. “This is a story about service delivery. The technology we use is a medium to serving people.”