Philly transit agency uses cloud-based site to handle massive demand during pope’s visit

With Pope Francis coming to town last September, Philadelphia transportation officials found themselves with more people interested in buying subway tickets than they could handle.

The public transit agency serving the city, the Southeastern Pennsylvania Transportation Authority, had to deal with an influx of more than 1 million visitors when the pontiff swung through Philadelphia. Though the department drastically reduced service, Mike Zaleski, director of emerging and specialty technology, said the agency still had to find a way to efficiently sell thousands of tickets for specialized subway service to coincide with the pope’s visit.

Though Zaleski and his team experienced their fair share of setbacks, including a full website crash, they were ultimately able to build a cloud-based online portal to get papal tourists the tickets they needed.

“The media had a field day building up the hype, saying, ‘You’ve got to get in and get tickets or you won’t see the pope,’” Zaleski said Tuesday at the Amazon Web Services Public Sector Summit. “So the big focus at the time was, ‘We’ve got to sell these tickets.’”

Zaleski remembered first hearing from upper management at the agency, also known as SEPTA, that they needed to build a dedicated website for papal ticket sales in “six weeks,” so that it was ready by late July.

Making matters more complicated, Zaleski’s superiors told him that the website needed to sell tickets “on demand,” instead of using a lottery or “take a number system,” and they didn’t want to outsource ticket sales to a third party like eBay.

“We wanted to keep it ours, keep it branded and designed just so,” Zaleski said. “We spent lots of time on the [user interface], the look and feel.”

Accordingly, SEPTA’s IT team built their own site with the open source web platform Drupal hosted in the Amazon Web Services cloud. But Zaleski noted they expected that tickets would “sell over the course of a week, not a few minutes.”

“There was a major disconnect between what we anticipated and what actually happened,” Zaleski said.

Indeed, Zaleski said roughly 54,000 people visited the new site within the first minute in launched, and it soon crashed under the massive load.

“Once the first site folded over, there was backlash from upper management,” Zaleski said. “We thought, ‘We need to find a way to make this work.’”

That sent Zaleski back to the drawing board, and he turned to Amazon’s engineers for guidance. Mike Kuentz, a senior solutions architect with AWS, said they quickly realized that part of the problem was that the site was designed to send an email confirmation as soon someone bought a ticket, which tended to slow things down.

Kuentz said they identified other “bottlenecks” in the code and expanded the capacity of the back-end servers, but it still wasn’t enough.

“We got to 900 transactions per second in our tests, but we needed to get to 1,700 transactions per second,” Kuentz said. “It was a matter of looking at the problem the wrong way.”

Zaleski said they truly started believing that the “on demand” approach was untenable when they asked major online ticket vendors for help.

“They said, ‘Based on requirements, there’s no way we can do it,’” Zaleski said. “It was ‘Thanks, but no thanks.’”

[Read more: State Department sharing IoT energy-tracking techniques with cities]

But SEPTA was eventually able to turn to a local online ticket seller, Ticketleap, which agreed to partner with them to build a site with one important caveat.

“We thought selling in real time would be disaster, and we wanted to do a lottery system,” said Tim Raybould, Ticketleap’s CEO. “We knew enough to be appropriately scared about the problem. They weren’t, but they got scared when it was in the public eye. So it seemed pretty obvious what the strategy should be.”

Zaleski said his bosses still weren’t thrilled about moving to a lottery approach, but after their struggles with the first site and the strong reaction from other ticket sites, he noted that it became “easy sell and people really got it.”

“It was a more egalitarian way to determine who gets the most tickets,” Zaleski said. “It reduced number of scalpers as well, since you couldn’t have a team of people buying all at once.”

Ticketleap and SEPTA were able to get the new site online by the start of August and held a one-day lottery for tickets, Zaleski said. He noted that most people that entered got tickets, and SEPTA was able to let the winners know the good news within a week.

In all, the process contributed to a  successful papal stopover, and Zaleski said it taught his team an important lesson in the process. 

“This really shows the value of load testing,” Zaleski said.

Contact the reporter at alex.koma@statescoop.com, and follow him on Twitter @AlexKomaSNG.