- Train and equip 110,000 county employees to work remotely while maintaining government’s continuity of operations.
- Stand up an online portal and virtual call center so the county’s Human Resources Department could help employees with teleworking, payroll, benefits, coding timesheets, handling worker’s comp issues, psychological or work-related counseling and any other HR issues that can arise during a crisis.
- Do it all as quickly as possible, with no time for RFPs or other normal procurement tools.
“There was really no chance to really do any kind of analysis, market share, best product, best of breed, blah blah,” Wesolik said. “It was like, ‘Slap this baby up and let’s go.’”
Masood said: “Essentially, from start to end, we set up an employee hotline and call center in less than nine business days. The implementation was timely: Within the first six days of operation, the HR hotline for remote-working employees received about 600 calls. And this allowed telework for almost the entire county in less than a week. That, to me, was one of the biggest accomplishments.”
Wesolik said, too, that the county got lucky — “If you can say that in a pandemic” — in a couple of ways:
- L.A. County had already held several continuity-of-operations (COOP) drills, so many employees had practiced remote working and were familiar with the procedures.
- Almost a year before the pandemic struck, Wesolik’s ISD had begun shifting some county departments from their legacy call centers to a cloud-based operation, with soft phones and other technology that allowed for more flexibility.
- Vendors stepped up.
In less than three days from start to finish, Masood said, Amazon’s AWS Connect platform was up and running.
A spokesperson for Amazon Web Services acknowledged the sprawling nature of L.A. County’s need.
“While collaborating with L.A. County, AWS had to make adjustments as priorities shifted, but the elasticity of our cloud allowed L.A. County to find the right capacity for this effort,” the spokesperson told Techwire in an email. “In such scenarios, customers will utilize existing data center solutions like VDI (virtual desktop infrastructure) and VPN (virtual private network), but those solutions are nearly impossible to scale to meet demand, especially in a very short timeframe.”
So far, Wesolik and his team of hundreds of technologists have set up nine other county department call centers, in addition to Masood’s HR Department. Masood said the average call to the employee HR hotline is now answered within 20 seconds and lasts more than 11 minutes. He credited county employees’ flexibility.
“Actually, we stood up a strike team to staff this call center,” he said. “We took staff who otherwise were conducting other HR responsibilities, providing services like recruiting or training, and since a lot of those activities are now being deferred or postponed, that staff was redirected toward providing call center support services.”
The training, which could be a monumental task for so large an organization, “was fantastic,” Masood said, indicating Wesolik’s team did six hours of training; then the two groups forged standard operating procedures and training quick-reference guides in less than 12 hours, for a total of around 18 hours of staff time.
The inevitable question of security was also not as big a challenge as expected.
“There are a couple of things we have in place for this,” Wesolik said. “If you’re on a county laptop, we have a host-checking function that allows us to make sure you’re current with your anti-virus, your anti-malware and a Cisco umbrella, so you’re not clicking on any bad links with your county-provided equipment. If you’re not on county-provided equipment, you’re going through our AppStream portal, which lets you get to certain functions as well as your email, which is also a secure portal. And we’ve just completed, countywide — another perfect timing — county cybersecurity awareness training. We’ve just got 99 percent of our staff through that, so that came in handy.
“And the last thing we’re doing is we’re using ZScaler remote access,” Wesolik said. “One of the challenges we’re having right now is a lot of folks aren’t able to use a county-provided piece of equipment, so when they use their own, they only get to certain things that are protected from a security perspective. So they can get to their email, but some of those smaller department-specific applications that they have to use — like the dogcatchers have a dogcatcher app, and the Beaches and Harbors people have an app — and all these have been harder to access. …
“We worked with ZScaler, a fairly small company, and Cisco … and we’re going to use Duo (Cisco multifactor authentication) for 90 days, free. So next week, we’re going to be able to allow county employees who are using their own personal devices to log in, to download ZScaler, and then we do a host check. And the host check is going to see, ‘Do you have anti-virus? Do you have malware? Do you have umbrella?’ — and if they don’t, it’ll automatically load it on their local machine, reboot the machine, it’ll come back up and they’ll have ZScaler, and then they can use their local machine as if it were a county machine.
"All of the (protection) — the McAfee, the Cisco Amp, the Cisco Duo — they’ve all signed up to give us a 90- to 120-day free use of licenses for as many employees as want to use it on their personal devices.”
Wesolik said his team “had been working on a cloud-first approach mentality to our IT services. … We’re coming from a very legacy operation — data centers, on-prem equipment, and on-prem services — so for the last year, we’ve been working to do a lot of transition into a cloud service offering. We’ve negotiated contracts with four cloud services — Azure, Amazon, Google and IBM Red Hat — so we had agreements in place for services that we could consume when we wanted to, and same thing with cloud-based infrastructure and security offerings. … The fact that we had put a lot of things in place really allowed us to cut through a lot of red tape and to work with extreme agility on this thing. I’m just so grateful that my team had been working on a lot of this stuff ahead of time, so we could react the way we did.
“We built portals, we made hosted desktops, we made access to applications that they didn’t have exterior before. We set up SSL VPN situations, we worked on call-forwarding a lot of business lines to other lines. We created WebEx sessions so that the Board of Supervisors could do their public hearings remotely for the first time ever in their lives — and on it went.”
Time was the enemy, Masood and Wesolik agreed.
“Murtaza pretty well cracked the whip on it,” Wesolik said. “He said, ‘We need this up yesterday,’ so we set our own deadlines, which was ‘Work around the clock until it’s up.’”
The AWS spokesperson cited the company’s cloud capabilities as a key to the quick turnaround.
“We did not have any hardware or software deployed yet,” the spokesperson said, “but we moved rapidly to support their needs and scale without hardware or software in place because of the nature of AWS cloud. The COVID-19 crisis accelerated L.A. County’s requirement to support up to 80,000 employees in under a week, and AWS was able to rise to the challenge, supporting nearly 30 county agencies through AWS AppStream.”
Wesolik characterized the project in non-technological terms: “There’s a gene in government public workers that doesn’t come out until you see these emergencies happen. It’s kind of like a dog that sits around the house and then, all of a sudden, when you go out hunting, you see a different animal. We went hunting.”
