The city of Lodi, which opted to rebuild from backups rather than pay attackers in bitcoin after a breach last year, saw the DR process play out firsthand that spring. The breach, which released malware, impacted phone and financial systems as well as payment data — though public information and most of the muni’s workstations were unaffected. Recovery took about a month to get to roughly 95 percent and restore more than 100 virtual machines. Lodi was already underway with an IT procurement when the attack occurred; it later opted to purchase Rubrik’s Cloud Data Management platform with a three-year support contract. That solution has been in place for nearly a year and has proven its worth, Dale Taylor, a network technician at the city, told Techwire. Among the takeaways:
- Practice your disaster recovery and backup procedures ahead of time. That’s because there can be significant differences between the way backup and DR are supposed to work and the way they actually work, particularly when an actual incident occurs and a response is urgently needed. In Lodi’s case, its previous backup solution was at its end-of-life — and could require an hour or two to search back to a specific “point in time,” Taylor said, “because then, it had to piece together all the increments that happened after its initial base image.”
“That’s a pretty normal practice to have your backup, how you’re going to restore and/or how you’re going to back up and make sure it actually works the way you think it does,” Taylor said. “And how are you going to restore a particular file? Do you have any encryptions on those files to protect private information or anything along those lines? You have to keep that all in mind as you’re going.” Beyond its ease of installation, Rubrik’s speed enables file restoration within minutes and its performance enables one server to be stood up as another is being rebuilt. - Where cybersecurity’s concerned, don’t just watch your front end — make a wish list of what your government needs on the back end for its next backup solution. After all, losing your backups to encryption could mean very difficult choices. Lodi sought a product that was “a lot easier to work with,” Taylor said. The Rubrik platform, he said, is more autonomous and reflects service levels across all servers — compared to their old process, in which “you’d have to go touch each one and set up its own backup schedule.” It’s forward-looking but integrated well with older systems and solutions as well as the city’s virtual environment.
- Look for resiliency and segmentation. Lodi had to physically disconnect its backup appliance from its network to isolate it once it learned of the ransomware attack, or risk losing its backups. “With Rubrik, because it’s sort of separate pieces there … you don’t have to do that,” Taylor said.
Dan Raynes, state and local senior account executive at Rubrik, described its technology as “immutable” to ransomware, adding: “You can’t get into our file system through ransomware. That was a big thing for them.”
