This week, Scott Gregory, the state chief technology innovation officer and head of OET, gave Techwire an update of sorts on the progress since then. In a phone interview, Gregory talked about how he’s added some staffers and shifted his team into overdrive — how processes that used to take weeks or months are now being done in days, hours and even minutes. Gregory emphasized automation as a key to OET’s ongoing evolution into a digital rapid-response team of sorts. Following is that exchange, edited lightly for clarity and brevity.
Techwire: If there’s a theme or a thread running through what OET is working on, what is that?
Gregory: We’re doing a lot around automation in a way that lets us get products to market a lot faster, more securely and efficiently. The team’s been involved in a lot — a lot — between COVID-19 and fires, and one thing we’ve discovered is that we’re asked to do a lot of things quickly. One of the things we’ve started to do is explore automation on a very grand scale for a lot of the work that we do. A majority of the work we do is building and delivering websites, building and delivering Web applications, data products, data analytics and data science. But underlying all that requires this very robust infrastructure that helps us automate the pre-planning, the development and the push-out of some of these products, whereas things that might have taken 30 days, we’re now pulling that time frame really tight, down to less than an hour, when it comes to deploying the infrastructures to push out really important things.
One real good example of that is a repository that we’ve curated (with) automation really central to its core, code.ca.gov. It’s a repository for open-source-produced code and content for state agencies, departments and partners. We’re able to use that as part of our infrastructure to push out code and automate the development and curation of infrastructure to support websites and Web applications.
This effort to build out this automation really started about three years ago, to move from an on-premise infrastructure approach to more of a cloud-based approach, where we’re hosting and developing a lot of these mission-critical applications in the cloud. And here’s the thing that’s great about that: By doing that, we’re able to fail fast, so we have this ability to do faster deployments, which translates and equates to speed-to-market for the tools we’re being asked to build and develop. By virtue of having a cloud-based, cloud-focused strategy, it lets us fine-tune security, fine-tune releases, and allows us to look at questions and answers in resolving defects. In software development, you’re going to have issues — nothing is ever perfect. This (cloud) environment lets us follow that concept of fast failure and quick recovery before items go into production.
Another interesting aspect to what we’re doing: At (the California Department of Technology) CDT, it’s not just the developers who are out there building code and then deploying it. We’ve engendered this new process — maybe a new paradigm within CDT — it’s called DevSecOps: development, security and operations. And by having all of those pieces in line, we’re able to mitigate for operational components we weren’t aware of, or development components we weren’t aware of — most importantly, the security piece to it. We work very closely with the state CISO (acting Chief Information Security Officer Vitaliy Panych) and that office, as well as some of our security counterparts across the state, to do things like vulnerability scanning, to be able to manage software release with security in mind first and foremost.
I have, for a number of years, made security the primary driving force, that little voice on everybody’s shoulder: “You need to be considering security at the front end, always, always, always.” The result of that is we get better reliability. We prevent issues from going further than they should.
Techwire: What prompted this focus on DevSecOps?
Gregory: We felt that this area of our operations required, based on the demands that are put upon us, to have a fresh look at it, and that’s when we began to build out what we call our DevOps pipelines, or CICD — continuous improvement, continuous development. That’s a principle within DevOps. It changed an already very nimble team to a team that, to give you an example, is asked on a Friday evening to have Web infrastructure, a website, and some complexities around Web applications built, developed and deployed by Monday afternoon. And we do that regularly. And that’s set a precedent. … Now we need to sustain and foster this going forward, because that’s the new normal, right? We’ve seen it with COVID-19 and how fast events have changed and how much information and science have been at the core of the decisions that the administration has made, and they’re sound because they’re based on data. The infrastructure that supports those, the pipelines that support those — the information products, which might be in the form of a database or a website or a dashboard — those are all fed and nurtured by our automation behind the scenes.
Techwire: Is it accurate to say DevOps/DevSecOps is part of a new mission for OET?
Gregory: I found that more and more, whether it was day-to-day or more strategic implementations of technology across the state, we had to have a way to manage it more fluidly, securely and efficiently. We are able to do things in a matter of hours that used to take us days, weeks. And that helped us not only support some of our departmental focus and goals for technology, but it also just helped us answer the mail to our constituents, to our customers. They were kind of shocked: “Things got turned out fast, and I like that.” Why can’t we? Why not? It’s not just the private sector that can do that.
Techwire: You’ve raised the bar for yourselves on delivery. Is that sustainable?
Gregory: It’s kind of like the iron sharpening iron, right? The environment requires you to get sharper, to evolve and adapt, and that’s what it’s done. And inadvertently, to me, it’s created this new demand for DevOps engineering in the state. There’s an expectation now that if folks are asking for something and it’s of an urgent nature, they’re not having to wait a couple of months. They’re literally having to wait a couple of days, and it’s done, end to end.
Techwire: How has the idea of change management played into this transformation?
Gregory: I’ve always been a big believer in the fact that teams don’t fail, but leadership does. And when leadership isn’t directing and guiding and leading in a way that gets people excited, gets them fired up and drives them toward “What’s coming around the corner?” then you’ve got a problem. These people in OET are in technology for a reason: Because they love it. They love to get their hands dirty. They want to. They’re hungry. I was able to provide an opportunity to folks to be able to use the things, and do things, that they studied in school, that they’re just not doing on the job — things that, frankly, some of them do as side jobs. Some of them do this work in their garage lab, but they want to do it at work. That’s the same principle as the Innovation Lab: We took those same operating principles and applied them. And we’ve built this incredibly talented team that I would say is unique to California government. I have Amazon-certified (and Microsoft) Azure engineers on staff, and they’ve done that on their own accord. It’s not part of a training climate I approved — although I would have. That’s how motivated these folks are to getting the job done and doing the hard things. And that’s a very descriptive statement of the DevSecOps team. They thrive in an environment where they have to do the hard things: “OK, what’s next?” They know what they’re doing; that’s why they’re there. In a way, it’s been revolutionary to how we look at the deployment of technology going forward. And I believe it’s only going to mature as time presses on.
Techwire: How does the Office of Enterprise Technology set its agenda? Do you get requests, or assignments, or do you decide what needs to be done?
Gregory: It’s a mixed bag. In our office, we have to continually keep our ear to the rail, listening to what’s coming, seeing what’s around the curve. That, in a way, dictates broader high-level strategy that we then share with our leadership, to either get course correction or “You’re right on target.” We also have received, many times, direction from the (Gov. Gavin Newsom) administration through our leadership to accomplish certain tasks or build certain things.
And there’s another element: It continues to keep morale high and keeps people motivated to want to deliver and go above and beyond. I take this distributed-leadership model where … I will give the intent of where we need to go, based on what I’m hearing, and then I rely on my experts to go and build me something or build me a process, we work it together, and then come to a conclusion on that solution, and then that’s what gets delivered. I never come into a scenario where I’m prescriptive and saying, “You have to color it blue, and this has to be red, and this has to be green.” No. There’s lots of ways to skin a cat, and that’s where the creativity comes into the team. It’s said that you don’t manage creatives; you lead them. And that’s what these folks are: They’re programmers, they’re engineers, but they’re incredibly creative, and I don’t want to get in front of that train. I want that to flourish and grow. And that’s how we’ve been able to do some of the great things that OET and CDT, collectively, have been able to do: Just get out of the way.
