Commentary: A Draft Manifesto for State IT Modernization

This post and the next describe a set of ideas that frame a modern software engineering program. I would like to see these concepts become a mandatory part of every software effort undertaken. I believe that building software using current best practices, and keeping those practices current, can be far more impactful than life cycle checklists in improving outcomes.

I’ll apologize in advance if these posts sound preachy. It is hard to suggest principles and guidelines with the emphasis required to help them stick and avoid it sounding like a liturgy. (The use of "we" throughout is intended to convey that the manifesto is written in the voice of the state.)

These practices I’ll suggest are captured in a great many articles and manifestos, including:

— Agile Methods: The Manifesto for Agile Software Development and 12 Principles Behind the Agile Manifesto

— Modern Software Engineering: The Twelve-Factor App and Continuous Integration

— Cloud Computing: Cloud Native Infrastructure

— Digital Government: Code for America and the UK Government Digital Service

Nonetheless, I have added my spin here and there. So, I get the blame for missteps, while the credit goes to the original thinkers.

As you read through the concepts, please recognize how technology continues to change and consider how pace of change accelerates. As the state looks to incrementally build modern business applications designed for change, it must also look for vendor partners who can help it keep pace. Technology can change in between increments, and the days of picking a programming language and a stack and a hardware vendor, building an application, and sticking with them for 10-plus years are certainly over.

The ideas and concepts in this State IT Manifesto are divided into three categories: Guiding Principles, Technology Guidelines, and IT Culture. Today's column will address Guiding Principles. Next week's column will address Technology Guidelines and IT Culture. 

GUIDING PRINCIPLES

Listed below are some guiding principles that we expect will endure.

Agility

Agility is built into agile methods in a variety of ways, but making change control a natural part of the process is important. From the other side, it is important not to embed a contentious, time-consuming and litigious change-management protocol. With the agile method, users, state staff and vendor staff all place user stories into a backlog. The end users, product owners, prioritize the stories every two to three months, after every product increment. It is expected that stories that were once low-priority may become higher priority and that new stories may be included and land in the next product increment. This is fine. Just accept that priorities have changed and move ahead.

Code First

"Code first" means we don't want to document every step of the process, producing artifacts that never keep up with the pace of change. We want to spend as little time as possible in building artifacts that are not code. What matters is working code, accepted by a real product owner and assembled into a product increment that is deployed in production. We will strive to put code first, using agile methods, modern tools and transparency.

Transparency is vital to enable oversight, executive management, vendor management, project and product management, end-user acceptance, customer management, and development velocity. The state must implement tools that automate the collection of data from the agile process and expose the data far and wide. Every developer will be able to see the progress of every other developer. Project managers will see progress automatically and objectively posted. Executive managers, California Department of Technology (CDT) oversight staff, and the Legislature have direct access to these automatic reports. Status is objective, not the result of a storytelling verbal process that results in green boxes in a presentation.

Transparency creates trust. The state should plan on trusting its vendor partners and ending the relationship when trust ends. State agencies work with trusted departments that do not hide failure from the broader organization; and via transparency work together to identify issues and fix them fast.

The state can execute on this principle by implementing a continuous integration (CI) modern development environment (MDE), where all code is stored for each product. We will implement an approach where “everything is code” — every piece of programming, every parameter file, every software build profile, everything. This code will be versioned and controlled. Whenever possible, code will be required to include a test of the code, and the MDE will automatically run the tests with each change and track the results. Tools will be built in to ensure that the code follows coding guidelines and to measure what percentage of the code is covered by the tests. Progress is the automatic measure of tested code.

Complete transparency will lead to an environment of open and honest communication and to a blameless culture where everyone pulls together to get as much done as possible. I know this sounds too good to be true; but it is a best practice commonly used in every Silicon Valley development shop. 

Communication

State IT will enable communication by deploying collaborative tools that interface with the MDE and that allow end users and developers to connect around user stories. Note that the agile methods do not rely on written requirements; they capture high-level user stories and depend on the developers and the end users to verbally work out the details. Acceptance of code as complete is entirely in the hands of the end users, so communication is critical for developers to compose acceptable code.

Quality

State IT will use the MDE to ensure quality. Using the tools described above, we will set standards that require 95 percent of all program code to pass automated developer unit tests and 70 percent of all code to pass automated user acceptance tests before release into production. Every bug found will be categorized by its severity, and each product will have release standards set by the business, not by IT, that describes the quality of the code required before release.

To get to this high level of quality, the state will provide a modern development environment for every project. Using container technology in the cloud, the MDE will provide a fresh build of the latest code to each developer every day. Developers will have their own complete build where they can work and test without concern that they may bring down a shared environment that affects the whole team. Providing individual test systems is inexpensive in the cloud and is a huge productivity boon. 

Use cost-effective software

There is a growing catalog of capable free software available for use in application development programs. The state must encourage vendor partners to consider free software in every case but to choose open source software only when it is cost-effective. The state should show that it has no bias toward open source software for its own sake, but should have a bias toward providing the most cost-effective solution.

"Buy versus Build" has transformed into "Buy versus Borrow versus Build." It is often the case that the state may look into the open source libraries provided by GitHub, GitLab, SourceForge, Apache and others, and find a large percentage of the code required already developed. Companies like Facebook, Google and Amazon do not develop large applications for a fraction of the cost in a fraction of the time that the state does because their programmers type faster. They are fast because they borrow significantly larger percentages of every application from open source libraries.

With all of this said, buying is often the best strategy. The state will buy when they can find modern products (cloud-native, 12-Factor-compliant, etc.) that support the business functionality required and that can be extended using a modern development environment to ensure that the custom modifications meet the same high-quality standards as other custom code. The most important aspect here is that bought software provides code that distinctly offers business functionality the department needs but that is not freely available.

Free and Open Source Software (FOSS) 

Applications and services developed with government funds should be owned by the people. The state should strive to deliver any software developed into the government domain as FOSS for use by other states and municipalities and by the federal government.

We recognize the power of community development. We can see a day when a state application published for use by other states might be enhanced by another state to the benefit of the citizens of California.

Work with users

The teams will make sure that they are working toward real needs for real users and the organization, regardless of whether teams are working on front-end user interfaces or putting together clusters of back-end infrastructure. This is to ensure that the work being done accomplishes real outcomes for the people the department serves, while reducing time and money waste.

Whenever possible, the state will connect developers with the actual users of a business process, not with surrogate subject matter experts hired into IT as a proxy for the actual business user. When the business user is the public, the state will engage with public representatives of actual users to solicit continual input. When we can, we will share the results and progress during development to get feedback from these public representatives.

Deliver iteratively and often

We need to get feedback from users often in order to fail fast and fail small. The more positive spin on the idea of “fail fast” is to succeed fast and to gain user acceptance along the way. Incremental acceptance is a huge benefit, an effect of agile methods. If progress is measured by incremental acceptance rather than outcomes, final acceptance will almost always be acceptable. So agile methods and incremental acceptance are how government oversight executes its responsibilities in a modern agile product program. If the scope is right and the actual user is accepting results every two weeks, then the outcome will be right. If development velocity is within norms, then the cost outcome will be right. 

Use modern and modular technology

The principles outlined in the Twelve-Factor Application manifesto and collected in the concepts associated with cloud native computing provide the basis for building modular applications that will evolve as technology changes and evolves, and for deploying those applications on the most cost-effective extensible infrastructure. The state will strive to build or buy modern applications always and to deploy them on modern technology.

Modular technology is developed as fine-grained services, each with an API (application programming interface). Modules can easily be replaced with new modules. This requires both modern software engineering and new software licenses that do not bundle modules in a manner that locks the state into a solution. 

Next week: Technology Guidelines and Staff Culture, the second and third parts of this Manifesto.