Soapbox: DevOps with the Ad Hoc Infrastructure team

Soapbox is an open source software platform for building and deploying web applications using cloud providers like AWS. It represents our take on DevOps-done-right.

It’s informed by our existing work with government agencies, like the Centers for Medicare & Medicaid Services and the Department of Veterans Affairs, where we’ve successfully managed deployment processes and infrastructure by “baking in” practices such as blue-green deployments, monitoring, logging, alerting, and feature flags, among others.

It also provides a training ground for reinforcing and expanding Ad Hoc’s institutional DevOps capabilities. It’s owned by the Ad Hoc Infrastructure Team - an internal team that gives every employee an opportunity to contribute to internal projects and products during month-long rotations.

Existing hosting and deployment platforms, such as Heroku, Elastic Beanstalk, OpenShift, and Cloud Foundry, address various niches and use-cases, but generally they all promise to make provisioning, packaging, and deployment of web apps simple. While they are useful in some instances, they are, in our experience especially working with government agencies, usually some combination of too ambitious, too arcane, or too closed-source.

Our approach begins with a contract. Applications hosted on Soapbox must provide:

  1. A GitHub repo,

  2. A Dockerfile that describes how to package and execute the app, and

  3. A health check endpoint.

In return, the app is provided a port to bind to.

Over the course of a month, the Infrastructure Team built a proof-of-concept that can:

  • Deploy a web-based application that adheres to the contract, load-balanced across multiple availability zones,
  • Build an isolated VPC per application with a common-sensible approach to network security, and
  • Provide low-risk, zero-downtime, rolling blue-green deployments out-of-the-box.

The prototype includes an API server and CLI built using Go and a web frontend implemented with Rails. In addition, the cloud service provider infrastructure is managed by Terraform.</p>

An overview of Soapbox's deployment model

An overview of Soapbox’s deployment model

Soapbox is focused on:

  1. Minimizing layers of abstraction — It relies on git, docker, and AWS. If you’re familiar with these, it’s easy to understand what Soapbox is doing under the hood.

  2. Embracing convention — Using best-practices for underlying infrastructure out-of-the-box means you don’t have to spend time to figure out The Right Way.

  3. Security — We aspire to meet U.S. government standards for security practices. While we’re not going through a formal certification process any time soon, we want to be forward-looking.

  4. Building trust in your infrastructure — Consistency, reliability; erring on the side of caution for uptime’s sake means you can trust your deployment strategy and sleep soundly at night.

  5. Making configuration overridable — There’s always an “escape hatch”; in extenuating circumstances, you are ultimately in control of your infrastructure.

Why it matters

Helping the government deliver services to people more efficiently and effectively is at the core of Ad Hoc’s mission.

Soapbox can help by simplifying the process of standing up cloud infrastructure and deploying software with agility in a secure and predictable way. Our government partners often don’t have the bandwidth or resources to transition to and maintain cloud infrastructure for their software. That’s where Soapbox can be helpful: it is an encapsulation of the best-practices we’ve learned deploying digital government services to the cloud.

The ultimate scope of the project is evolving. The next step involves a review of work done so far, cleaning up the prototype, and testing its efficacy with Ad Hoc’s internal apps. Later iterations are expected to: include the ability to deploy scheduled/cron tasks, make it easy for developers to add feature flags to their apps, and bolster monitoring, logging, alerting and security measures.

If you would like to know more, check out Soapbox on GitHub and view the in-progress roadmap.