Examining the covidtests.gov architecture

A day early, the Biden Administration has quietly launched the new website that allows households to order four COVID-19 rapid antigen tests.

The reporting from last Friday, when details of the website were first made available, indicated the U.S. Digital Service (USDS) and the U.S. Postal Service (USPS) were involved, and that the site would use USPS’s existing web properties and facilities to process orders and deliver the tests.

Now that the site is live and accepting orders, let’s look at the outside indicators to see how it was constructed.

The landing page

The www. domain name is a CNAME record that ultimately resolves to Akamai. This isn’t surprising, as Akamai has performed as a CDN for U.S. government properties before.

The A record for the naked top-level covidtests.gov name are IP addresses in the Akamai network.

So the landing page and its CSS and JS dependencies are likely objects in Akamai’s NetStorage product fronted by its CDN product.

The call to action button “Order Free At-Home Tests” takes the user to a USPS web property, https://special.usps.com/testkits, which is the actual order form.

The order page

This domain name resolves to AWS CloudFront, its CDN offering. The authority section of the NS record for special.usps.com is ns-418.awsdns-52.com, which indicates special.usps.com is managed by AWS’s Route53 DNS hosting product.

The order page itself is an object in AWS S3 object store as an origin server, fronted by the CDN.

Submitting the order form is an HTTP POST to yet another new domain, special-api.usps.com:

This domain name appears to map to AWS API Gateway.

The form POSTs a JSON blob to the API endpoint (just the root path in this case), and returns confirmation data to the user. That is the entirety of the order transaction.

async function confirmOrder(
  recipient: KitRecipient,
  sender: KitSender
): Promise<Response> {
  store.dispatch(setIsActionPending(true));
  const backendHost = APIEndpoints().API_BASE;
  const body = {
    firstName: recipient.firstName,
    lastName: recipient.lastName,
    address1: recipient.address1,
    address2: recipient.address2,
    city: recipient.city,
    state: recipient.state,
    zip: recipient.zipCode,
    email: sender.email,
    uCode: recipient.uCode,
  };

  const res = await fetch(`${backendHost}/`, {
    method: "POST",
    mode: "cors",
    headers: {
      "Content-Type": "application/json",
      "X-Api-Key": "V2FrZSB1cCwgTmVvLi4u",
    },
    body: JSON.stringify(body),
  });

  store.dispatch(setIsActionPending(false));
  showError(res.status);
  return res;
}

This is the extent of what we can tell about the implementation and internals of the site from outside signals. I suspect what’s happening from here is that API Gateway is in front of a Lambda function (indeed, this is a common design pattern that AWS documents) that does minimal-to-no processing of the JSON blob and puts it in a database, likely DynamoDB given the overall managed services flavor of this implementation. At that point, backend processes can take over, likely kicking off notifications, enqueuing business tasks, and talking (via some peer/VPN) to the USPS enterprise network.

Building the frontend of government

What’s interesting about covidtests.gov is that, in spite of the reporting of the use of existing USPS web properties to aid in fulfillment, this appears to be a new custom implementation of the order form. It seems to bring together a set of AWS-managed services (again, speculating on the degree to which this is true all the way down into the stack) using largely statically-rendered content served from object stores and CDNs. There is a single point of transaction from the user’s perspective, the simple POST of a small amount of JSON data to the API endpoint.

This is a very common overall design in the modern web services stack; nothing out of the ordinary here. Indeed, the relative “boringness” of the architecture and seeking managed services from ISPs like AWS and Akamai, who are proven at the largest scales, was likely a desired aspect of this implementation.

I’m not aware if the USPS is typically a big cloud user. In fact, I would be surprised if they had this architecture laying around to be picked up by the Administration. It’s more likely that USDS and AWS worked together to use the USPS design language and backend integration points to achieve this relatively single-use, purpose-built site (Edit: A source tells me the USPS did in fact design and implement the cloud architecture of the site.). Contrast this approach with 2013 and the era of the HealthCare.gov launch when the state of the art in government was to administer servers in private data centers with numerous moving parts and failure modes without the resources and experience to handle intense public demand and traffic.

The rate at which the “frontend” of government (i.e., that with which the public primarily interacts) is moving to new digital services is increasing, and so are the public’s expectations for these services when they do appear. They expect them to be available, responsive, usable, on the devices they use, and for their interaction to fulfill their needs.

Simple, dependable architectures such as the one covidtests.gov seems to employ are proven at scale. This affords agencies the space to focus on improved user experience and service delivery, rather than consuming large resources keeping sites up and running. This takes operational experience and know-how, though; even with the use of managed services, composing a full, end-to-end digital service experience takes skill. It takes experience to identify the patterns of digital services that are best addressed by various system designs. But scalable web services are, in 2022, a commodity available to all. There is no excuse for getting this wrong. It’s heartening to see that the team seems to have gotten it right.

There is still no turnkey solution for UX, but success in government digital services lies in getting the fundamentals right, building public trust, and inspiring confidence from agency staff that they can meet extraordinary needs. Effective, mission-critical software at scale need not be complicated. Serving the public by fulfilling their online requests quickly, reliably, without undue hassle, regardless of how many people come at once, respects their time and humanity, and improves their overall experience with their government.