Pantheon Autopilot


Autopilot  is a website management tool that provides a continuous, fully automated visual QA review for CMS updates. The tool is used by a wide variety of customers from small business to large enterprise customers. The tool is also adaptable enough to work with a variety of complex plugins and themes.  It’s one of several features provided by Pantheon that enables website teams (developers, marketers, designers) to create and maintain their sites in a collaborative way while reducing repetitive manual work.

Autopilot was an acquisition made by Pantheon in 2019. As the design lead, I led our effort to integrate the tool into the Pantheon user interface and added a set of key features that made the product compelling to our customers. The project was a tremendous success, with consistent growth in usage (roughly 10% month over month per account), and Autopilot is now a key part of the Pantheon product portfolio.
  • Timeline: October 2019 - Now
  • The Team: One designer (me), One product manager, 6 engineers

Problem Statement

Background: Pantheon acquired StagingPilot (renamed to Autopilot) in early 2019. When I joined the design team, my first major project was to integrate this legacy product into Pantheon’s dashboard, and to identify the key features that would make it into a market-leading solution. 

The Problem: Website owners need to update their sites to introduce new functionality and to stay secure. However, each update introduces the risk of breaking the site's visual appearance or business-critical functionality. Especially when it comes to security updates, many businesses choose to save development resources and defer for months or even years.

This leads many to get to a point where it would be easier to "start from scratch" rather than take on all of the incurred technical debt of updating properly. This cycle can stifle innovation and can leave sites outdated.

Getting to the problem statement
We began by conducting research—a heuristic evaluation of the legacy product, user interviews to better understand what potential customers are looking for, and an in-depth competitor analysis. 

Screenshot of some of the analysis we did:

Through our interviews, I learned that website maintenance work requires expertise and continual effort, an especially daunting task when managing a large portfolio of sites. There are new updates and security disclosures every day, so sites are almost always out of date.

We also learned a bit more about how users traditionally handle website updates. They usually have to do one or more of the following.

  • Pay ongoing maintenance fees to the agency who built the site
  • Continuously pay a service provider to update sites for them (Valet, WPSiteCare, DrupalSquad, myDropWizard)
  • Use significant internal development resources to do maintenance themselves
Together with the Product Owner, we ran a product vision workshop and defined our ideal future state for this product.

The solution
When using Autopilot:

  • Everything on the customer's site is always up-to-date. All available updates for CMS core and 3rd party modules are applied safely to the site.
  • The site visitor's experience remains the same. The layout isn't broken, images aren't missing, elements of the page haven't moved around unexpectedly. At a glance, everything looks good. The visitor can still achieve their goals (purchase products, submit contact forms, log in to the site, etc).

Defining Scope

With this happy path in mind, I was able to break down the solution into 7 key user flows. I captured the flows in a user story map (built with Miro), so that other stakeholders could easily digest the information and provide input.

This was purposely kept high level in an effort to help define the scope of each of our planned releases. As a team, we reviewed my proposals and decided to depriortize 4 of the proposed features, primarily due to technical debt. We agreed on 3 initial features for the MVP:

  1. Site Setup: This was a configuration process for Site Owners who activate Autopilot for their site for the first time.
  2. Activity Feed: This displays the most recent activity related to Autopilot for a given site. The target audience for Autopilot are larger, risk-averse corporations managing over 100 sites. It’s important that theres a paper trail for them
  3. Visual Regression Testing: This was a tool that enabled users to review the visual impact of their site’s updates. The Visual Regression Tool (VRT) takes a screenshot of their site’s pages before and after updates were applied. Users can then compare the pixel differences and decide if the updates should be deployed.
Once I had a clear sense of product scope, I developed more detailed user flow diagrams and shared them with the engineers as soon as I could. Because Pantheon’s product development teams utilize the three-legged stool model, it’s critical that design, engineering and product management are always in lockstep.

Screenshots below:

Design Explorations

Multiple rounds of design iteration, testing with users and presenting to stakeholders allowed us to take advantage of quick feedback loops and build confidence in the solution we were implementing. This product was highly anticipated within Pantheon, and we were facing pressure from stakeholders to deliver. Design at Pantheon runs parallel sprints to the development team, so we were consistently a month ahead of our engineers. This enabled us to break down designs into eng tasks right away. Every one of the features below was built into a Figma prototype and shared with real customers for validation.

1. Autopilot Site Setup

This is a three step process which allows Autopilot users to customize the feature for their site's needs. They can set the desired scope of the tool, exclude specific themes or plugins from the automated updates, and define pages for visual review.


Latest version in production:

2. Visual Regression Test Tool

Automated regression testing for key pages ensures that UX is consistent while securing your site and implementing new features.


Latest version in production:

3. Activity Feed

The Autopilot Activity Feed displays historical reporting of the updates and testing that were performed on a given website.


Latest version in production:

4. Status View

The Status page gives real-time information to users who are viewing Autopilot for a specific site.


Latest version in production:

Early Access User Testing

In March 2021, Autopilot was ready to be shared with a small number of customers through our Early Access Program. As Design Lead, I was fortunate enough to partner with our sales engineers and product manager to hand select the first 10 customers. I set up hour-long calls with each of these customers, and alongside my product manager, we used this time to conduct moderated usability testing. We enabled Autopilot on each of their accounts, and observed them as they navigated the interface and shared their feedback.

All of our notes from these sessions (including the script, and our analysis) are captured in a sheet like the one below:

This process took a few weeks. At the end of it, we analyzed our notes and I created a research report that was presented to the entire executive team and sales organization.

Everyone was eager to ship the product to a larger audience, so it was important that we took a moment to share our learnings from the Early Access group. This gave stakeholders confidence that we are prioritizing urgent bugs and UX issues prior to releasing it to the general public.

Below are some screenshots of a few of the slides I presented:

Major Iterations

From April - July 2021, we gradually released Autopilot to more customers (About 5 per week). Every week during this period, our scrum team  reviewed and discussed  the customer feedback, so that we could start implementing bug fixes and improvements. 

I’ve highlighted a few of those UX Improvements below:

1. Improved Progress Indicator

When a user sets up Autopilot for the first time, it kicks off a series of functions on the backend for checking for updates, applying updates, taking page screenshots and deployment. Due to some technical constraints, this process takes about 5 minutes from start to finish. For the first few months, the number one complaint from customers was that this took too long, and that they had no indication of how much time was left.

2. Excluded Web Elements

The biggest product differentiator for Autopilot is the Visual Regression Tool. At a high level, this feature takes a screenshot of your website’s pages both before and after updates are applied to your site. We received feedback that customers were getting false or accidental failed visual regression tests, because the tool would take a screenshot very quickly, before the page loaded. Additionally, some websites with dynamic content (such as a carousel, slider or a video player) would automatically cause the visual regression test to fail, even though it was an expected change. 

To solve this, we introduced a feature that lets users exclude CSS selectors on their site from Autopilot. After 2 weeks of design and rapid user testing, we began development on this feature.

3. Workspace Overview

Our target audience for Autopilot is companies or agencies that are maintaining a large portfolio of sites. The initial release of Autopilot was hyper-focused on setting up the feature for one site at a time, via the Pantheon Site dashboard,  but as time went on and more customers were adopting it, it was clear that customers wanted a central location to view Autopilot related activity.

After digging into the specific feedback, I proposed that we introduce a tab in the global navigation to make the feature more discoverable. Clicking on the tab would provide customers with a summary of the ROI Autopilot is providing them, as well as a list of sites that are actively using the feature. 

This was released shortly after June and received positive feedback, especially from our agency customers. It correlated to a significant spike in feature adoption.

4. Better Error Messages

By far the most frequent issue reported into our customer support team was the unexpected error alerts in the Autopilot UI. At the time of launch, we only provided generic placeholder error messages, since we didn't have specific error data available that could be passed to the front end. Tackling the number of errors, as well as the specificity, became top priority for us in August.

I took a few days to re-write all of our error messages, and designed a few new error alerts so that customers could better self-serve their issue. Once we had these implemented, we saw a 20% reduction in support tickets within 2 weeks. In addition to the UI, these error messages are documented publicly.


Autopilot launched into General Availability in early August. While the initial feature adoption rates was low due to the amount of undiscovered bugs, we saw a dramatic improvement after a few weeks.

After 3 months here is a summary:Autopilot is being used on 17,320 websites, across 300 different accounts. Of all the eligible customers, 10% are using Autopilot. On average, it saves each account roughly 4 hours per week of time. That extra time back empowers website teams to focus on more strategic work.
As part of the launch, myself and a few colleagues hosted an external webinar. I demo'd the product from end-to-end and answered questions in the chat from our customers.

What I Learned

As always, our plans didn’t go as expected. Below are a few of the major things I learned from the project. 

Lack of design predictability Our product design team at this time was a small group of 3. All of us were working on at least 3 projects simultaneously. At the start of the Autopilot project, design work wasn’t being tracked in a formal way (no JIRA, n Asana..) and worse, we weren’t estimating our design tasks. This led to the inability to factor design work as part of the official project timelines. It sometimes felt as if I was holding back the engineers, or blocking them from being able to implement frontend work. Unfortunately, it also led to shortening some of our UX research activities so that I could speed up the design cycle. Luckily, we corrected this about 3 months into the project once we developed a project management playbook for design. 

Overlapping projects
Aside from Autopilot, another major project I was working on at the time (2020) was a complete redesign of Pantheon’s dashboard UI and navigation. This project started to pick up around the same time that Autopilot was being developed. Midway through development for the MVP of Autopilot, the executive team recommended we pause all frontend work and restart in the new dashboard interface. If our legacy dashboard would get sunsetted soon, it would make more sense to develop all new features in the new UI. While this was great for our customers (aside from a more modern looking UI, the new dashboard performed much better) it did extend our release timelines by several month. This decision required me to redo several of the designs for the updated interface, using our new design system components.

Looking Forward

Autopilot is projected to be a major contributor to upsell revenue in the following years, which means our work has just started. In December 2021, I partnered with our researcher and product manager and launched an in-app customer satisfaction survey for existing Autopilot users.

We’re hoping to use this data to better understand why customers aren’t using Autopilot for 100% of their website portfolio. Our hypothesis is that the lack of awareness and frequency of repeat bugs is causing some hesitation. Our goal is to present our findings to the company and adjust our roadmap based on our results.