Skip to content

Sigstore Google Season of Docs 2023 Case Study

Jump to bottom
ltagliaferri edited this page Nov 21, 2023 · 1 revision

This case study will be published in our wiki and will be publicly accessible. We will include a link in the Google Form to this public case study.

  • Project Description

Creating the proposal

How did you come up with your Google Season of Docs proposal? What process did your organization use to decide on an idea? How did you solicit and incorporate feedback?

Sigstore is a distributed open source project with an active Slack, biweekly community meetings, and additional communication on GitHub and at meetups and other events. We announced the plan to make a proposal for Google Season of Docs on Slack and in a community meeting, and we collaborated on what we wanted to achieve. Lisa Tagliaferri took point and worked closely with other Sigstore maintainers to ensure that the voices of each Sigstore subproject was incorporated.

As a community, we knew there were a number of issues with the Sigstore docs, as we fielded reports from Slack, Twitter (now X), and Issues on GitHub. There were a number of frictions being reported, and people could benefit from more targeted help. Our plan to update, reorganize, and expand the Sigstore docs was a first pass to make sure our house was in order and accessible to the developer audience.

We developed a form to solicit statements of interest and shared this with the community and through social channels. We received 135 responses and also had an influx of people into the Slack community, which was a great benefit to Sigstore!

Budget

Include a short section on your budget. How did you estimate the work? Were there any unexpected expenses? Did you end up spending less than the grant award? Did you allocate funds properly or were some items you budgeted for more/less/unnecessary? Did you have other funds outside of Google Season of Docs that you were able to use?

Our budget was based on 20 hours of work a week during the course of Google Season of Docs (26 weeks). We benchmarked an hourly rate against the U.S. Bureau of Labor Statistics, with $28.85 an hour amounting to just under the 25th percentile of U.S.-based technical writers.

As we went with an hourly rate and worked closely with Patrick Smyth on meeting hourly expectations, we did not run into any overages or underages for this budget. We did not have access to other funds outside of Google Season of Docs for this project.

Participants

Who worked on this project (use usernames if requested by participants)? How did you find and hire your technical writer? How did you find other volunteers or paid participants? What roles did they have? Did anyone drop out? What did you learn about recruiting, communication, and project management?

Patrick Smyth (smythp) was the technical writer we hired for this project. Lisa Tagliaferri (ltagliaferri), as the lead maintainer of Sigstore docs, served as the manager for this project and provided most of the code and writing reviews. Lisa and Patrick also worked together to cut over the site on our host.

We received 135 responses to our request for statements of interest. We were impressed with Patrick's work as a web developer and technical writer, as we knew there were a number of web development issues hurting the user experience of the Sigstore docs site. Patrick also is committed to accessibility which we believe to be aligned with Sigstore's vision and open source work in general.

We were lucky that no one needed to drop out of the project. Patrick was expecting a new addition to his family, and we were able to plan around this effectively due to close communication (and many congratulations to Patrick and his family!).

While we are very happy to have been able to work so closely with Patrick, and believe that Sigstore greatly benefited from his expertise and efforts, we did have to turn away many other candidates who were excited about Sigstore and the opportunity. We learned that communicating with this group early was important to ensure that everyone had a good experience.

Throughout this work, we also learned that distributed open source projects require frequent communication. Patrick and Lisa met several times with technical writers at Google in order to ensure that we did not duplicate efforts and to remain aligned.

Timeline

Give a short overview of the timeline of your project (indicate estimated end date or intermediate milestones if project is ongoing).Did the original timeline need adjustment?

The technical writer, Patrick Smyth, onboarded in late May. After an initial period of following GitHub, Slack, and email communications, and after meeting with the core maintainers of the Sigstore documentation repository, Patrick created his first issue, a friction log, on June 7th. Throughout June, Patrick made a set of smaller issues and pull requests and spoke with maintainers and other stakeholders on how to best contribute to the project during the Season of Docs.

In early July, after conversations on GitHub and with maintainers, it was observed that a major source of friction with the Sigstore documentation were a set of longstanding bugs and maintenance issues associated with the nuxt theme and associated build process (see the Results section below for more details). It was decided in mid-July that Patrick would take on a full re-platform of the docs site as a major project.

Through July and most of August, Patrick worked on re-platforming the full documentation site to Hugo. This period was the most active for Patrick's involvement with the project, as this was a relatively large undertaking. On August 18th, content contributions to the site were temporarily frozen to prepare for the re-platform. On August 21st, the new Hugo site went live.

From August 21st through early September, Patrick resolved issues related to the re-platform, including issues related to changed semantics of the content hierarchy and redirects for moved content. Patrick also contributed to an ongoing major effort to make the docs site more comprehensible to new users and performed a sweep of all documentation and external links for incorrect and 404 links. He also communicated with the maintainer of the new Hugo theme (Doks) to resolve issues with the site's new search functionality.

Through September and October, Patrick also worked on a second major project, an accessibility evaluation of the site. Patrick worked with a fully blind screen reader user, Krishna Bodawala, to identify accessibility issues, and coached Krishna on creating GitHub issues. He also used automated tools to identify other accessibility issues. In this period, Patrick also created PRs to resolve these issues as they came up.

In November, Patrick worked on creating Hugo template logic to allow for external links to project-specific documentation to appear on the site's sidebar. Patrick also worked on a initial glossary for the site. In late November, Patrick began a planned pause for the birth of a child.

While most major planned milestones have been reached, in December a PR for the initial glossary is expected and some smaller issues related to project installation will be closed. In addition, Patrick will perform a final pass for any unnecessary files, metadata, and content not needed in the new Hugo site.

Results

What was created, updated, or otherwise changed? Include links to published documentation if available. Were there any deliverables in the proposal that did not get created? List those as well. Did this project result in any new or updated processes or procedures in your organization?

The major projects undertaken in this Season of Docs include:

  • A replatform of the full documentation site to Hugo., resolving longstanding user issues with the site and positioning the documentation site for the addition of new features.
  • A full accessibility review of the documentation site, with emphasis on accessibility for screen reader users.
  • The addition of a glossary to the documentation site (in progress).
  • Audit full site for broken links and update redirects for renamed and moved pages, maintaining inbound links
  • Support an initiative to reorganize content to be more comprehensible to new users.
  • Update documentation README to provide more granular information on building locally, performing fundamental tasks, and updating content.
  • Provide clear links to project-specific documentation hosted in other Sigstore repositories.
  • Create friction logs to identify and close issues related to new user stories.
  • Onboard new contributors by performing code reviews.
  • Close miscellaneous longstanding issues.

Re-platform

The largest initiative undertaken during this period was a full site re-platform. This re-platform had been previously discussed by core maintainers of the docs site, but had not been specifically proposed as a project within Season of Docs. However, the technical writer also had background in web development and static site generators such as Hugo, and in June and July, following community discussion, it was decided that the replatform would be an opportunity to resolve major ongoing issues and set the site up for proposed additions.

Issues with the prior version of the site included:

  • A caching issue that resulted in older versions of the site being rendered to visitors. This was particularly vexing for those who maintained projects across Sigstore and those who visited the docs site frequently.
  • A relatively slow local site build and refresh process.
  • The necessity for the use of a workaround involving enabling a legacy OpenSSL provider on many local setups.
  • Issues with site search stuttering and freezing the site, as reported by users.
  • Absence of a dark mode
  • Cluttered site navigation bar requiring scrolling to see all headings.

Hugo was considered a strong choice for the re-platform, as Go is the primary language used in Sigstore tooling. In addition, Go is considered one of the fastest SSGs, and the project is widely used. The Doks theme was chosen for its functionality and for some continuity in layout between the old and new docs sites, and also because of its emphasis on speed and accessibility.

The re-platform required significant original code to be written, specifically related to bypassing the default Doks home page, which assumes the use of a full landing page, blog, etc., and the integration of JavaScript-based site search with the theme. In addition, logic for rendering categories and hierarchies was rewritten to produce more intuitive output. Patrick communicated with the creator of the Doks theme on some of these issues :

The re-platform resolved all of the issues above, including the below longstanding and persistent issues reported by Sigstore users:

After the re-platform, Patrick resolved an issue for semantics in the site hierarchy that would have resulted in dead inbound links by rewriting Doks theme template logic.

Patrick also updated the docs site README with instructions on performing common tasks, such as reorganizing sidebars and creating new content pages, as well as documenting the new build process.

Hugo and the Doks theme provide functionality useful for implementing some specific user requests, such as documentation versioning, localization, and a glossary. Some of this work has been started, but is not yet completed.

Part of the re-platform included a faster and more responsive site search, which also resolved issues with the site stuttering or freezing while using search.

Finally, the visual design of the new docs site has been well received by users in the community, as reported on Slack and in community meetings.

Accessibility Review

The Doks theme has a focus on accessibility, passing most tests on Lighthouse and axe. However, a full accessibility audit had not been performed on the documentation site. The technical writer worked with a fully blind screen reader user to identify a set of issues related to keyboard navigation and screen reader functionality on the site. These issues have now been resolved. A sample of these issues:

After this audit, the new docs site is more accessible to keyboard users and blind and low vision users, especially those users using a screen reader, requiring high contrast, or using color inversion, as determined through both automated and manual testing.

Resource Audit

A programmatic audit was performed to resolve all broken internal links on the site, including broken resources such as images. A crawler navigated the site and complied a list of missing resources. These broken links and resources were then manually resolved. This work was primarily undertaken in the below PR:

Further, additional redirect rules were created to maintain inbound links to resources that had been recently moved or renamed.

Support Content Reorganization and Update

Before Patrick's onboarding, a content initiative was undertaken by core project maintainers to make the documentation more comprehensible for new users. While this work was not undertaken with Season of Docs resources, Patrick did support this initiative through code reviews, coordination during the re-platform, and other specific assistance:

Update README

In order to support future maintainers, and in order to document new build processes following the re-platform, Patrick updated the README file for the documentation site, providing specific information on contributing to the site and specific information on the the technical stack used to create the site.

Surfacing External Docs

Sigstore includes a number of projects, each with their own repositories. Some of these projects elect to maintain their own project-specific documentation files. These project-specific docs are sometimes difficult to discover for new users. To surface these docs, Patrick created Hugo template logic to allow maintainers to include links to external docs on the site sidebar. This functionality is not provided natively by the theme, which only allows internal links on the sidebar by default.

Friction Log

Early in the Season of Docs, Patrick created a friction log for signing a blob. These logs were intended to guide Patrick as he created resources for new users. After the pivot to the replatform as a major project in June/July, the creation of further friction logs was planned but de-prioritized. However, a small set of issues were identified and closed based on this initial log.

Other Contributions

During his time as a technical writer with Sigstore, Patrick participated in most twice-monthly community meetings, performed code reviews for new contributors, and closed a number of miscellaneous issues.

Metrics

What metrics did you choose to measure the success of the project? Were you able to collect those metrics? Did the metrics correlate well or poorly with the behaviors or outcomes you wanted for the project? Did your metrics change since your proposal? Did you add or remove any metrics? How often do you intend to collect metrics going forward?

The proposed metrics for this Season of Docs were:

  • Decrease in user help-related messages on the Sigstore Slack due to outdated docs
  • Decrease the amount of drifting-away contributors (based on Linux Foundation Insights)
  • Monthly new unique visitor counts to documentation increase by at least 10%
  • Monthly returning visitor counts to documentation increase by at least 10%
  • At least 2 new contributors filing issues or making a pull request to Sigstore documentation

While some of these goals have been achieved for this period (~6 new contributors, decreased questions related to outdated docs in Slack), given the shift in focus early in the season, we do not find these metrics to be fully Representative of the efficacy of the season.

Given the focus of the work performed by the technical writer in this period, we feel that two other metrics capture some of the impact of the project:

  1. Closed issues related to longstanding or severe bugs.
  2. Site score based on automated tools such as Lighthouse and axe.

The closure of a set of severe issues related to the site, including the persistent caching bug and the site sometimes freezing on search, can be considered major quality-of-life improvements for users and contributors.

Before the re-platform and audit, the site registered 12 accessibility issues on the homepage considered serious by axe. The homepage now has 0 issues considered serious.

While it is not appropriate for an ongoing metric, the initial local build time for the site has gone from 20.69 seconds to 969 milliseconds as a result of the migration to Hugo. This change will primarily benefit contributors and maintainers of the docs site.

Analysis

What went well? What was unexpected? What hurdles or setbacks did you face? Do you consider your project successful? Why or why not? (If it's too early to tell, explain when you expect to be able to judge the success of your project.)

While the project took a substantially different course from the Season of Docs proposal, we consider the project to have been a success. Qualitatively, we consider the project to have achieved the following aims:

  1. Developed a more maintainable documentation site.
  2. Realized quality-of-life improvements for users, contributors, and maintainers.
  3. Aligned the docs site with other technologies used in the Sigstore project, most significantly the Go language.
  4. Simplified contributions through improvements to the site's build process.
  5. Provided faster responses to new contributors, resulting in acquisition of ~6 new contributors to docs alone and one new frequent contributor in this period.

We have seen a decrease in help-related messages on Sigstore Slack and in Issues, many of these having been related to the poor user experience of nuxt (the previous iteration of the Sigstore docs site). Drifting-away contributors has gone down relative to previous time periods, at an increase of only 0.88%. The contributor strength grew the most between the time period Jul 31, 2023 and Aug 16, 2023 during the last 6 months, providing a strong indicator that our documentation efforts have been working to pull in new contributors. Additionally, pageviews and unique visitors have been increasing: we have received 60,800 pageviews and 14,000 unique visitors over the past 30 days. Finally, the docs repo has pulled in 6 new contributors, beating our goal of 2 new contributors. More analytics on Sigstore are available via the LFX Platform.

The project proposal suggested a focus on creating resources to guide and onboard new users. However, technical writer Patrick Smyth began the season just as other core contributors took on a major content reorganization with this aim. After learning that Patrick had a background in web development and SSGs, it made sense to focus on a re-platform as a major deliverable for the period following a month of learning about the project.

We feel that staying flexible as to the nature of the season's deliverables resulted in substantial improvements to the documentation site. Functionally, Sigstore has a new documentation site thanks to resources provided through the Season of Docs, and we feel that the re-platform has had a high impact for users and contributors. Patrick has also expressed an interest in continuing to work with the Sigstore docs site as part of his broader commitment to open source.

summary

In 2-4 paragraphs, summarize your project experience. Highlight what you learned, and what you would choose to do differently in the future. What advice would you give to other projects trying to solve a similar problem with documentation?

We feel that our shift in focus led to more substantial contributions over the course of the period from our technical writer. Open source projects move quickly, and resources and priorities are difficult to predict. We advise being flexible as to ways the technical writer can contribute to the project.

One possible insight is to choose a technical writer with a background or skills not represented among core maintainers. Patrick's background in web development and accessibility provided to be as relevant as his background in technical writing. With that said, traditional technical writing skills and attention to detail were also key, and should be considered critical when choosing a technical writer. We did not closely supervise Patrick's progress during the period or request reports outside of our usual GitHub issue or Slack channels. We feel that this worked well for us as all work was done in the open and was visible to users and contributors alike.

We feel that the onboarding period is critical for a successful project. Patrick has reported that his creation of a friction log early in the process helped him get a handle on the project as a whole in ways that passive reading and following communications did not. We recommend the active creation of resources such as friction logs and user stories as a way to onboard new technical writers. We also suggest highlighting some small issues at the start of the project period, as touching markup and code right away seems helpful for quickly developing familiarity with project structure, CI processes, and policies.

Patrick mentioned that he might have used more Season of Docs materials and channels during the project, for networking and community building. Perhaps Season of Docs could generate an onboarding email with Google-specific resources to send to new technical writers as they spin up. However, we do not feel this was a major issue.

We believe that Sigstore is a high-impact and necessary project and that docs are a big, if sometimes underappreciated, element of our community. The docs are often how users first encounter Sigstore, and the docs also hold together different teams of tool-specific contributors. We feel that this period has set up Sigstore to move forward with a more maintainable, functional, and featureful docs site. We would like to thank the Season of Docs team and Google for supporting Sigstore and the open source community by sponsoring this project.

Clone this wiki locally