From 4a99ce414cf6c30ce11ff53444b62fb350cfdc45 Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Fri, 4 Nov 2022 19:42:47 +0000 Subject: [PATCH 1/7] WIP --- .../storing-source-code/index.html.md.erb | 118 ++++++++++++++---- 1 file changed, 97 insertions(+), 21 deletions(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index 5b66b98f..4ca73c58 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -6,33 +6,109 @@ old_paths: # <%= current_page.data.title %> -All our source code is open by default, and stored on well-known, -public code hosting services. At the DfE, we use GitHub. -We follow the principles set out in the service manual for managing the -code that we write: -- [use version control](https://www.gov.uk/service-manual/technology/maintaining-version-control-in-coding) -- [make source code open](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable) +## GOV.UK Service Manual -You should keep secrets separate from source code, and keep them private. +We follow principles set out within the [GOV.UK Service Manual](https://www.gov.uk/service-manual) +for managing code we write. -## GitHub +### Sections of relevance: -New repositories for products and services live in the -[Department for Education Digital organisation](https://github.com/DfE-Digital) -on GitHub. New repositories must be created within the [Department for Education Digital](https://github.com/DFE-Digital) organisation, whether they contain service production code or prototypes. Work created outside of the DfE Digital organisation should be transferred into the DfE Digital organisation at the earliest opportunity. [Guide to transferring a repository](https://help.github.com/en/articles/transferring-a-repository). +- [Service Manual, Technology: Use version control](https://www.gov.uk/service-manual/technology/maintaining-version-control-in-coding) +- [Service Manual, Technology: Make source code open](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable) +- [Service Manual, Technology: Securing your information](https://www.gov.uk/service-manual/technology/securing-your-information) -You can use your personal GitHub account (but you should [add your DfE -email address to your account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/), -and [use it for notifications](https://help.github.com/articles/managing-notification-emails-for-organizations/)). -Ask your delivery manager to request you being added to the Github organisation. +### Summary / Highlights: -Another Github organisation account heavilly used by the DfE but not the default for DfE-Digital is ['SkillsFundingAgency'](https://github.com/SkillsFundingAgency/). +- Changes to source code _must_ be tracked +- Code we produce _should_ be made available via an internet source code repository +- Published code _should_ be under an Open Source initiative compatible licence +- Due-care _must_ be given to security considerations, including: + - Suitable protection of confidential information and secrets + - Departmental/governmental rules related to the use of cloud/3rd-party tooling + - Proper process and accountability/approvals for making code changes -Repositories should be [clearly named](/standards/naming-things/), -and have an [appropriate licence](/standards/licencing-software-or-code) -and enough documentation that someone new can get started with the -project. +Additional detail and information is available via the links above. -Private repositories are not a good way to protect secrets, and should only be used where access to the code might reveal draft policy decisions. Secrets should be managed at the platform level. + +## Types of source code + +- Infrastructure as code +- Project + +## Git / GitHub + +At the Department for Education, we use git and the MIT Licence. + +While some projects remain on Azure DevOps + +and should be migrated across + +Specifically, we use GitHub for new and migrated work though projects remain on Azure DevOps. + +### GitHub Organisations + +Department for Education source code repositories on GitHub should be stored under an appropriate +organisation, thus giving appropriate oversight and protections to these repositories. + +Specifically: +- The [Department for Education Digital](https://github.com/DfE-Digital) + GitHub organisation is used for new and existing source code repositories + - This is applicable to production and prototype code +- Work created outside the DfE Digital organisation should be transferred into + the DfE Digital organisation at the earliest opportunity. + - [GitHub: Guide to transferring a repository](https://help.github.com/en/articles/transferring-a-repository) +- The [Skills Funding Agency](https://github.com/SkillsFundingAgency/) + GitHub organisation is also used by the DfE + - Not the default for DfE-Digital + + +There is a small cost for accounts to be added to a GitHub organisation. +Ask your delivery manager to request your account being added to these GitHub organisations. + +### Work vs Personal GitHub accounts + +You may use your personal GitHub account, but you should: + +- [GitHub: Add your DfE email address to your account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/) +- [GitHub: Use your secondary (DfE) email address for notifications](https://help.github.com/articles/managing-notification-emails-for-organizations/) + + +### Repository Requirements + +Repositories should be: +- [clearly named](/standards/naming-things/), +- have an [appropriate licence](/standards/licencing-software-or-code) +- have enough documentation that someone new can get started with the project + + +## Data Protection Considerations - Git Repositories + +### Personal Data + +Storage of a git repository must be treated with due care and consideration. +This applies whether it is within a central hosted environment or stored +elsewhere such on a developer's computer. + +- Changes to source code, commits, are annotated with authorship details. + Typically, this is a name and an email address. +- Where the commit is cryptographically-signed, the GPG key used will also have + personal information associated with it such as an email address. + +Additionally, note that git is _explicitly_ a _decentralised_ source versioning and control system. + +- It is, therefore, not possible to delete/change information within one copy of the + repository (e.g., GitHub) and force all other copies to be updated also +- It is, therefore, extremely important to prevent non-public content from ever + being added to the git repository in the first place + + +### Secrets + +You _must_ keep secrets separate from source code, and keep them private. + +Private repositories are a poor way to protect secrets, and may only be used +where access to the code might reveal draft policy decisions. + +Secrets should be managed at the platform level. From 9888fc36e0b0b107f802171862553a6ddf6f759b Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Tue, 10 Jan 2023 21:27:51 +0000 Subject: [PATCH 2/7] Elaborate on types of source code and git/github --- .../storing-source-code/index.html.md.erb | 79 +++++++++++++++---- 1 file changed, 64 insertions(+), 15 deletions(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index 4ca73c58..26a62133 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -13,11 +13,11 @@ old_paths: We follow principles set out within the [GOV.UK Service Manual](https://www.gov.uk/service-manual) for managing code we write. -### Sections of relevance: +### Service Manual Sections of relevance: -- [Service Manual, Technology: Use version control](https://www.gov.uk/service-manual/technology/maintaining-version-control-in-coding) -- [Service Manual, Technology: Make source code open](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable) -- [Service Manual, Technology: Securing your information](https://www.gov.uk/service-manual/technology/securing-your-information) +- [Service Manual > Technology > Use version control](https://www.gov.uk/service-manual/technology/maintaining-version-control-in-coding) +- [Service Manual > Technology > Make source code open](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable) +- [Service Manual > Technology > Securing your information](https://www.gov.uk/service-manual/technology/securing-your-information) ### Summary / Highlights: @@ -34,25 +34,74 @@ Additional detail and information is available via the links above. ## Types of source code -- Infrastructure as code -- Project +Source code is broader and wider than just business and presentation code. -## Git / GitHub +How source code is stored and structured will vary by project: -At the Department for Education, we use git and the MIT Licence. +- A project may be composed of multiple services where each has its own repository +- A monolith may have all source code for all purposes stored within the same source code repository +- A mixture may apply where project source code and tests are within one repository, + while infrastructure code may be stored within a separate repository -While some projects remain on Azure DevOps +Examples of source code types and purposes: -and should be migrated across +- **Project source code** + - Code used to meet a user need - i.e., what is normally considered when describing "source code" +- **Test code** + - Code used to evaluate the correctness of the project code + - Depending on the project, test code may involve provisioning infrastructure, deploying a build, + and even running the project code + _(e.g. a headless browser to test the presentation and accessibility of a web page)_ +- **Infrastructure as code** + - Code used to provision and configure the infrastructure a project runs upon +- **CI configuration** + - Code used to inspect, validate, and potentially gate-keep changes being made to project code + - May include GitHub Actions and Azure Pipelines + - Typically triggered on a merge/PR event, but other examples include being triggered on + creation of a particular tag (e.g., one in the format `vX.Y.Z`) or on a timer/cron-basis +- **Deployment code** + - Code used to build, test, and deploy project source code into a running environment + +## Source Code Versioning: Git + +At the Department for Education (DfE) we use [Git](https://git-scm.com/) for source code versioning. + +- Git is decentralised - this means all copies of the repository include the WHOLE history of the repository, + not just a snapshot +- Branches are "cheap" - creating a new branch (or tag) involves just a new pointer at a specific commit + (thus minimal compute and storage implications) +- Hashes/checkums for each file and commit depend on the entire tree - thus, the repository is safe from + surreptitious / malicious / accidental changes to earlier versions of a file without it being + very visible to other users + + + +## Git Repository Hosting - GitHub and Azure DevOps (ADO) + +While not required, most git users will nominate one copy of the git repository to be the authoritative copy. + +- It is possible to self-host a git server for this purpose but, often, this will be a hosted solution such as + GitHub, Azure DevOps, GitLab, or any of the numerous other commercial services available. +- A "hub and spoke" is easier to reason about and keep synchronised +- Integrations with other tools will work with less friction, where they have a single copy to work with + (e.g., automated test/deployment tools, issue/bug management) + + +### GitHub + +Historically, some projects use (and may remain on) private Azure DevOps and/or private GitHub repositories +for legacy reasons, though we are now required to [make new source code open](https://apply-the-service-standard.education.gov.uk/service-standard/12-make-new-source-code-open.html). + +Specifically, we use GitHub for new and migrated work. -Specifically, we use GitHub for new and migrated work though projects remain on Azure DevOps. ### GitHub Organisations -Department for Education source code repositories on GitHub should be stored under an appropriate -organisation, thus giving appropriate oversight and protections to these repositories. +Department for Education (DfE) source code repositories on GitHub should be stored under an appropriate +organisation, thereby giving appropriate oversight and protections to these source code repositories. Specifically: + - The [Department for Education Digital](https://github.com/DfE-Digital) GitHub organisation is used for new and existing source code repositories - This is applicable to production and prototype code @@ -63,10 +112,10 @@ Specifically: GitHub organisation is also used by the DfE - Not the default for DfE-Digital - There is a small cost for accounts to be added to a GitHub organisation. Ask your delivery manager to request your account being added to these GitHub organisations. + ### Work vs Personal GitHub accounts You may use your personal GitHub account, but you should: @@ -101,7 +150,7 @@ Additionally, note that git is _explicitly_ a _decentralised_ source versioning - It is, therefore, not possible to delete/change information within one copy of the repository (e.g., GitHub) and force all other copies to be updated also - It is, therefore, extremely important to prevent non-public content from ever - being added to the git repository in the first place + being added to the git repository in the first place because it cannot be removed ### Secrets From 84b533ef1ee8342b10eec7f77080ae71f64c660d Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Tue, 10 Jan 2023 21:29:10 +0000 Subject: [PATCH 3/7] Auto format --- .../storing-source-code/index.html.md.erb | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index 26a62133..126fa618 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -7,7 +7,6 @@ old_paths: # <%= current_page.data.title %> - ## GOV.UK Service Manual We follow principles set out within the [GOV.UK Service Manual](https://www.gov.uk/service-manual) @@ -31,7 +30,6 @@ for managing code we write. Additional detail and information is available via the links above. - ## Types of source code Source code is broader and wider than just business and presentation code. @@ -74,8 +72,6 @@ At the Department for Education (DfE) we use [Git](https://git-scm.com/) for sou surreptitious / malicious / accidental changes to earlier versions of a file without it being very visible to other users - - ## Git Repository Hosting - GitHub and Azure DevOps (ADO) While not required, most git users will nominate one copy of the git repository to be the authoritative copy. @@ -86,15 +82,14 @@ While not required, most git users will nominate one copy of the git repository - Integrations with other tools will work with less friction, where they have a single copy to work with (e.g., automated test/deployment tools, issue/bug management) - ### GitHub Historically, some projects use (and may remain on) private Azure DevOps and/or private GitHub repositories -for legacy reasons, though we are now required to [make new source code open](https://apply-the-service-standard.education.gov.uk/service-standard/12-make-new-source-code-open.html). +for legacy reasons, though we are now required +to [make new source code open](https://apply-the-service-standard.education.gov.uk/service-standard/12-make-new-source-code-open.html). Specifically, we use GitHub for new and migrated work. - ### GitHub Organisations Department for Education (DfE) source code repositories on GitHub should be stored under an appropriate @@ -115,7 +110,6 @@ Specifically: There is a small cost for accounts to be added to a GitHub organisation. Ask your delivery manager to request your account being added to these GitHub organisations. - ### Work vs Personal GitHub accounts You may use your personal GitHub account, but you should: @@ -123,15 +117,14 @@ You may use your personal GitHub account, but you should: - [GitHub: Add your DfE email address to your account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/) - [GitHub: Use your secondary (DfE) email address for notifications](https://help.github.com/articles/managing-notification-emails-for-organizations/) - ### Repository Requirements Repositories should be: + - [clearly named](/standards/naming-things/), - have an [appropriate licence](/standards/licencing-software-or-code) - have enough documentation that someone new can get started with the project - ## Data Protection Considerations - Git Repositories ### Personal Data @@ -152,7 +145,6 @@ Additionally, note that git is _explicitly_ a _decentralised_ source versioning - It is, therefore, extremely important to prevent non-public content from ever being added to the git repository in the first place because it cannot be removed - ### Secrets You _must_ keep secrets separate from source code, and keep them private. From 8d16595e431c8c425e42efd6bf8276b152630595 Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Tue, 10 Jan 2023 21:56:12 +0000 Subject: [PATCH 4/7] Minor tweaks to git/github organisation content --- .../storing-source-code/index.html.md.erb | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index 126fa618..d3edddc3 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -34,7 +34,7 @@ Additional detail and information is available via the links above. Source code is broader and wider than just business and presentation code. -How source code is stored and structured will vary by project: +How source code is stored and structured will vary by project, based on needs and historical convention: - A project may be composed of multiple services where each has its own repository - A monolith may have all source code for all purposes stored within the same source code repository @@ -107,8 +107,14 @@ Specifically: GitHub organisation is also used by the DfE - Not the default for DfE-Digital -There is a small cost for accounts to be added to a GitHub organisation. -Ask your delivery manager to request your account being added to these GitHub organisations. +If your account is added to a repository without the account being a member of the owning organisation, +it will be counted and labelled as +an ["outside collaborator"](https://docs.github.com/en/organizations/managing-user-access-to-your-organizations-repositories/adding-outside-collaborators-to-repositories-in-your-organization). + +To join a GitHub organisation, follow the guidance and request forms available +via [Digital Tools Support](<%= data.site.digital_tools %>). +As there is a small cost implication for accounts to be added to a GitHub organisation, +this should normally be done via / with support from your Delivery Manager. ### Work vs Personal GitHub accounts @@ -133,10 +139,12 @@ Storage of a git repository must be treated with due care and consideration. This applies whether it is within a central hosted environment or stored elsewhere such on a developer's computer. +Places where we may normally find personally-identifiable information: + - Changes to source code, commits, are annotated with authorship details. - Typically, this is a name and an email address. -- Where the commit is cryptographically-signed, the GPG key used will also have - personal information associated with it such as an email address. + Typically, this is a real name (or username) and an email address. +- Where a commit is cryptographically-signed, the GPG key used will also have + personally-identifying information associated with it (such as an email address). Additionally, note that git is _explicitly_ a _decentralised_ source versioning and control system. @@ -144,6 +152,7 @@ Additionally, note that git is _explicitly_ a _decentralised_ source versioning repository (e.g., GitHub) and force all other copies to be updated also - It is, therefore, extremely important to prevent non-public content from ever being added to the git repository in the first place because it cannot be removed + with 100% confidence (being able to do so is an edge case, not the norm) ### Secrets From df1b65441b69b0096c94e4919816a7645d073dac Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Wed, 11 Jan 2023 11:51:52 +0000 Subject: [PATCH 5/7] Minor re-ordering of paragraphs --- .../storing-source-code/index.html.md.erb | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index d3edddc3..68470f1c 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -34,13 +34,6 @@ Additional detail and information is available via the links above. Source code is broader and wider than just business and presentation code. -How source code is stored and structured will vary by project, based on needs and historical convention: - -- A project may be composed of multiple services where each has its own repository -- A monolith may have all source code for all purposes stored within the same source code repository -- A mixture may apply where project source code and tests are within one repository, - while infrastructure code may be stored within a separate repository - Examples of source code types and purposes: - **Project source code** @@ -60,6 +53,14 @@ Examples of source code types and purposes: - **Deployment code** - Code used to build, test, and deploy project source code into a running environment +How this source code is stored and structured will vary by project, based on needs and historical convention. +For example: + +- A project may be composed of multiple services where each has its own repository +- A monolith may have all source code for all purposes stored within the same source code repository +- A mixture may apply where project source code and tests are within one repository, + while infrastructure code may be stored within a separate repository + ## Source Code Versioning: Git At the Department for Education (DfE) we use [Git](https://git-scm.com/) for source code versioning. From 3fe7cf5b5e8bb38b0add124b083027cc6256f5ce Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Mon, 5 Aug 2024 10:50:47 +0100 Subject: [PATCH 6/7] Fix typo --- source/standards/storing-source-code/index.html.md.erb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index 68470f1c..afd6f3cc 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -69,7 +69,7 @@ At the Department for Education (DfE) we use [Git](https://git-scm.com/) for sou not just a snapshot - Branches are "cheap" - creating a new branch (or tag) involves just a new pointer at a specific commit (thus minimal compute and storage implications) -- Hashes/checkums for each file and commit depend on the entire tree - thus, the repository is safe from +- Hashes/checksums for each file and commit depend on the entire tree - thus, the repository is safe from surreptitious / malicious / accidental changes to earlier versions of a file without it being very visible to other users From 6577913923c2e9d2c9cb1c94537e22bd9baff5c4 Mon Sep 17 00:00:00 2001 From: Roger Howell Date: Mon, 5 Aug 2024 10:51:16 +0100 Subject: [PATCH 7/7] Remove section re: work vs personal GitHub accounts to take into account PR feedback --- source/standards/storing-source-code/index.html.md.erb | 7 ------- 1 file changed, 7 deletions(-) diff --git a/source/standards/storing-source-code/index.html.md.erb b/source/standards/storing-source-code/index.html.md.erb index afd6f3cc..b20dd395 100644 --- a/source/standards/storing-source-code/index.html.md.erb +++ b/source/standards/storing-source-code/index.html.md.erb @@ -117,13 +117,6 @@ via [Digital Tools Support](<%= data.site.digital_tools %>). As there is a small cost implication for accounts to be added to a GitHub organisation, this should normally be done via / with support from your Delivery Manager. -### Work vs Personal GitHub accounts - -You may use your personal GitHub account, but you should: - -- [GitHub: Add your DfE email address to your account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/) -- [GitHub: Use your secondary (DfE) email address for notifications](https://help.github.com/articles/managing-notification-emails-for-organizations/) - ### Repository Requirements Repositories should be: