From 0543864828d429aeab74aa53becaab625655de7e Mon Sep 17 00:00:00 2001 From: Markus Neteler Date: Wed, 4 Oct 2023 18:28:51 +0200 Subject: [PATCH] docs: update and cleanup of infrastructure.md (#3157) - add Weblate, remove Transifex (after migration to OSGeo Weblate) - major markdown cleanup with `npx prettier --print-width 80 -w doc/infrastructure.md` - various improvements (Co-authored-by: @wenzeslaus) - IRC wording cosmetics - Weblate troubleshooting requires administrator rights - Weblate access rights: point to https://wiki.osgeo.org/wiki/SAC:Weblate --- doc/infrastructure.md | 417 ++++++++++++++++++------------------------ 1 file changed, 182 insertions(+), 235 deletions(-) diff --git a/doc/infrastructure.md b/doc/infrastructure.md index 797f44b0897..8e72b8145de 100644 --- a/doc/infrastructure.md +++ b/doc/infrastructure.md @@ -1,11 +1,7 @@ # How the GRASS GIS Webserver and related infrastructure works -written by M. Neteler -Last changed: June 2023 - -Related Wiki documents: - -* (historical document) +Author: Markus Neteler +Last update: Sep 2023 ## GRASS GIS Source code repository @@ -16,109 +12,105 @@ than in SVN). The GitHub repositories are: -* GRASS GIS core (7+): -* GRASS GIS legacy (3.x-6.x): -* GRASS GIS Add-ons: -* GRASS GIS promotional material: -* GRASS GIS Website (hugo site): -* Github mirror at OSGeo: +- GRASS GIS core (7+): +- GRASS GIS legacy (3.x-6.x): +- GRASS GIS Add-ons: +- GRASS GIS promotional material: +- GRASS GIS Website (Hugo site): +- Github mirror at OSGeo: Git usage: -* [CONTRIBUTING.md file](../CONTRIBUTING.md) -* +- [CONTRIBUTING.md file](../CONTRIBUTING.md) +- Issues: -* -* old trac instance: - -Trac related notes: - -* For easier linking in the Trac Wiki, some macro definitions are used for manual - page refs (G7:modulename) - * -* ZIP file download support in trac (was needed for g.extension) * on trac.osgeo.org: - -```text -/var/www/trac/grass/conf/trac.ini -[browser] -downloadable_paths = /grass-addons/grass7/*/*,/sandbox/*/* -``` +- +- trac instance: (old bugs only) Statistics: -* -* +- ## GRASS Web server Maintainer: M. Neteler -* - * osgeo7-grass: LXD container on osgeo7 () - * OS: Debian Buster - * Apache Server with hugo () - * for migration details (7/2020), see - * ssh login: via jumphost hop.osgeo7.osgeo.org - * deployment via cronjob: -* (CMSMS, replaced in 2020 by above hugo based solution) - * Shared virtual OSGeo machine (osgeo6) hosted at Oregon State University +- + + - osgeo7-grass: LXD container on osgeo7 () + - OS: Debian Buster + - Apache Server with Hugo () + - for migration details (7/2020), see + - ssh login: via jumphost hop.osgeo8.osgeo.org + - deployment via cronjob: + +- (CMSMS, replaced in 2020 by above Hugo based solution) + + - Shared virtual OSGeo machine (osgeo6) hosted at Oregon State University Open Source Lab server: osgeo6.osgeo.osuosl.org) - * Login: via OSGeo LDAP, there is a "grass" LDAP group - * Software: - * OS: Debian Wheezy - * Apache Server with PHP - * Login: via OSGeo LDAP, there is a "grass" LDAP group -* Backups: - * osgeo7-grass: container on osgeo7 is backup'ed, see -* Mirrors: - * rsync, see --> Mirror - * mirror list, see -* RSS feed: offered by hugo at , used at - -* Weekly software snapshots (generated Saturday morning Portland (OR), US time): - * Source code tarball of git (GitHub) - * Linux binary snapshot is compiled on osgeo7-grass - * GRASS is compiled with GDAL, PROJ, SQLite, MySQL, PostgreSQL, FFTW, C++ support - * binary tar.gz and manuals are moved into Web space - -* GRASS user manual HTML: - * generated during compilation of weekly Linux binary snapshot on osgeo7-grass - -* GRASS addons manual HTML: - * generated during compilation of weekly Linux binary snapshot on osgeo7-grass - -* GRASS programmer's manual () - * HTML: cronjob run Wednesday morning Portland (OR), US time - * HTML: cronjob run Saturday morning Portland (OR), US time - * disabled: PDF: cronjob run Saturday morning Portland (OR), US time - -* i18N translation statistics () - * generated during compilation of Linux binary snapshot, stats of - `(cd locale; make)` are extracted into text file - * text file parsed by PHP page and shown as table - * GRASS GIS version is coded in devel/i18n_stats.inc - * for Transifex integration, see below - -* Mailman mailing lists + greylisting (at lists.osgeo.org since 11/2007) - * Mailman is doing the job, only registered users can post - * messages from unsubscribed people is auto-discarded without notification - * the open "weblist" operates instead like this: - * User -> grass-web at lists osgeo.org -> greylisting -> Mailman - -* Backup of mailing lists (mbox files) - * nightly backup at OSGeo.org, bacula - -* Web statistics - * See URL at - * cronjob script: /osgeo/scripts/update_logs.sh + - Login: via OSGeo LDAP, there is a "grass" LDAP group + - Software: + - OS: Debian Wheezy + - Apache Server with PHP + - Login: via OSGeo LDAP, there is a "grass" LDAP group + +- Backups: + + - osgeo7-grass: container on osgeo8 is backup'ed, see + +- Mirrors: + + - rsync, see --> Mirror + - mirror list, see + +- RSS feed: offered by Hugo at , used at + +- Weekly software snapshots (generated Saturday morning Portland (OR), US time): + + - Source code tarball of git (GitHub) + - Linux binary snapshot is compiled on osgeo7-grass + - GRASS is compiled with GDAL, PROJ, SQLite, MySQL, PostgreSQL, FFTW, C++ support, + see + - binary tar.gz and manuals are moved into Web space + +- GRASS user manual HTML: + + - generated during compilation of weekly Linux binary snapshot on osgeo7-grass + +- GRASS addons manual HTML: + + - generated during compilation of weekly Linux binary snapshot on osgeo7-grass + +- GRASS programmer's manual () + + - HTML: cronjob run Wednesday morning Portland (OR), US time + - HTML: cronjob run Saturday morning Portland (OR), US time + - disabled: PDF: cronjob run Saturday morning Portland (OR), US time + +- Mailman mailing lists + automated greylisting (at lists.osgeo.org since 11/2007) + + - Mailman is doing the job, only registered users can post + - messages from unsubscribed people is auto-discarded without notification + - the open "weblist" operates instead like this: + - User -> grass-web at lists osgeo.org -> greylisting -> Mailman + - for greylisting, see + +- Backup of mailing lists (mbox files) + + - nightly backup at OSGeo.org, bacula + +- Web statistics + - Matomo: (not publicly accessible; + access: Markus Neteler) + - Selected stats: Summary: The system should run almost autonomously. ## WinGRASS maintenance scripts -* +See ## GRASS Mailing lists @@ -126,167 +118,109 @@ Maintainer: Markus Neteler Available lists: -* at OSGeo.org (): - * grass-abm Integration of GRASS with JAVA based agent based modeling - (ABM) - * grass-announce GRASS announcements - * grass-commit Mailing list to distribute GRASS-CVS commits - * grass-dev GRASS GIS Development mailing list - * grass-es La lista de correo de GRASS GIS en español - * grass-psc GRASS-PSC: GRASS Project Steering Committee - * grass-stats GRASS and statistical software - * grass-translations Translation of GRASS (i18N) - * grass-user GRASS user list - * grass-web GRASS website mailing list - -* OLD, UNUSED: at FBK-irst (): - * grass-commit-addons Mailing list to distribute GRASS Addons-SVN commits - * grass-gui GRASSGUI mailing list - * grass-qa GRASS Quality Assessment and monitoring list - * grass-windows winGRASS * Using GRASS on MS-Windows systems mailing list +- at OSGeo.org (): + - grass-abm: Integration of GRASS with JAVA based agent based modeling + (ABM) + - grass-announce: GRASS announcements + - grass-commit: Mailing list to distribute GRASS Github commits + - grass-dev: GRASS GIS Development mailing list + - grass-es: La lista de correo de GRASS GIS en español + - grass-psc: GRASS-PSC: GRASS Project Steering Committee + - grass-stats: GRASS and statistical software + - grass-translations: Translation of GRASS (i18N) + - grass-user: GRASS user list + - grass-web: GRASS website mailing list Notes: -* grass-announce: - * moderated by M. Neteler - * has monthly password reminder disabled to avoid leakage into publicly +- grass-announce: + - moderated by M. Neteler + - monthly password reminder is disabled to avoid leakage into publicly archived lists -* grass-commit is receiving posts from the GRASS SVN at osgeo.org. Not open for +- grass-commit is receiving posts from the GRASS Github. Not open for other postings, they will be trashed automatically -* grass-web is an open list (posting without subscription possible) with (Google) - spam filter - * moderated by M. Neteler to avoid spam -* OLD, UNUSED: grass-qa is receiving posts from the GRASS Quality Control System - at Ecole Polytechnique de Montreal, Canada. Not open for other postings. +- grass-web is an open list (posting without subscription possible with + moderation), moderated by M. Neteler to avoid spam ## GRASS Wiki Maintainer: Martin Landa, Markus Neteler -* -* Mediawiki -* mirrored at CZ Tech University -* requires registration to keep spammers out +- +- Mediawiki software +- requires registration to keep spammers out -Summary: The system should run almost autonomous. An eye must be be kept on people -trying to spam the site +Summary: The system should run almost autonomous. An eye must be be kept +on people trying to spam the site. Several layers of registration protection +are in place due to excessive spam. Macros for manual pages (src, cmd, API, ...): -* +- ## GRASS IRC -Channel: irc://irc.freenode.net/grass +Channel: irc://irc.libera.chat/grass Web based client: See -* channel owner: Alessandro Frigeri < afrigeri unipg.it > ("geoalf") -* quasi guru level: Markus Neteler ("markusN") -* further operators: - * Jachym ("jachym") - * Luca ("doktoreas") - * Soeren ("huhabla") - * Brad ("bdouglas") +- channel owner: Alessandro Frigeri ("geoalf") +- quasi guru level: Markus Neteler ("markusN") +- original (freenode) operators: + - Jachym ("jachym") + - Luca ("doktoreas") + - Soeren ("huhabla") + - Brad ("bdouglas") ## GRASS Bugtracker Current bugtracker (Jan 2020 - today): -* - -Old bugtracker (Jan 2008 - Jan 2020): - -* -* posted new bugs and comments to grass-dev list -* Settings: - -Old tracsvn (OSGeo server) (Dec 2007 * Mai 2019) - -```text -/var/www/trac/env/grass/conf/trac.ini - downloadable_paths = /grass-addons/grass7/*/*,/sandbox/*/* - path = /var/www/grass/htdocs - link = - src = site/grasslogo_vector_small.png - smtp_always_cc = grass-dev@lists.osgeo.org - smtp_replyto = grass-dev@lists.osgeo.org - url = - .dir = /var/www/svn/repos/grass - base_url = - database = postgres://postgres@/trac_grass -``` - -Very old bugtracker (Jan 2007 * Dec 2008): - -* -* gforce, sponsored by Intevation GmbH, Germany -* spamassasin spamfilter locally, bogofilter at grass-dev list -* needs `noreply*wald.intevation.org` to be enabled as alias in Mailman - -Very very old bugtracker (Dec 2000 * Dec 2006): +- -* -* webRT, sponsored by Intevation GmbH, Germany -* spamassasin spamfilter locally, bogofilter at grass-dev list -* reports are directly sent to GRASS Developers mailing list for notification -* TODO: migrate to trac +Old bugtrackers: see -## GRASS Addons +## GRASS GIS Addons Maintainer: Martin Landa and Markus Neteler Details: -* Windows-addons: grass-addons/utils/addons/README.txt -* Addon manual pages cronjob: -* Rendered manuals: +- Windows-addons: grass-addons/utils/addons/README.txt +- Addon manual pages cronjob: +- Rendered manuals: -The redirect to the latest grass7x directory is defined on grass.osgeo.org: - /etc/apache2/includes/grass.osgeo.org.inc +The redirect to the latest grassX directory is defined on grass.osgeo.org: +/etc/apache2/includes/grass.osgeo.org.inc Procedure building of binaries (Windows): -* Addons module are compiled on build server, currently at the CTU in Prague) and +- Addons module are compiled on winGRASS build server, at the CTU in Prague) and publishing their manual pages on publishing server, i.e. grass.osgeo.org. -* A new compilation is triggered every time when a commit is done in the Addons-SVN. -* Logs: - * Linux log files: (compiled on +- A new compilation is triggered every time a commit is done in the Addons repo. +- Logs: + - Linux log files: (compiled on `grasslxd` on `osgeo7`) - * Windows log files: + - Windows log files: Procedure of granting write access to Addons repo: -* Request procedure: -* Adding OSGeo-ID: -* Adding contributor: - (via SVN commit) -* Confirm request in grass-psc and give instructions concerning code style etc +- Request procedure: +- Adding OSGeo-ID: +- Adding contributor: + (via git commit) +- Confirm request in grass-psc and give instructions concerning code style etc (see archive for examples) -XML file for g.extension: +XML file for g.extension: -* generated in grass-addons/utils/addons/grass-addons-publish.sh +- generated in grass-addons/utils/addons/grass-addons-publish.sh ## GRASS Travis CI Maintainer: Martin Landa -* -* -* OLD: -* - -Travis CI control files: - trunk/.travis/ - linux.before_install.sh - linux.install.sh - linux.script.sh - -Maintenance script: - -* - -The github update is run as a cronjob on server "geo102" (CTU, CZ). +- +- ## GRASS CI: GitHub Actions @@ -294,56 +228,69 @@ Started Apr. 2020 Maintainer: Vaclav Petras -* -* Details: - -* CI workflow with: - * A build job which is not parallelized and is meant for clear & relatively fast +- +- Details: +- CI workflow with: + - A build job which is not parallelized and is meant for clear & relatively fast check of compilation and building in general. (Duplicating what is running on Travis) - * A test job which of course needs to build, but the main focus is to run tests, + - A test job which of course needs to build, but the main focus is to run tests, so the compilation is parallelized (depending on nproc) and thus potentially less readable. This runs the whole test suite. (You need to run it locally to see the actual error, but you can see which tests are failing.) -* Static code analysis/Code quality check using Flake8 with separate checks for +- Static code analysis/Code quality check using Flake8 with separate checks for python/grass, gui/wxpython, scripts and temporal directories. - * Configurations ignore different lists of Flake8 errors. The idea is to reduce + - Configurations ignore different lists of Flake8 errors. The idea is to reduce that to minimum. - * Code in testsuite directories is also ignored for now, but should not be in + - Code in testsuite directories is also ignored for now, but should not be in the future. -Helper files placed to .github/workflows +Helper files placed in .github/workflows/ ## GRASS Coverity Scan Maintainer: Markus Neteler -* +- + +## User message translation management (i18N) + +Messages are extracted with `gettext` message macros. + +Translations may be done using the OSGeo Weblate platform: + +- Weblate: + - GRASS GIS Weblate server: + +Anyone with OSGeo-LDAP access can work on the translations. + +For technical background and access rights of the Weblate installation, +see . -## Transifex translation management +### How Weblate works -i18N gettext messages: +When a developer makes a GRASS GIS repo commit on GitHub, GitHub then calls +the webhook on Weblate which triggers a refresh of Weblate's git copy of the +GRASS GIS repo. -* Dashboard: -* Auto-update URL to fetch files: - * - * Menu: Resources - * Use: "Auto update resources" button -* Weblate: +For pushing translations back to GitHub, there is a setting in Weblate for that +which defaults to 24 hrs (accumulates translations over a day). Then a pull +request with the translations will be opened in the GRASS GIS GitHub repo. -## OLD: GRASS Quality Control +### Weblate troubleshooting -Maintainer: Prof. Giulio Antoniol +In case the Weblate's git copy of the GRASS GIS repo does not update due to +a conflict or whatever reason: -* offline. -* - was implemented and sponsored by Ecole Polytechnique de Montreal, Canada -* Realtime analysis has been sent to: +Log into Weblate (requires administrator rights) and switch to +. Therein click on +"Manage" -> "Repository Maintenance", choose the "Update" button, +"Update with merge without fast-forward". If successful, this will create +another pull request in the GRASS GIS repo (trigger with "Push" button). -Further notification/functionality test systems: +## Related Wiki documents -* posts into #grass IRC channel -* posts into #osgeo-commits IRC channel +- (historical document) ## Previous hosting sponsors @@ -353,10 +300,10 @@ Source Lab (OSUOSL), Oregon, USA Numerous institutions have sponsored the GRASS Project with Hardware/Bandwidth (list of master site hosting): -* 1997-1999: Institut fuer Landschaftspflege und Naturschutz (ILN), Universitaet +- 1997-1999: Institut fuer Landschaftspflege und Naturschutz (ILN), Universitaet Hannover, Germany -* 1999-2001: Institut fuer Physische Geographie und Landschaftsoekologie, +- 1999-2001: Institut fuer Physische Geographie und Landschaftsoekologie, Universitaet Hannover, Germany -* 2001-2008: ITC-irst, Trento, Italy -* 2009-2010: Telascience.org at San Diego Supercomputer Center, California, USA -* 2010-today: Oregon State University | Open Source Lab, USA +- 2001-2008: ITC-irst, Trento, Italy +- 2009-2010: Telascience.org at San Diego Supercomputer Center, California, USA +- 2010-today: Oregon State University | Open Source Lab, USA