Increasing modularity of install instructions in workshop website template

[ErinBecker]

Context

Instructors who teach a Carpentries workshop create a workshop website using our workshop-template. This template includes information on variable features of workshops such as location, time and date, and (human) language, as well as fixed information such as our Code of Conduct.

The Setup section of the template is pre-populated with installation instructions for the Bash shell, Git, nano, Python, R, SQLite, and OpenRefine. Instructors creating their workshop website are instructed to remove the installation instructions that are not applicable to their workshop. This is not a complete list of all of the technology used at our workshops, and no one workshop uses all of these tools. Additionally, many DC workshops require learners to download data, and/or install specific libraries or packages before the workshop. These extra steps are not reflected in the template.

Including installation instructions in the workshop template was a good strategy when all of our workshops taught the same or nearly the same set of tools (i.e. when only SWC was using the template), but has become more challenging as we continue to introduce new curricula. In April 2019, I analyzed the sixteen most recent workshop websites for user error and found the following:

• 38% (6) had unnecessary or conflicting setup instructions
• 80% of those with correct setup instructions (8 of 10) created the syllabus manually
• 18% (2) modified syllabus / setup manually but still had incorrect setup
• 6% (1) showed the wrong lesson program
• 6% (1) didn’t use our template
• The only websites with correct installation instructions were those for SWC workshops (but not all SWC workshops had correct instructions).

(Detailed information on this analysis can be found in this Google doc).

This issue has been recognized since at least January 2018, with different proposed solutions ([1] [2]) being rejected or delayed due to difficulty of implementation.

Current status

The current proposal is to add two new variables to the template:

• a curriculum variable - which chooses among different curricula within a lesson program (e.g. dc-geospatial or dc-genomics). This is currently only applicable for DC workshops but is likely to become appropriate for LC workshops in future as new curricula are released.
• a flavor variable - which chooses between different versions of a curriculum (e.g. Python or R)
  (These new variables are in addition to the already existing carpentry variable which specifies the lesson program.)

These variables will work together to populate the schedule, syllabus, and setup sections of the website. For LC and SWC workshops, the setup instructions will remain in the workshop website. For DC workshops, the setup instructions will be removed and learners will be pointed to the setup instructions on the individual curriculum homepage. An example of a curriculum setup page can be seen here for the DC Geospatial workshop.

@fmichonneau has put together a pull request to implement this change.

How will these changes affect me?

• DC Geospatial and DC Genomics Maintainers - these changes will not impact you. Install instructions for these lessons are already localized to a single setup page per curriculum.
• DC Social Science and DC Ecology Maintainers - you will be asked to create a single landing page with all setup instructions for the lessons in this workshop. @fmichonneau and @ErinBecker are available to make this migration if help would be desired!
• LC Maintainers - these changes will not impact you. The workshop website template will populate with install instructions for the Bash shell, Git, and OpenRefine when lc is set as the value of the carpentry variable (nano, Python, R, and SQLite will no longer appear).
• SWC Maintainers - these changes will not impact you. The workshop website template will populate with install instructions for the Bash shell, Git, nano, Python, R, and SQLite when swc is set as the value of the carpentry variable (OpenRefine will no longer appear).

Timeline

If you have any questions or concerns about these proposed changes, please post in this thread by Wednesday, 25 September midnight UTC. @fmichonneau, myself, and members of the @carpentries/lesson-infrastructure-committee will respond here to answer questions and address concerns. If you would like to comment on the technical implementation, you may also leave feedback on the PR in the workshop template repo which implements this change.

[libcce]

Just double checking @ErinBecker the curriculumvariable if set to dc-example will point people to setup specific information in the dc lessons? Using one example, LC OpenRefine, I believe the setup information is more up-to-date in the lesson vs what is in the workshop template? CCing @ostephens just in case. Instead of pointing to the dc lessons, would it be best if this information was shared with all the lessons, for instance, one common OpenRefine setup instructions?

[ErinBecker]

You raise a good point @libcce. I don't know the answer, but I'll try to lay out my thought process so maybe together we can come to a solution!

Trying to untangle the best route for install instructions for DC lessons is challenging, because there are a few things shared with other lessons (e.g. the OpenRefine lessons in the Ecology and Social Sciences workshops is in common with LC), but a lot of install instructions are only relevant to that particular curriculum (e.g. the specific R packages needed for the Geospatial workshop). There are also sometimes datasets that need to be downloaded before the workshop (because of size), so having a single page that covers data and software setup is really nice.

I'm not familiar with the LC OpenRefine lesson - does it require any add-on packages in addition to the basic OpenRefine install? The DC Social Sciences and Ecology lessons use bare-bones OpenRefine.

The LC lessons also have data to download in advance of the workshop, right? Would it be useful to have that on a single page with the software install info for learners?

[fmichonneau]

Just to comment on this:

I believe the setup information is more up-to-date in the lesson

Yes, that's what we see generally. That's the motivation behind the change. The setup instructions in the template will only be a link to the setup instructions that are part of the lesson and tend to be better maintained than in the workshop template.

[ErinBecker]

@fmichonneau - I know that this is the plan for DC workshops (because each DC curriculum has a centralized "workshop overview" page where all of the setup instructions for the full workshop live). My understanding of the current proposal, however, is that this wouldn't be the case for LC workshops - and that the install instructions would continue to appear in the workshop website rather than in the lesson. Am I misunderstanding?

[fmichonneau]

Yes, sorry I can see how my last comment could have been confusing. For LC lessons, (at least in the current PR), the setup instructions would remain on the workshop website rather than pointing to the lessons.

[libcce]

I'd say that the LC OpenRefine installation/setup instructions are more up-to-date than what you find in the worskhop website template. My guess is that LC Unix Shell and Git instructions/setup mirror SWC. LC has its own set of files to work with of course. LC SQL setup varies from SWC. This is just to give you an overview... but my main concern was a duplication of effort and/or setup updates are not shared (where it can be beneficial to the other lessons in the other programs). There are some lesson specific instructions that sit outside of this regarding files used, extensions, etc but isn't there a core part that is common throughout the lessons? I myself have updated the setup for LC OpenRefine and haven't thought about updating the other Carpentries lessons that use OpenRefine (which is a shame because I think it would benefit us all to have this cross lesson Maintenance).

[ErinBecker]

@libcce - I think you've hit on the core issue. Some of our lessons share the same technology (e.g. LC OpenRefine and DC Social Science and Ecology OpenRefine lessons), which suggests having centralized installation instructions as a solution. These centralized instructions have lived until now in the workshop template repo, which is maintained by the @carpentries/lesson-infrastructure-committee. Since these committee members aren't familiar with all of the details of each curriculum (and certainly aren't expected to be!), they're not in the best position to keep instructions up to date with the needs of each lesson. The individual lesson Maintainers are the most familiar with their lessons and therefore better positioned to keep instructions up to date. However (as you've identified), in cases where multiple lessons share the same technology, this means that install instructions can get out of synch. So there are downsides to both options.

I'm of the opinion that having install instructions in the individual lessons is a more robust solution than having them centrally for a couple of reasons:

1. Having them in the lessons gives a single "point of truth". Right now, instructions on workshop websites and lessons often conflict (see stats in initial issue).
2. Learners would only need to look in a single location for both install instructions and data files.
3. Different lessons teaching the same technology would be free to diverge. For example, if one R lesson wants to do everything using tidyverse packages, they can have their learners download that set of libraries during setup. The other R lessons don't need to make that change.
4. Related to Increasing modularity of install instructions in workshop website template #3, but different - lessons on the same technology can have different version requirements. E.g. the DC Geospatial lesson depends entirely on a recent update to the ggplot2 package, so learners need to have that version or newer. Other R lessons can use older versions of the package.

I definitely understand your point @libcce about wanting to keep things in synch in cases where that's appropriate. But I think this be better achieved by having clearer/more systematic communication pathways across lessons on the same technology than by requiring those lessons to use the same (central) install instructions.

[ostephens]

I think these are really good points @ErinBecker - I think these issues - especially 3 & 4 for me - mean its probably better to sacrifice some efficiency in favour of having the instructions in the individual lessons

[hoytpr]

Absolutely yes this seems good to me @ErinBecker . Out local group has well-vetted local "versions" of the python and R lessons (not huge differences but based on learner feedback) and this will keep the official repos consistent. So will there be an official site for groups (e.g. a member Univ.) to contribute customized images, lessons, datasets? My memory says you considered this, and IMO it would be a great resource (but would it need to be monitored/maintained?).

[fmichonneau]

Thank you all for the feedback. The pull request that implements the changes described here is now merged in the workshop template repository.

We opened a second RFC regarding how to update the template for SWC and LC workshops. See #4

[ErinBecker]

If anyone is still watching this thread, I would love a review on the single page setup instructions for Ecology in this PR. I'll be working on a similar PR for the Social Sciences curriculum today and will add a link here for review when it's ready.

[hoytpr]

@ErinBecker Not knowing how much detail you want, and just wanting to help:

If all you wanted to know was if the links work:
"Episodes" link at top not working:
"Setup" link links to setup properly, but the "For a full description of the data used in this workshop see the "data page". links back to setup page (although URL changes to "setup/data")
All other links work.

tl;dr

• Data download link works but it's 22.4 MB is not 38.4 MB. Might be nice to give zip file name as "1314459.zip"
• "data page" link did not work (linked back to itself at /setup/data)
• RStudio cheatsheet link auto-downloads rstudio-ide.pdf (not a problem, but it was different behavior than other links and goes to download location silently)
• You need to install R before you install RStudio (<== emphasis needed)
• Under the "Windows" callout to install R & RStudio, there's only one option which is: If you don’t have R and RStudio installed.
  The double callout is not needed without other options.
  If there are options for previously installed versions, will they be updates or uninstalls?
• R autodownloads v 3.6.1 currently from Cran website. Do we want it to autodownload or do we want to choose a version like with RStudio?
• RStudio needs to be downloaded manually, and for Windows, Is the latest version i.e. 1.2.5001 always correct?
• Might want to tell Windows users the RStudio download is an .exe file.
• RStudio console window should be specified and used to type in "install.packages(c("tidyverse", "RSQLite"))" command.
  NOTE: be careful to avoid the period at the end of the sentence for copy and paste-ers. Might want to place command on separate line.
• There might be confusion saying we are instaling "two" packages in RStudio, when the computer begins installing lots of "packages". Maybe I did something wrong but entering the command "install.packages(c("tidyverse", "RSQLite"))" has taken well over 2hours and is still running!
• There's no mention of how to update your existing Anaconda? Should you update Anaconda, or uninstall? You'd want to do this before anything else... right?
• Should Windows users open Anaconda "Navigator" or use a different choice on Windows? (My Start menu also has "Jupyter Notebook" and "Spyder" under "Anaconda")
• Comments: During and after Anaconda/miniconda installs, the other software installs are light on documentation. A beginner could be very frustrated if the installs are not going perfectly smoothly (for example; why is it taking so dang long for my RStudio to update?). I'd suggest more reassuring text and hand-holding at this point. For beginners installing R+RStudio, Anaconda+Jupyter Notebooks, SQL+DB browser... is a lot.

[ErinBecker]

Wow! Thank you so much for the excellent feedback @hoytpr! This is wonderful. Thank you so so much.

Just one quick note to your comment at the end - no learner would need to install all of the software listed here - the plan is that @fmichonneau is going to set up two different setup URLs that get added to the workshop website depending on whether the instructor puts flavor as R or Python. We may also be able to make the SQL install instructions modular and only included if the workshop actually teaches SQL. My job for this PR is to get the instructions right and then @fmichonneau can do the magic behind the scenes to get the right install instructions to populate depending on the workshop website variables. 😉

I'm going to go through these comments and update the PR. Thanks again so much!

[hoytpr]

Glad to help.
EDIT: After stopping RStudio (I had to actually kill the process) then restarting RStudio this morning, the command: install.packages(c("tidyverse", "RSQLite")) worked perfectly in only a few seconds. No idea what happened yesterday, but if RStudio seems to hang (at least in Windows) the usual fix (stop and restart) seems to work.