… helix config in accordance to new save file structure

I have created the parse_mbox_latest_date and refresh_mbox functions. The latter function deletes the latest year and month mbox file that is currently downloaded (identified by parse_mbox_latest_date), and redownloads that along with any file beyond up until the current year. The naming convention of the downloaded files are also changed to what we have agreed on. Just to note, download_mod_mbox REMAINS UNCHANGED since I'm only using download_mod_mbox_per_month.

…ted refresh_pipermail, updated news

Found out that the pipermail downloader function already downloads the files by month and year, so all I really needed to do was change it so that it downloads the files as mbox files (change the extension from .txt to .mbox). Created the refresher for pipermail. I had no need to create a parse latest pipermail since they were mbox files anyway.

…stency

…to ensure it does not download files past current year and month

Added checks in the aforementioned functions so that the refreshers won't download "mail from the future"

…ountered

Done as requested by Carlos

- Remove archive_url and archive_type parameters from download_pipermail().
- Add start_year_month and end_year_month parameters for date filtering.
- Remove convert_pipermail_to_mbox() function, as download_pipermail() now handles file conversion automatically.
- Change file naming convention to 'kaiaulu_'YYYYMM.mbox'.
- Attempt to download and decompress files directly without saving .gz to disk, but could not establish a valid connection.

Signed-off-by: Dao McGill <[email protected]>

…mail()

- Modified helix.yml to use [[“mailing_list”]][[“pipermail”]][[“project_key_1”]]
- Added project_key_2 to helix.yml
- Created /vignettes/download_mail.Rmd to document information about pipermail downloader
- Made function calls explicit for external libraries
- ISSUE: Build -> Check is not passing. Seems to be having issues with utags_path, even though I changed the path to the one for universal-ctags in tools.yml

…process_gz_to_mbox_in_folder()

- download_pipermail: Attempts to download .txt file first. If unavailable fallback to .gz. If using .gz file, unzips and writes output in .mbox
- Added log messages
- download_pipermail: Added timeout parameter to deal with case that server takes too long to respond
- Added refresh_pipermail function
- Updated vignettes/download_mail.Rmd to include refresh_pipermail
- Added process_gz_to_mbox_in_folder function

…il refresher.

- Replaced paste0 with stringi::stri_c
- Removed create directory if does not exist
- Added more verbose descriptions/comments
- Added dividers within functions
- Added verbose parameter
- Added else block for refresher
- Added call to process_gz_to_mbox_in_folder at end of refresher
- parse_mbox: stri_replace_last was not working, changed it to stringi::stri_replace_last_regex
- Tested parse_mbox. Perceval was not returning any output. I will look further into why this is happening.

…il refresher.

- Replaced paste0 with stringi::stri_c
- Removed create directory if does not exist
- Added more verbose descriptions/comments
- Added dividers within functions
- Added verbose parameter
- Added else block for refresher
- Added call to process_gz_to_mbox_in_folder at end of refresher
- parse_mbox: stri_replace_last was not working, changed it to stringi::stri_replace_last_regex
- Tested parse_mbox. Perceval was not returning any output. I will look further into why this is happening.

Signed-off-by: Dao McGill <[email protected]>

…uh/kaiaulu into 284-mbox-download-refresher

Updated parameters for download_mod_mbox to use Apache Pony Mail links as Apache lists now redirect there
- Modified downloads to use YYYYMM  instead of YYYY
- Removed the option for downloading by year for clearer functionality.
- Updated vignette/download_mail.Rmd

Signed-off-by: Dao McGill <[email protected]>

- Created `refresh_mod_mbox` function to automatically refresh mailing list archives downloaded using Mod Mbox.
- The function checks for the latest downloaded file, deletes it, and redownloads the archive from that month to the current date.
- Added documentation for `refresh_mod_mbox` to the notebook.

Signed-off-by: Dao McGill <[email protected]>

- Updated vignettes/download_mail.Rmd to working version
- Fixed errors in helix.yml
- Minor edits in mail.R

Signed-off-by: Dao McGill <[email protected]>

- Check works locally
- Commit all changed files

- Renamed for match with convention set by issue #230

Signed-off-by: Dao McGill <[email protected]>

- Reverted name change of save_folder_mail
- Removed previous documentation file for mail (download_mod_mbox.Rmd)
- Updates to dowmload_mail.Rmd

This reverts commit f0027dc.

- parse_mbox_lateset_date() now uses new naming convention for files
- Added to download_mail.Rmd
- Fixed documentation for download_pipermail()

Signed-off-by: Dao McGill <[email protected]>

- added parse_mbox_latest_date

- Update pkgdown.yml
- Set eval to False for notebook
- Added warning for failed downloads
- Added check for missing months in the date range within save_folder_path
- Changed mbox_path in parsers to mbox_file_path
- Use gt package to view tables
- Made changes so Knit works for download_mail.Rmd
- Updated exec/mailinglist.R to use new functions
- To do: Use getter functions once they are merged

Signed-off-by: Dao McGill <[email protected]>

R/example.R contained an unused parameter,
triggering warnings on build.

Signed-off-by: Carlos Paradis <[email protected]>

Actions is failing due to being
unable to install XML. Some new error
yet again on Actions. Trying to make
the version requirement less strict
to see if it is able to install.

Signed-off-by: Carlos Paradis <[email protected]>

The story is a bit too dry and assumes much
of the user. The file format stored is not
brief. Modified it a bit to add an example
on how it can be revised.

Signed-off-by: Carlos Paradis <[email protected]>

In case the error of XML compile is tied to
this issue: r-lib/actions#559
revert to 4.1 to see if it solves the problem.

Signed-off-by: Carlos Paradis <[email protected]>

Issue seems to be tied to gcc compiler
not working. Attempt to bump OS X version
up rather than downgrade R.

See GitHub Action for CHECK on the line:

"checking whether the C compiler works... no"

right before: "ERROR: configuration failed for package ‘XML’"

Signed-off-by: Carlos Paradis <[email protected]>

- Refactored the download_mail.Rmd notebook to expect the use of the getters from R/config.R (i #230 contains the getter functions in R/config.R).

- This should fail until the getters are merged.

Signed-off-by: Dao McGill <[email protected]>

- The project configuration section of a notebook was incorrectly using the project directory (kaiaulu/) as its working directory rather than the directory that it resides in (/vignettes/) as its working directory.

This commits perform a major refactoring of how Kaiaulu interface with config files, and the suggested folder organization to store rawdata and analysis. 

The configuration files are generalized to account for anomaly cases when performing project analysis. For instance, long-lived projects may contain multiple repositories, issue trackers, mailing list, etc. The new template of the configuration file allows to account for this information. 

Moreover, changes to the config template cascaded in changes to all notebooks, as the access to the config was hardcoded to the file organization. A new set of get_ functions should make this the last commit that change in template cascades into notebooks. All actively maintained notebooks  (not prefixed by underline under vignettes/) have been updated to use the get functions. Future changes, therefore, will only affect the get() functions in R/config.R.

The folder organization of the filepaths has also been modified. Previously, filepaths assumed as default in the versioned config files suggested organizing code as rawdata/git_repo/projectX ; rawdata/jira/projectY. This organization was not practical for sharing data manually, as the user would need to zip several folders individually. The new organization is now rawdata/projectX/git_repo ; rawdata/projectX/jira. This means users only need to zip projectX and that will contain all the data wanted to be shared.

A minor typo on graph.R was also fixed for merge function calls from `sorted=` to `sort=`.

Moves some of the narrative to functions. Function
documentation can be accessed on demand instead of
placed on Notebook. Some of the docs was moved to
inline on the function of pipermail.

Signed-off-by: Carlos Paradis <[email protected]>

revert utags to match master

Signed-off-by: Carlos Paradis <[email protected]>

Signed-off-by: Carlos Paradis <[email protected]>

The rest of kaiaulu does not use this.

Signed-off-by: Carlos Paradis <[email protected]>

added url tag, etc.

Signed-off-by: Carlos Paradis <[email protected]>

…ration

- Use refresh in exec
- Use getters in exec
- Change kaiaulu version in exec
- Fix paths in helix.yml
- Remove unused parameters
- Change cat to message

Signed-off-by: Dao McGill <[email protected]>

- Change exec from 'tabulate' to 'parse'
- Will update issue 310 to use this exec instead of its own

Signed-off-by: Dao McGill <[email protected]>

- Fixed uri parameter in parse_mbox for perceval
- Small change in example

Signed-off-by: Dao McGill <[email protected]>

Signed-off-by: Dao McGill <[email protected]>

This reverts commit ffb5c9c.

This reverts commit fd97af0.

This reverts commit 56dff9c.

This reverts commit 092e2ab.

This reverts commit c797219.

This reverts commit e55b6e2.