284 MBox Refresher

[ian-lastname]

… helix config in accordance to new save file structure

I have created the parse_mbox_latest_date and refresh_mbox functions. The latter function deletes the latest year and month mbox file that is currently downloaded (identified by parse_mbox_latest_date), and redownloads that along with any file beyond up until the current year. The naming convention of the downloaded files are also changed to what we have agreed on. Just to note, download_mod_mbox REMAINS UNCHANGED since I'm only using download_mod_mbox_per_month.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 206 lines in your changes are missing coverage. Please review.

Project coverage is 36.42%. Comparing base (2bc8d14) to head (b5be04e).
Report is 3 commits behind head on master.

Files	Patch %	Lines
R/mail.R	0.00%	206 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #295      +/-   ##
==========================================
- Coverage   39.79%   36.42%   -3.37%     
==========================================
  Files          20       20              
  Lines        3091     3495     +404     
==========================================
+ Hits         1230     1273      +43     
- Misses       1861     2222     +361     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[carlosparadis]

Thank you @ian-lastname. I will try to make a pass before our meeting!

[carlosparadis]

@ian-lastname thanks!

[daomcgill]

Hi @carlosparadis,

I've refactored the download_pipermail() function.

Proposed Changes

• Removed archive_url and archive_type parameters to simplify the function interface.
• Added start_year_month and end_year_month parameters for date filtering.
• Removed convert_pipermail_to_mbox(), since download_pipermail() handles file conversion automatically.
• Changed the file naming convention to ''kaiaulu_'YYYYMM.mbox' for consistency.

I've added temporary configuration entries in helix.yml for testing purposes:

conf <- yaml::read_yaml("conf/helix.yml")
mailing_list <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["mailing_list"]]
start_year_month <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["start_year_month"]]
end_year_month <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["end_year_month"]]
save_folder_path <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["save_folder_path"]]

And this function call:

download_pipermail(
  mailing_list = mailing_list,
  start_year_month = start_year_month,
  end_year_month = end_year_month,
  save_folder_path = save_folder_path
)

Testing Results

• Retrieves links according to date range
• Downloads and renames files from links
• Unzips and cleans gz files
• Saves files as mbox with new naming convention
• Note: I attempted to download and decompress the files directly without saving the .gz to disk, but could not establish a valid connection. The current implementation still downloads the gzipped file and then removes it after unzipping.

[carlosparadis]

@daomcgill I made an inline comment to reply to your question of changes, since there may be a bit of misunderstanding. Let me know if you can't find it.

Two sanity checks:

• Did you try running parse_mbox() to see if it works on the renamed files yet?
• Does running the refresh function pointed at a empty folder downloads the files up to date?
• Does deleting the most recent mbox file download only the newest file back?
• Does running the refresh without deleting any file also update the more recent file only? (to add more recent day's worth of data to the current file)
• Suppose you ran the current refresh today, Sep 16. Then also supposed the next time you executed again the refresh was Oct 2. Would it overwrite the Sep file, and then download a Oct file?

[carlosparadis]

@daomcgill i've added some inline comments, for sanity sake you can reply to each comment directly in line on these since they are more specific.

[carlosparadis]

@daomcgill look! the check for CMD is passing!! What sorcery you did to fix this thing? I never seen this test error before. Hmm..

[daomcgill]

@carlosparadis I wish I could tell you which change I made to make the CMD check pass, but I really have no clue. Hopefully I can get the tests working next. The error message is very strange.

[carlosparadis]

The R version change is a trap: I see the error reverted back to the one of my first fix a month ago:

Package required but not available: ‘XML’

XML is not available before 4.4 anymore was my conclusion. At 4.5 other type of error appeared. Also, I'd revert back to the commit at least your unit test worked, now both are failing!

[daomcgill]

@carlosparadis I changed it back to the version that was passing the check. Now it is not working with the same version that was passing the checks before.
Locally, I had the exact same errors in my tests, which resolved themselves when I changed both the tools in kaiaulu/ and the one in the tests to have working paths to utags and perceval. Not sure where to go from here.

[carlosparadis]

At the rate this is going, we might as well call an exorcist. Focus on your M3, and I will take a look at this again. Don't commit again so we don't enter in a merge conflict.

[daomcgill]

@carlosparadis Sounds good. I resolved most of the review points, but did not change the notebook narrative. Should I work on that before going back to M3?

[carlosparadis]

I would say work on it later on. Let's wrap the functionality of M3 first and use that as a backup task!