Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use R_READABS_PATH as a cache #49

Closed
HughParsonage opened this issue Sep 30, 2019 · 3 comments
Closed

Use R_READABS_PATH as a cache #49

HughParsonage opened this issue Sep 30, 2019 · 3 comments

Comments

@HughParsonage
Copy link
Collaborator

We could use the path R_READABS_PATH:

  1. to cache the results of read_abs as a .fst file, to improve performance, and perhaps harness the random-access features of that file.
  2. using the modification time, the maximum date in the file, or, say, a file containing the MD5 hashes, to determine whether the local file needs to be updated when reading. (add ability to check if local file exists & is up to date #46 )

The eventual goal is for the user to just type read_abs("6345.0") and for readabs to:

  1. if the file exists and is up-to-date---simply read it immediately
  2. if the file does not exist or is not up-to-date---download the .xls file, tidy it, and then read the tidy files immediately on subsequent occasions.

without the user having to tradeoff performance against accuracy/recency.

@MattCowgill
Copy link
Owner

I like the idea @HughParsonage!

The ABS Time Series Directory (the crummy old API that the package is built around) does return a value for "Frequency" and "ProductReleaseDate" (which isn't always the same as the modification date) for each time series. We could use those to figure out whether the file is (likely to be) up to date

@HughParsonage
Copy link
Collaborator Author

Cool. Could I get push access to this repo? Makes it easier to collab rather than messing about with forks. Don't need master branch push permission.

@MattCowgill
Copy link
Owner

Done

Let me know what you're thinking!

HughParsonage added a commit that referenced this issue Oct 5, 2019
MattCowgill added a commit that referenced this issue Nov 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants