- Added a default expiration time for disk cache in
fetch_iquizoo_mem()of 7 days.
- Moved data.iquizoo to Suggests. The logic behind is the same as that of preproc.iquizoo.
- Remove
project_namecolumn fromusersdata.
- Added
cacheargument totar_prep_iquizoo()to optionally specify how to cache the fetched configuration data. - Support to customize the columns to be kept in the output of
users. See [get_users_props_names()] for all available columns, and customize it by settingsubset_users_propsargument intar_prep_iquizoo(). - Update contents sql template to support setting course name and game name (#90).
- Let
usersupdate based on progress hashes. This is important when more users are added in projects.
- Enhance documentation of
tar_prep_iquizoo(). - Support new dark theme of pkgdown.
- Let
paramsandcontentsarguments be mutually exclusive intar_prep_iquizoo().
- Fixed deprecation of
groupsargument in favor ofgroup.
- Added
"bit64"to packages option of targets in order to keep the integer64 class in the output data.
- Fixed a regression issue of
tar_prep_raw()thatpreproc_data()is not correctly tracked by targets pipeline.
- Separate
tar_prep_proj()intotar_prep_hash()andtar_fetch_users().
- Added
parse_data(), previously named aswrangle_data(). See psychelzh/preproc.iquizoo#86.
- Fixed compatibility issue with 'data.iquizoo' 2024.03.31-2.
- Bumped minimum version of 'data.iquizoo' to 2024.03.31-2.
- Added
"preproc"option inaction_raw_dataargument oftar_prep_raw().
- Now
queryargument offetch_data()is optional. If not specified, the default query stored in the package will be used.
- Added
suffix_formatargument tofetch_data()to specify the format of suffix in the query file. This is useful when you want to use a different format of suffix in the query file. - Enhanced the documentation of
fetch_data(). - Let package not depend on dplyr, tidyr and purrr packages (#84).
- Exported more targets factory functions:
tar_prep_proj(),tar_fetch_data(),tar_prep_raw(). - Do not add
progress_hashobjects whencheck_progressis set toFALSEintar_prep_iquizoo().
- Make the default cache location more intuitive as
~/.cache/tarflow.iquizoo. This will unavoidably invalidate existing caches for old pipelines, but the pipeline targets will not be affected.
- Convert
fetch_iquizoo_mem()as a function factory to avoid cache location error.
- Exported
fetch_iquizoo_mem(), which is a cached version offetch_iquizoo(), i.e., the results offetch_iquizoo()will be cached.
- Ensure all internal SQL query templates end with semicolon. This will unavoidably invalidate existing targets for old pipelines.
- Removed
preproc_data()andwrangle_data()functions. Now all data preprocessing are done inpreproc.iquizoopackage.
- Added workaround using tidytable package when the type compatibility triggers an error in
preproc_data(). - Added a warning when no non-empty raw data found in
preproc_data(). - Added support for triggering a warning for
wrangle_data()parsing error andpreproc_data()data binding error.
- Fixed an issue of fetching data when games were distributed on different projects. A regression issue introduced by 3.8.0.
- Added
combineargument intar_prep_iquizoo(). This will enable users to specify freely how to combine the data from branches. See?tar_prep_iquizoofor details.
- Let data from single games be fetched into one targets so that the total targets number could be reduced.
- Fix a bug in
fetch_data()because the data table name in the database is actually based on the project creation time.
- Remove
"all"option fromwhatargument. Specify multiple values if you want to fetch multiple types of data.
- Renamed
use_targets_template()asuse_targets_pipeline(). Although arbitary,"pipeline"is a little more accurate than"template".
- Used Apache License 2.0 instead of MIT License now.
- Fix a compatibility issue with R 4.2.0 which was introduced since 3.6.0.
- Renamed
use_targets()asuse_targets_template()to avoid name masking withtargets::use_targets(). - Renamed
prepare_fetch_data()astar_prep_iquizoo()to obey the name convention of targets factory.
- Enhance the organization of pkgdown reference.
- Enhanced documentations for
prepare_fetch_data().
- Let
fetch_data()extractcourse_dateautomatically. In this way, thecontents(regardless based on template or feeding directly) does not require thecourse_datecolumn from now on.
- Use
data.iquizoo::match_preproc(). - Added
quietlyargument tosetup_option_file(), so now messages are suppressed when loading package. - Added more test cases against database settings. Note test cases do not cover odbc driver for now.
- Let pipeline perform raw data parsing and indices calculation on combined raw data from single tasks (#73).
- Let
prepare_fetch_data()signal error whencontentscontains no data. - Rename the target
contentsascontents_originto avoid possible name conflict withcontentsinput argument. This is a limitation of {targets} package.
- Added
contentsargument inprepare_fetch_data()to support pre-fetchedcontentsas contents configuration. This is useful if you have already fetched the contents data and want to use it directly.
- Support
action_raw_dataargument inprepare_fetch_data()to specify the action of raw data. This is useful when you want only the parsed raw data and not the indices, e.g.,action_raw_data = "parse"will not perform indices calculation. - Added
raw_data_parsedtargets combination and removedraw_datatargets combination, which should be a potential bug for the unparsedraw_datatargets combination is not really the intent.
- Fix a roundtrip issue, see this issue from {bit64} package.
- Ensure
tarchetypes::tar_map()only rely on columns ofproject_id,game_idandcourse_datefrom template SQL output.
- Added two functions
setup_option_file()andcheck_source()to help setup the database connection option file and check if the database is ready (#71).- To ensure option file is correctly set up, you should specify these three environment variables:
MYSQL_HOST,MYSQL_USER,MYSQL_PASSWORD. See option file template for details. - Note if
check_source()returnsFALSE, this package will callsetup_option_file()once to setup the option file in loading phase. You should check if the option file is correctly set up.
- To ensure option file is correctly set up, you should specify these three environment variables:
- Let
userstemplate be independent of input parameters. This means advanced users could minimally specify thecontentstemplate only.
- The
RMariaDB::MariaDB()database driver is detected with higher priority, andodbc::odbc()is technically deprecated, although still supported (not sure for working) for now (#68). - Renamed
fetch_parameterized()asfetch_iquizoo(). The latter is shorter and echoes with the package name. - Removed
fetch_batch()as vector parameters are supported byDBI::dbGetQuery().fetch_iquizoo()handles both vector and scalar parameters.
- Supported to remove duplicates in users. Useful when different projects from the same organization share the same users.
- Enhance some internal logic.
- Added
setup_source()to specify the data source. Argumentsdriver,dsnandgroupsoffetch_parameterized()now live in this function. - Let
paramsargument offetch_parameterized()be optional (but not after...) when no parameters inquery. Note this is not checked beforehand, so you should take care of whether there are parameters. - Supported custom SQL templates. When users want to use different parameter names other than the default one, they could support their own paired with SQL templates. See
setup_templates()for details (#66).
- Enhanced progress hash.
- The name is changed from
project_hashtoprogress_hash, which is more informative. So the argumentalways_check_hashofprepare_fetch_data()is changed tocheck_progressaccordingly. - Now if there are multiple projects, the hash will be separated for each project.
- The SQL now is independent of the user's parameters (#67).
- The name is changed from
- Fixed a bug of named parameters when
RMariaDB::MariaDB()is used.
- Fixed a bug that
fetch_preset()does not pass...tofetch_parameterized()correctly.
- Changed parameters to organization name and project name. Former course specification is removed.
- Support specify
dsnandgroupsin options bytarflow.dsnandtarflow.groupsrespectively. This is useful when default values are not suitable for you. - Added project users to pipeline.
- Added support for auto-check whether the projects data are up-to-date. This turned on by default, and you can turn it off by setting
always_check_hashtoFALSEinuse_targets().
- Fix a bug that targets flow will fail with meaningless message when
odbcdoes not configure correctly.
- Remove unicode characters from document.
- Fix encoding issue in Rd.
- Fix internal issues. No user-level updates.
- Supported the new database design of IQUIZOO. Now the Rmarkdown templates were removed, and users should call
tarflow.iquizoo::use_targets()to generate the pipeline. - Limited the parameters settings to only
course_nameandcourse_periodfor now. Other parameters will be added in future. - Added
game_stageandgame_starto scores data (#29).
- Supported
I()to input literal query inquery_fileargument forpickup(). - Added project names and game version names into data query template.
- Fix some typos in documentation.
- Enhanced templates: now
content_orginal_data_detailandcontent_score_detailtables lived iniqizoo_content_dbdatabase.
- Added more tests. No user-level updates.
- Fixed a bug in
.onAttach()that will display incorrect message when loading the package.
- Fixed option
"tarflow.driver"not working. - Removed
...argument fromconnect_to_db()for it is just an internal function.
- Now
pickup()supportdrvargument to specify the database driver. This is useful when you want to specify which database driver to use. For example, you can usedrv = RMariaDB::MariaDB().- The default is from the value of option
"tarflow.driver", which will find the first available driver fromodbcandRMariaDB. If neither of them is available, a message will be prompted to inform user to install one.
- The default is from the value of option
- Added a new argument
add_keywordtocompose_where()to allow adding keywordWHEREto the where clause. This is useful when you want to compose a where clause for a subquery.
- Removed
search_games_mem()for it will not behave as expected. Especially,memoise::forget()will not work. A working version is added in rmarkdown template.
- Enhanced code quality.
- Fixed errors caused by
<integer64>class. Nowdata.iquizoo::game_infoand data returned bypickup()storesgame_idas<integer64>class from bit64 package, buttarchetypes::tar_map()does not support such class, and here we convert it to<character>class as a workaround.
- Fixed recoverable error message pattern to match more cases.
- Implemented new updates of data.iquizoo package, and now
inputandextrashould be configured ingame_infodata from that package. - Suggests {preproc.iquizoo} 2.4.0 or higher now, because we require the preprocessing functions support
.byagain.
- Supported setting custom variable name for parsed raw data.
- Fixed bug of
integer64type.
- Now
preproc_data()do not usepurrr::possibly()to suppress errors.
- Now parsed data are nested into the
data.frame()of data, which will be more efficient (#50). - Now
pickup()support input literal sql query string through argumentquery_file. To be recognized as literal sql query, the string must contains at least one new line. - Now raw data will also fetch
"game_version"column.
- Now
preproc_data()returnsindicesaftertidyr::pivot_longer(), so the column names are now"index_name"and"score", consistent among all games. - Now
wrangle_data()also changes character values to lower case. - A better logic dealing with data preprocessing after preproc.iquizoo functions accept
.inputand.extrainputs.
- Remove old content from github README.
init()and its related functions were totally removed, which were deprecated in"0.2.0"and later.fetch()andfetch_single_game()were renamed aspickup()andpickup_single_game()to avoid name masking ofDBI::fetch()(see #45).
- Fix a bug of
preproc_data()occurred when using complex dplyr verbs (see #43).
- Added more tests for many core functions except
pickup()(see #41).
- Adapt to new preproc.iquizoo and data.iquizoo packages.
- Fix a target name problem.
- Now
init()is deprecated, and rmarkdown template is recommended instead.
- Support setting up pipeline in rmarkdown using template (#35).
- Unified user identifier to
'OrganizationUserId'.
- Fixed an exception when fetching
usersfrom new version of database. Nowbase_grade_classwas obsolete and removed from query.
- Fix exception of encoding issue of SQL query by removing the recoding of user sex.
- Fix an exception caused by new database design of IQUIZOO.
- Set
known_onlytoFALSEwhen downloading pre-calculated scores (#32).
- Fix an exception when using key as one target (#33).
- Fix an issue of
TarScript()when there are only onecodes(thusunique()is not necessary to be called) to update.
- Add
known_onlyargument (default toTRUE) tosearch_games(), which uses the games indataproc.iquizoo::game_infoonly. You cannot set it asFALSEwhen fetching original data.
- Add
encodingargument (default to"UTF-8") tofetch(), which is used to set the encoding of SQL query files. This will also fix some encoding issues on Windows OS.
- Correct logic of
step_query(). The query namedgamesare not to be included in targets. And queries namedusersandabilitiesshould be always fetched. Other queries depend on whether it is separated or not.
- Supported games abilities.
- Adapt to new database design in IQUIZOO.
- Moved
wrangle_data()(#24) to dataproc.iquizoo package now. That is, the data preprocessing parts are all moved to dataproc.iquizoo package now. This better fits our mental models about these packages.
- Removed
calc_indices(#24) and addedwrangle_data()to fit the jobs for this package. This new function will parse inputjsonstring and stack them into long format, but keep other meta info in the attribute"info"of output. With all these, the output added a subclass"tbl_meta"and aprint()method.
- Added
fetch_single_game()to treat special case of fetching dataset from a single game. This is used especially when using branches. - Unexported
compose_where()andinsert_where(), because they are just for usage in current package.
- Use
stringr::str_detect()to remove empty json string. Now"[]"and"{}"are both removed.
- Fix issue of empty json string by removing empty (i.e.,
"[]") json string data in validation step ofcalc_indices().
- Fix issue of invalid json string by introducing a data validation step in
calc_indices().
- Fix an issue of data name case, and now all the names are translated to lower-case ones before preprocessing.
- New feature is introduced with a huge change that
use_targets()has been removed. Andinit()is used to replace it. With this function, we can define many other schemas. Maybe in future, the logic should be modified to better handle schemas.
- Fix the encoding issue on Windows system.
- Add new schema "original" to download original data only.
- Use a new yaml configurations api.
- Support more types of
config_whereinfetch_from_v3(). Besides thelisttype, now you can specify adata.frametype and even acharactertype ofconfig_where. This is most helpful whenconfig_whereis generated not byyamlconfig, but by R code directly. - Rename
fetch_from_v3()asfetch(), and the API is nowfetch(query_file, config_where, dsn). That is to say, you can now pass the data source name of your database to it. - Export S3 method
compose_where(), which originally was named ascompose_where_clause(). - Support new argument
separateinuse_targets()to optionally separate fetching into branches by games. - Support new argument
ignore_tarinuse_targets()to optionally ignore internal data from targets package in version control system (i.e., ".gitignore"). - Now
use_targets()will skip "config.yml" file silently if there already exists one. - Moved internal data named
game_infoto "dataproc.iquizoo (>= 0.2.6)" package.
- Added a
NEWS.mdfile to track changes to the package.