Skip to content

Commit

Permalink
1.0.0 draft before major version upgrade
Browse files Browse the repository at this point in the history
  • Loading branch information
@mcaceresb
mcaceresb committed Jul 21, 2018
1 parent 82e3b14 commit 32c294f
Show file tree
Hide file tree
Showing 45 changed files with 9,257 additions and 9,660 deletions.
19 changes: 8 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -403,18 +403,15 @@ TODO
----

`gtools` is feature-frozen. Other than bug fixes, no new features will
be added until version 1.0. All that remains is improving the coverage
of the debug checks before submitting to SSC.
be added until version 1.0.

Roadmap to 1.0

- [ ] Improve coverage of debug checks.
- [X] Improve coverage of debug checks.
- [X] Test `nunique` for gegen and gcollapse (vs `gunique`)
- [ ] Have corner cases for ALL commands
- [ ] Test all the options in every command
- [ ] Test errors (i.e. make sure commands fail as expected).
- [ ] Test weights in `gtop`.
- [ ] Test `gen` and `nolocal` in `glevelsof`.
- [X] Have corner cases for ALL commands
- [X] Test all the options in every command
- [X] Test errors (i.e. make sure commands fail as expected).
- [X] Test weights in `gtop`.
- [X] Test `gen` and `nolocal` in `glevelsof`.
- [X] Weights in `gquantiles`.
- [X] Add (partial) support for `strL` variables.
- [X] Add support for `by` in `gunique`
Expand All @@ -436,7 +433,7 @@ Roadmap to 1.0
- [X] Add weights in `gtop`.

These are options/features I would like to support, but I don't have an
ETA for them (and they almost surely won't make it to the 1.0 release).
ETA for them (and they won't make it to the 1.0 release).

- [ ] Add support for binary `strL` variables.
- [ ] Minimize memory use.
Expand Down
3 changes: 3 additions & 0 deletions build/_gtools_internal.ado
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2057,6 +2057,9 @@ program _gtools_internal, rclass
J(1, max(`=scalar(__gtools_xtile_nq2)', `nquantiles' - 1), 0)
local __gtools_xtile_nq_extra `__gtools_xtile_nq_extra' quantiles
}
else if ( (`=scalar(__gtools_xtile_nq2)' > 0) & ("`_pctile'" != "") ) {
* matsize for nq2 was already checked
}
else if ( "`_pctile'" != "" ) {
disp as txt "(option _pctile ignored)"
}
Expand Down
34 changes: 31 additions & 3 deletions build/changelog.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
Change Log
==========

## gtools-1.0.0 (2018-07-21)

First official release! This will go on SSC once all the tests have
passed.

------------------------------------------------------------------------

## gtools-0.14.2 (2018-07-21)

Feature freeze!
Expand All @@ -17,6 +24,10 @@ Feature freeze!
* Added `gen()` to the quick `glevelsof' showcase
* Added `gen()` to the `glevelsof' usage page and examples.

### Bug Fixes

* `gegen()` with `replace` sets missing values outside `if in` range.

## gtools-0.14.1 (2018-07-19)

### Features
Expand Down Expand Up @@ -49,6 +60,8 @@ Feature freeze!
existing variables gets a warning.
* Fixes https://github.com/mcaceresb/stata-gtools/issues/42 `gunique` typo.

------------------------------------------------------------------------

## gtools-0.13.3 (2018-05-06)

### Features
Expand Down Expand Up @@ -105,6 +118,8 @@ right-exclusive, would give the wrong quantile.
- `skew`, `kurt` give the correct answer when the group is a singleton
or when all observations are the same.

------------------------------------------------------------------------

## gtools-0.12.8 (2018-04-23)

### Features
Expand Down Expand Up @@ -192,6 +207,8 @@ requests them with weights, the weighted version will still be called
(OSX pending). gegen does not support weights with multiple
sources (e.g. `sum(x y z)`).

------------------------------------------------------------------------

## gtools-0.11.5 (2018-01-16)

### Features
Expand Down Expand Up @@ -279,6 +296,8 @@ requests them with weights, the weighted version will still be called
more characters than contained in the numfmt local macro. This
could cause problems on some systems.

------------------------------------------------------------------------

## gtools-0.10.3 (2017-11-12)

### Bug fixes
Expand Down Expand Up @@ -328,6 +347,8 @@ requests them with weights, the weighted version will still be called
- Fixed numerical precision issue with quantiles.
Now ((ST_double) N / 100) is computed first.

------------------------------------------------------------------------

## gtools-0.9.4 (2017-11-03)

### Bug fixes
Expand Down Expand Up @@ -397,6 +418,8 @@ requests them with weights, the weighted version will still be called
- Added quotes to `cd cwd` in `gtools`; fixes https://github.com/mcaceresb/stata-gtools/issues/22
- `gcontract` available; fixes https://github.com/mcaceresb/stata-gtools/issues/23

------------------------------------------------------------------------

## gtools-0.8.5 (2017-10-30)

### Features
Expand Down Expand Up @@ -476,6 +499,8 @@ requests them with weights, the weighted version will still be called
* Fixes https://github.com/mcaceresb/stata-gtools/issues/18
* Fixes https://github.com/mcaceresb/stata-gtools/issues/11

------------------------------------------------------------------------

## gtools-0.7.5 (2017-10-08)

### Features
Expand Down Expand Up @@ -545,6 +570,8 @@ requests them with weights, the weighted version will still be called

* `gcollapse, unsorted` no longer supported (due to internal sorting)

------------------------------------------------------------------------

## gtools-0.6.17 (2017-09-17); fixes #15

### Bug fixes
Expand Down Expand Up @@ -761,7 +788,7 @@ requests them with weights, the weighted version will still be called
the implementation this may be slower because adding variables takes
longer with more variables in memory.)

---
------------------------------------------------------------------------

## gtools-0.5.2 (2017-06-15)

Expand Down Expand Up @@ -870,7 +897,7 @@ requests them with weights, the weighted version will still be called
* If you sort the data in C, then assert the sort is unique and
print "(hashed correctly grouped observations: resulting sort is unique)"

---
------------------------------------------------------------------------

## gtools-0.4.1 (2017-05-29)

Expand Down Expand Up @@ -929,7 +956,8 @@ requests them with weights, the weighted version will still be called
print "(hashed correctly grouped observations: resulting sort is unique)"
* Be smart about memory management when J is small relative to N.

---
------------------------------------------------------------------------


## gtools-0.3.3 (2017-05-21)

Expand Down
11 changes: 6 additions & 5 deletions build/fasterxtile.ado
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,16 @@ program define fasterxtile
///
method(passthru) /// Quantile method: (1) qsort, (2) qselect
strict /// Exit if nquantiles > # non-missing obs
///
compress /// Try to compress strL variables
forcestrl /// Force reading strL variables (stata 14 and above only)
Verbose /// Print info during function execution
BENCHmark /// Benchmark function
BENCHmarklevel(passthru) /// Benchmark various steps of the plugin
HASHmethod(passthru) /// Hashing method: 1 (biject), 2 (spooky)
BENCHmark /// print function benchmark info
BENCHmarklevel(passthru) /// print plugin benchmark info
HASHmethod(passthru) /// Hashing method: 0 (default), 1 (biject), 2 (spooky)
hashlib(passthru) /// (Windows only) Custom path to spookyhash.dll
oncollision(passthru) /// error|fallback: On collision, use native command or throw error
///
forcestrl ///
compress ///
debug(passthru) ///
GROUPid(passthru) ///
tag(passthru) ///
Expand Down
51 changes: 36 additions & 15 deletions build/gcollapse.ado
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ program gcollapse, rclass
///
merge /// Merge statistics back to original data, replacing if applicable
replace /// Allow replacing existing variables with output with merge
compress /// Try to compress strL variables
forcestrl /// Force reading strL variables (stata 14 and above only)
freq(passthru) /// Include frequency count with observations per group
///
LABELFormat(passthru) /// Custom label engine: (#stat#) #sourcelabel# is the default
Expand All @@ -40,16 +38,18 @@ program gcollapse, rclass
forcemem /// Use memory for writing/reading collapsed data
double /// Generate all targets as doubles
///
compress /// Try to compress strL variables
forcestrl /// Force reading strL variables (stata 14 and above only)
Verbose /// Print info during function execution
BENCHmark /// print function benchmark info
BENCHmarklevel(int 0) /// print plugin benchmark info
///
HASHmethod(passthru) /// Hashing method: 0 (default), 1 (biject), 2 (spooky)
///
hashlib(passthru) /// (Windows only) Custom path to spookyhash.dll
oncollision(passthru) /// error|fallback: On collision, use native command or throw error
///
debug /// (internal) Allow replacing by variables with output
DEBUG_level(int 0) /// (internal) Allow replacing by variables with output
debug /// (internal) Debug
DEBUG_level(int 0) /// (internal) Debug (passed to internals)
debug_replaceby /// (internal) Allow replacing by variables with output
debug_io_read(int 1) /// (internal) Read IO data using mata or C
debug_io_check(real 1e6) /// (internal) Threshold to check for I/O speed gains
Expand Down Expand Up @@ -395,7 +395,8 @@ program gcollapse, rclass

qui mata: st_addvar(__gtools_gc_recasttypes, __gtools_gc_recastvars, 1)
if ( `=_N > 0' ) {
cap noi _gtools_internal, recast(targets(`gtools_recastvars') sources(`gtools_recastsrc'))
cap noi _gtools_internal, ///
recast(targets(`gtools_recastvars') sources(`gtools_recastsrc'))
if ( _rc ) {
local rc = _rc
CleanExit
Expand Down Expand Up @@ -481,13 +482,16 @@ program gcollapse, rclass
local sources sources(`__gtools_gc_vars')
local stats stats(`__gtools_gc_stats')
local targets targets(`__gtools_gc_targets')
local opts missing replace `keepmissing' `compress' `forcestrl' debug(`debug_level')
local opts `opts' `verbose' `benchmark' `benchmarklevel' `hashlib' `oncollision' `hashmethod'
local opts missing replace `keepmissing' `compress' `forcestrl'
local opts `opts' `verbose' `benchmark' `benchmarklevel' `hashmethod'
local opts `hashlib' `oncollision' debug(`debug_level')
local opts `opts' `anymissing' `allmissing' `rawstat'
local action `sources' `targets' `stats'

local switch = (`=scalar(__gtools_gc_k_extra)' > 3) & (`debug_io_check' < `=_N')
local mem = ("`forcemem'" != "") | ("`merge'" != "") | (`=scalar(__gtools_gc_k_extra)' == 0)
local mem = ("`forcemem'" != "") ///
| ("`merge'" != "") ///
| (`=scalar(__gtools_gc_k_extra)' == 0)
local io = ("`forceio'" != "") & (`=scalar(__gtools_gc_k_extra)' > 0)

if ( `io' ) {
Expand Down Expand Up @@ -518,7 +522,10 @@ program gcollapse, rclass
* Benchmark adding 2 variables to gauge how long it might take to
* add __gtools_gc_k_extra variables.
tempvar __gtools_gc_index __gtools_gc_ix __gtools_gc_info
cap noi benchmark_memvars, index(`__gtools_gc_index') ix(`__gtools_gc_ix') info(`__gtools_gc_info')
cap noi benchmark_memvars, ///
index(`__gtools_gc_index') ///
ix(`__gtools_gc_ix') ///
info(`__gtools_gc_info')
if ( _rc ) {
local rc = _rc
CleanExit
Expand Down Expand Up @@ -561,10 +568,14 @@ program gcollapse, rclass
gtools_timer info 97 `"`msg'"', prints(`bench')

if ( ("`forceio'" == "forceio") & (`=scalar(__gtools_gc_k_extra)' == 0) ) {
if ( `verb' ) di as text "(ignored -forceio- because sources are being used as targets)"
if ( `verb' ) {
di as text "(ignored -forceio- because sources are being used as targets)"
}
}

if ( "`added'" != "" ) qui mata: st_addvar(__gtools_gc_addtypes, __gtools_gc_addvars, 1)
if ( "`added'" != "" ) {
qui mata: st_addvar(__gtools_gc_addtypes, __gtools_gc_addvars, 1)
}
local msg "Generated additional targets"
gtools_timer info 97 `"`msg'"', prints(`bench')

Expand Down Expand Up @@ -672,7 +683,10 @@ program gcollapse, rclass
* If we collapsed to disk, read back the data
* -------------------------------------------

if ( (`=_N > 0') & (`=scalar(__gtools_gc_k_extra)' > 0) & ( `used_io' | ("`forceio'" == "forceio") ) ) {
local ifcond (`=_N > 0') ///
& (`=scalar(__gtools_gc_k_extra)' > 0) ///
& ( `used_io' | ("`forceio'" == "forceio") )
if ( `ifcond' ) {
gtools_timer on 97

qui mata: st_addvar(__gtools_gc_addtypes, __gtools_gc_addvars, 1)
Expand Down Expand Up @@ -805,7 +819,10 @@ end

cap mata: mata drop gtools_get_collapsed()
mata
real matrix function gtools_get_collapsed(string scalar fname, real scalar nrow, real scalar ncol)
real matrix function gtools_get_collapsed(
string scalar fname,
real scalar nrow,
real scalar ncol)
{
real scalar fh
real matrix X
Expand Down Expand Up @@ -1212,7 +1229,11 @@ program parse_keep_drop, rclass
else {
* We only recast integers. Floats and doubles are preserved unless
* requested or the target is a sum.
parse_ok_astarget, sourcevar(`var') targetvar(`var') stat(`collstat') `double' `weights'
parse_ok_astarget, ///
sourcevar(`var') ///
targetvar(`var') ///
stat(`collstat') ///
`double' `weights'
local recast = !(`r(ok_astarget)')

if ( `recast' ) {
Expand Down
Loading

0 comments on commit 32c294f

Please sign in to comment.