major refactor of select_mock_targets

[moustakas]

This PR (which would close #261) includes a near-total rewrite of select_mock_targets (which really just calls mock.build.targets_truth) and introduces target-specific classes (in mock.mockmaker) which make it much easier, transparent, and extendable to connect mock catalogs to spectra. There are still a variety of outstanding issues (see below), but I'd like to get the current (functional) code merged nowish and go from there.

I have tried to demonstrate some of the core functionality in this notebook, which I'm hopeful will be used by various Working Groups to build smaller spectroscopic datasets. Suggestions for additional tutorials or demonstrations of capabilities are welcome.

I tested this at NERSC with a single healpixel (using the fix-densities branch of desitarget):

salloc -N 1 -C haswell -t 00:60:00 --qos interactive -L SCRATCH,project
source /project/projectdirs/desi/software/desi_environment.sh master
package=desitarget
module unload $package
export PATH=/global/cscratch1/sd/ioannis/repos/$package/bin:${PATH}
export PYTHONPATH=/global/cscratch1/sd/ioannis/repos/$package/py:${PYTHONPATH}

then

cd /global/cscratch1/sd/ioannis/refactor
time select_mock_targets -c refactor.yaml --verbose --seed 123 \
  --healpixels 6507 --nside 32 --overwrite --nproc 32
join_mock_targets --mockdir . --force
run_target_qa ./targets.fits /project/projectdirs/cosmo/www/temp/ioannis/refactor \
  --nside 64 --mocks

which took about 7 minutes, with QA output here:
http://portal.nersc.gov/project/cosmo/temp/ioannis/refactor

Major improvements include:

• Incorporated a new (but still simplistic) target density fluctuations model (new density fluctuations model #254).
• Full utilization of healpixel organization; only reference to bricks is when generating the imaging-like targets catalog.
• Significantly better speed and memory usage. I don't have numbers handy but this should be noticeably faster and scale much better to, e.g., the 2% footprint.
• Better documentation. streamlined functionality.

In addition:

• Simplified configuration (input) yaml file.
• Addresses should obsconditions be in mock targets catalog or not? #253 by removing obsconditions
• Fixes dc17b: discrete input high z qso #260 - discrete QSO redshifts (maybe? I couldn't really reproduce this...)
• Fixes Encode mock provenance into TARGETID. #88 (mock provenance in targetid -- should be tested, though)
• Fixes fix duplicate TARGETIDs in select_mock_spectra #178 (duplicate targetid)
• Addresses Add r magnitudes into truth for BGS mocks #201 (include r-band magnitude in truth catalog)
• Fixes add units to target and truth files #237 (add units to target and truth catalogs)
• Fixes ensure that the QSO target densities aren't too low #192 (too-low QSO and Lya target densities)
• QSO magnitudes are drawn from the BOSS/DR9 luminosity function (via simqso).
• Outputs are functional with nominal desitarget QA (see Add/organize mock target QA #258 and ADM mock QA #262).

Known issues / problems / shortcomings:

• Still using the DR2-derived Gaussian mixture models to get shapes and (some) rest-frame colors (see update Gaussian mixture models? #250). @kaylanb has more recent fancy ones we might incorporate.
• Stellar spectra (including white dwarfs) are still not being interpolated based on the physical values of effective temperature, etc., in the Galaxia mocks (documented in MWS colors are oddly gridded in the dc17b truth catalogs #251 and being tracked in interpolate stellar templates desisim#315).
• Contaminants exist but their densities have not yet been tuned. This issue will require some thought and care.
• I totally and utterly broke the "mock catalogs without spectra" infrastructure used by quicksurvey (I'm sorry, @forero!) but I have a plan for putting it back together.
• Milky Way dust extinction is still not being handled correctly and consistently. In particular, the "truth" spectra are not being reddened, but they should be...
• Only the new-format (CoLoRe) Lya skewers are supported (handling targets in north / south overlap regions #295) but maybe that's OK. (Also note that the CoLoRe mocks are not yet full-resolution and they don't extend to z=1.6; see New format for Lya skewers  desisim#295).
• Although the input seed provides reproducibility, it's non-trivial to regenerate the spectra from the output truth catalogs, particularly for the QSOs. This is probably OK but medium-term we should probably implement something like what was proposed in update/change data model for template metadata? desisim#290.

Additional missing features:

• Does not yet support the buzzard mocks (but trivial to add) - support the new Buzzard mocks in select_mock_targets #249.
• Top-level verbose option does not yet propagate to mockmaker.
• Still no bright-star mask for mocks (basic bright object mask for mocks #103).
• No DLAs yet in the QSO spectra (but should be trivial to add).

[moustakas]

This is nearly ready to merge, although I have not yet tested at NERSC. On my laptop (with 4 cores) generating roughly 23k targets (of all types) in a single nside=32 healpixel took 14m42s, or roughly 65s/deg^2/cpu.

[moustakas]

OK, I think all conflicts have been fixed -- thanks for the off-list help, @geordie666!

@sbailey once your tests have completed (and hopefully Travis tests pass and the documentation builds successfully!) please feel free to merge, otherwise post any crashes or bugs here or off-list.

[sbailey]

yea, verily, let us be done with this PR and merge it (and face the consequences of our rash deeds)