Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XMM-LSS master list #4

Open
yannick1974 opened this issue Jul 28, 2017 · 1 comment
Open

XMM-LSS master list #4

yannick1974 opened this issue Jul 28, 2017 · 1 comment

Comments

@yannick1974
Copy link
Member

No description provided.

@yannick1974
Copy link
Member Author

Commits relative to XMM-LSS masterlist before 2017/11/20:

  • 37aff91 Working on EGS. Halfway through pristine catalogues
  • 02688da HATLAS-SGP running on test square degree
  • b686052 XMM-LSS ready to run on Vulcain
  • 47c006c XMM-LSS merge hopefully ready to run on Vulcain.
  • b6bb8cf XMM-LSS merging: Added CFHT selection. Started HSC selection.
  • b210c6a XMM-LSS merge running
  • 3d4dca4 Produced starmask region and MOC files on Lockman-SWIRE.
  • db30a7e Close to finishing XMM-LSS merge notebook.
  • 58cda0b All XMM-LSS pristine catalogues running
  • 1a61f03 SXDS finished
  • cbd4d19 Pristine catalogues for XMM-LSS about 50 percent done.
  • 49c4304 First three XMM-LSS pristine catalogues completed
  • 06116b8 First three XMM-LSS pristine catalogues completed
  • f058d26 fixing table
  • b2dcb77 Starting on XMM-LSS. A lot of pristine catalogues to deal with
  • 6ccc541 Changes to ES1 based on Data planning updates
  • 812cb4c Created notebooks for every field with current field/survey spreadsheet
  • aa1c3d1 Add missing band-pass to validation reports
  • f1482a9 [dmu1_ml_GAMA-09, dmu1/dmu1_ml_GAMA-12, dmu1/dmu1_ml_XMM-LSS] Depth maps
  • 37ee4b9 DMU2 validation report for dmu1_ml_XMM-LSS
  • fd3cb8d Add GAMA fields and XMM-LSS master lists

raphaelshirley added a commit that referenced this issue Nov 21, 2017
@raphaelshirley
Low memory merging on XMM-LSS
1b1d41f 
On XMM-LSS #4 we have a memory issue. I am trying to perform the merge without the photometry and then add the photometry at the end. If this works we will probably have to have separate merging notebooks for the surveys with overlapping bands which include the choices i.e. merge uds and dxs in to a single UKIDSS table before the final merge. If the merge works with all columsn apart from those starting with f_, feer_, m_, or merr_ this method should work.
raphaelshirley added a commit that referenced this issue Nov 22, 2017
@raphaelshirley
HS82 running on test degree to merge
7f78616 
Perhaps it will run on Vulcain without changes now. 30Gb decals will reduce to 5 gb after processing and not as many surveys as xmm-lss. Yannick noticed major error in DECals - needs aperture correction and band names changing inline with Legacy survey on #2, #3, #4, #5, #7, #8, #15 and #16
raphaelshirley added a commit that referenced this issue Nov 23, 2017
@raphaelshirley
Fixing DECaLS across all fields
54be36b 
Merging and diagnostic notebooks may need changing to reflect the fact that some decam bands are removed from the masterlist because they are empty. #3 #4 #7 #8 #15 #16
yannick1974 added a commit that referenced this issue Nov 27, 2017
@yannick1974
[dmu1_ml_XMM-LSS] Try to save the file by millions
b396c58 
As the memory problem on XMM-LSS arises when we save the masterlist at
the end of the merging notebook, we try to save several files of one
million sources.

Relate to #4
raphaelshirley added a commit that referenced this issue Dec 1, 2017
@raphaelshirley
Memory test on xmm-lss
f71c153 
Perform cross match using only ra dec then join photometry in at end of notebook per 10000 objects #4
raphaelshirley added a commit that referenced this issue Dec 4, 2017
@raphaelshirley
Fix udeep to uDeep on xmm-lss #4
55da4e2
raphaelshirley added a commit that referenced this issue Dec 4, 2017
@raphaelshirley
bug fixes on xmm-lss merge test #4
33a0865
raphaelshirley added a commit that referenced this issue Dec 4, 2017
@raphaelshirley
I think XMM-LSS merge-low-memory may now run - ready to test on apollo …
6f2bb39 
…#4
raphaelshirley added a commit that referenced this issue Dec 5, 2017
@raphaelshirley
Added quick checks on all fields
a315c84 
Still a few fields miss the reporting of masterlist used #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, #19, #20, #21, #22, #23
raphaelshirley added a commit that referenced this issue Dec 5, 2017
@raphaelshirley
Brute force hack to make DECaLS run on laptop and Vulcain
50d677f 
We take psf with and without a space to define a star just to get the code running across both machines #2 #3 #4 #7 #8 #15 #16
raphaelshirley added a commit that referenced this issue Dec 6, 2017
@raphaelshirley
Fixed DECaLS id non-uniqueness on all fields
b68c130 
this means to get back to object we need to break decals_intid into brick plus objid or else add the identical unique decals_intid to the pristine catalogues #2 #3 #4 #5 #7 #8 #15 #16
yannick1974 added a commit that referenced this issue Dec 12, 2017
@yannick1974
[dmu1_ml_XMM-LSS] Try on vulcain
cff0bee 
- We tried to process the XMM-LSS master list on vulcain using an
  alternative saving method. But it failed.

- Relate to #4.
raphaelshirley added a commit that referenced this issue Dec 14, 2017
@raphaelshirley
Starting join based merge on XMM-LSS
7ee5a53 
#4 as well as some changes to starmask code
raphaelshirley added a commit that referenced this issue Dec 15, 2017
@raphaelshirley
Working on XMM-LSS
08bbc8e 
#4
raphaelshirley added a commit that referenced this issue Dec 20, 2017
@raphaelshirley
Working on XMM-LSS
63fabd1 
The code is a little convoluted - perhaps I can find a way to tidy it up #4
raphaelshirley added a commit that referenced this issue Jan 2, 2018
@raphaelshirley
Changes to XMM-LSS
3203241 
nearly ready to run on Vulcain. TODO: remove id columns and set correct number of objects per subcat and check output fits cat from test run. Also maybe correct masked values on sub cats. #4
raphaelshirley added a commit that referenced this issue Jan 9, 2018
@raphaelshirley
XMM-LSS ready to run on Vulcain
133a521 
Fingers crossed #4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
add dtype on boolean arrays
46460e3 
Notebooks ran on mac but not vulcain due to numpy defaults #4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
Adding more dtype=bool to XMM-LSS
c29fe22 
#4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
Removed specz_id name creation
852448e 
#4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
Remove unused id columns which interfere with join
985f82c 
#4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
bug fix in merge
6e102f7 
#4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
big fix missing column
47635e3 
#4
raphaelshirley added a commit that referenced this issue Jan 10, 2018
@raphaelshirley
Changed vista to vircam
9376713 
#4
raphaelshirley added a commit that referenced this issue Jan 11, 2018
@raphaelshirley
Checks on XMM-LSS ready for Vulcain
01a8b87 
#4
raphaelshirley added a commit that referenced this issue Jan 11, 2018
@raphaelshirley
XMM-LSS ran on Vulcain
987c958 
Checks needed this has the second biggest masterlist at 32 gb but checks still ran on it #4
raphaelshirley added a commit that referenced this issue Jan 15, 2018
@raphaelshirley
Added DES on all relevant fields
a1b13e6 
I havent added it to merge sections yet. I presume this will add a lot to some fields so I should add it to any fields not yet run. In fact I think it makes sense to add it to all fields but for fields already run through XID+ of photo-z etc it should not be the DR1 masterlist #4 #6 #11 #15 #17 #18 #20
raphaelshirley added a commit that referenced this issue Jan 17, 2018
@raphaelshirley
Working on DECAM merge on HS82
354d48e 
Also bug fixes on DES across southern fields #15 #4 #6 #17 #11 #18 #20
raphaelshirley added a commit that referenced this issue Jan 17, 2018
@raphaelshirley
Adding DES to merge on all southern fields
26ab937 
Still needs to be added to the checks. MAy have to rerun all DES stuff with new download as wavg fluxes may not be aperture fluxes. #15 #4 #17 #6 #11 #20 #18
raphaelshirley added a commit that referenced this issue Jan 19, 2018
@raphaelshirley
New DES on all DES fields
2f871f6 
Added aperture correction using new download #15 #4 #6 #17 #11 #20 #18
raphaelshirley added a commit that referenced this issue Jan 23, 2018
@raphaelshirley
Added DECam merge on xmm-lss
9eb48eb 
#4
raphaelshirley added a commit that referenced this issue Jan 26, 2018
@raphaelshirley
Replacing CANDELS-3D-HST with UDS
77e4f1b 
CANDELS 3D-HST seems to have an extra band f140w #4
raphaelshirley added a commit that referenced this issue Jan 29, 2018
@raphaelshirley
XMM-LSS ready to rerun on Vulcain
aca051e 
#4
raphaelshirley added a commit that referenced this issue Feb 1, 2018
@raphaelshirley
XMM-LSS merge run on Vulcain
89b02b4 
#4
raphaelshirley added a commit that referenced this issue Feb 2, 2018
@raphaelshirley
Column renaming
7d3f328 
#4
raphaelshirley added a commit that referenced this issue Feb 6, 2018
@raphaelshirley
XMM-LSS run on Vulcain
dc88174 
checks can run on whole catalogue #4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yannick1974