Errors when trying to install and use code

[fohring]

Hi,
I have been trying to install and use this code for a while now, without much success.
I already had astrometry.net successfully running on my Mac, and I wanted to try incorporating the Gaia index files.
I first ran into problems when trying to run the setup.py install script. It seems like there are already conflicting csv.py files in the python 3.7 anaconda3 installation that prevented installation. I solved this by renaming the csv.py file from this repo to gaia-csv.py and all the instances where it was called. I think I managed to successfully run create_gaia_index_files, but when I try to run solve-astrometry-gaia, I got the error
File "/Users/fohring/anaconda3/anaconda3/bin/solve-astrometry-gaia", line 11, in
load_entry_point('gaia-astrometry-net==0.0.1', 'console_scripts', 'solve-astrometry-gaia')()
File "/Users/fohring/anaconda3/anaconda3/lib/python3.7/site-packages/gaia_astrometry_net-0.0.1-py3.7.egg/gaia_astrometry_index_files/solve.py", line 60, in solve_frame
File "/Users/fohring/anaconda3/anaconda3/lib/python3.7/site-packages/astropy/io/fits/header.py", line 148, in getitem
card = self._cards[self._cardindex(key)]
File "/Users/fohring/anaconda3/anaconda3/lib/python3.7/site-packages/astropy/io/fits/header.py", line 1708, in _cardindex
raise KeyError(f"Keyword {keyword!r} not found.")
KeyError: "Keyword 'PIXSCALE' not found."
I was puzzled by this because I the same image had worked with the original astrometry.net code. I modified the code to have the pixel scale hard-coded, and rerunning I now get the error:
Could not produce source catalog
Do you have any idea what could be wrong? Thanks in advance!

[cmccully]

Sorry for the delay @fohring. I'm glad you got the index files to build. I have been thinking we should host those somehow. The solve command here is just a wrapper for astrometry.net with call parameters here:
https://github.com/LCOGT/gaia-astrometry-net/blob/master/gaia_astrometry_index_files/solve.py#L17

This assumes you have a pixel scale guess in the header as you surmised. Doing the full blind solve is not really tenable given the size of the gaia catalogs. So as an optimization we want you to give a guess (the guess doesn't have to be perfect) and we refine from there. We should really add that to the command line args.

That means something in the catalog.py threw an exception. The simplest way to see what it was is to remove that try except block in solve.py and run it again. We should also really do a better job of logging the exception rather than just aborting. If you could post the exception, I'm happy to try to advise you more.

[fohring]

Hi @cmccully , thanks for your help and sorry for the delay as well. I've been playing around with the installation and I got further, however I am still stuck. My problem earlier was actually that I needed to modify the line where the code was pointing to the LCO directory for the Gaia files. I downloaded the Gaia files from cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/ onto an external disk and changed the line everywhere to point to the location of these files. I am assuming these are the right files that the code needs to run on.
I was able to eventually run create_gaia_index_files and successfully created a gaia.db. However, it is only 103 GB in size. I am now having failures when I try running solve-astrometry-gaia. It looks like the gaia-catalog*.fits files all are empty (apart from the headers), even though there is information in the gaia.db files. I wonder if it is failing somewhere in get_sources_in_healpixel. I am now trying to recreate the gaia.db file again to see if perhaps there was some error when I first made it, or some error message that I overlooked. In the meantime, I will try to step through the index_file_creator.py with a debugger from the point of the existing gaia.db file, to see if I can find out more about why it's creating the almost empty fits files. I will let you know what I find.
Cheers,
Dora

[cmccully]

Hmm. Any luck on this? Let me know if there is more I can do to help.

[fohring]

Thanks for getting back to me about this.
I have been trying to recreate the gaia.db file, but each time it is only 110GB in size.
Upon inspecting the gaia.db file, the sources table is filled in, but the positions table is not. This is why I get sources_in_healpix = query_sources(ramin, ramax, decmin, decmax) producing {'flux': [], 'ra': [], 'dec': []}.
I am trying to see how the positions table is created in the dbs.py file, but that's as far as I got.