push new transients to YSE_PZ after they're promoted to "good" or "possible"

[djones1040]

If it's possible to write a module that sends transients to YSE-PZ via our API once they are promoted to "good" or "possible" it would remove a 1-2 hour lag in our ability to collect metadata/ZTF observations and request followup observations through YSE-PZ. This could also make it easier for the sniffing team to comment on new/interesting things and streamline our vetting/triage process. I'm happy to write a module for this with a little guidance on how the formatting would work.

[genghisken]

This looks very similar to what we already do for TNS. (Anything promoted to the good list automatically gets sent immediately to the TNS.) Shouldn't be too difficult. I just need to know the API details.

[djones1040]

thanks Ken, I'll write a short module for this and paste here once it's finished. Can do it later this week.

[djones1040]

Hi Ken, took 20 days apparently but I finally wrote a code to do this - see the very bottom for my test example, that's the only info your code will need to include to send things directly to YSE_PZ once they're promoted. syntax is a bit annoying but this should work! If you don't have YSE_PZ credentials for variables _dblogin and _dbpassword, I can send you mine in a private message. and I was hoping the "name" variable would either be the TNS name for things that have just been promoted, or the 10DYSEaaa-type name for "possible" candidates.

Any questions about the code or things I could improve?

#!/usr/bin/env python
import requests
import json
from requests.auth import HTTPBasicAuth
from astropy.coordinates import SkyCoord
import astropy.units as u
import numpy as np
import astropy.table as at
from astropy.time import Time
from dustmaps.sfd import SFDQuery
sfd = SFDQuery()

def date_to_mjd(obs_date):
	time = Time(obs_date,scale='utc')
	return time.mjd

def mjd_to_date(obs_mjd):
	time = Time(obs_mjd,scale='utc',format='mjd')
	return time.isot

def fluxToMicroJansky(adu, exptime, zp):
    factor = 10**(-0.4*(zp-23.9))
    uJy = adu/exptime*factor
    return uJy

def send_to_yse_pz(candid,name,ra,dec,rb_factor_image,sherlock_specz,
				   sherlock_object_id,sherlock_host_ra,sherlock_host_dec,
				   followup_flag_date,sherlockClassification):

	sc = SkyCoord(ra,dec,unit=u.deg)
	ebv = sfd(sc)*0.86

	if type(sherlock_specz) == np.ma.core.MaskedConstant:
		redshift = None
	else:
		redshift = sherlock_specz

	if type(sherlock_object_id) == np.ma.core.MaskedConstant:
		hostdict = {}
	else:
		hostdict = {'name':sherlock_object_id[:64],
					'ra':sherlock_host_ra,
					'dec':sherlock_host_dec,
					'redshift':redshift}

	if type(rb_factor_image) == np.ma.core.MaskedConstant:
		rb_factor = None
	else:
		rb_factor = rb_factor_image
		
	TransientDict = {'name':name,
					 'slug':name,
					 'ra':ra, # degrees
					 'dec':dec, # degrees
					 'obs_group':'YSE',
					 'mw_ebv':ebv,
					 'status':'New',
					 'host':hostdict,
					 'tags':['YSE'],
					 'context_class':sherlockClassification.replace('UNCLEAR','Unknown'),
					 'real_bogus_score':rb_factor,
					 'disc_date':followup_flag_date}

	r = requests.get(url='https://star.pst.qub.ac.uk/sne/ps1yse/psdb/lightcurveforced/%s'%candid)
	if r.status_code != 200: raise RuntimeError('problem accessing lc link %s'%self.options.yselink_summary)
	try:
		lc_forced = at.Table.read(r.text, format='ascii', delimiter=' ')
		if len(lc_forced): has_forced_phot = True
		else: has_forced_phot = False
	except:
		has_forced_phot = False
	
	PhotUploadAll = {"mjdmatchmin":0.01,
					 "clobber":False}
	photometrydict = {'instrument':'GPC1',
					  'obs_group':'YSE',
					  'photdata':{}}
	if has_forced_phot:
		for j,lf in enumerate(lc_forced[np.argsort(lc_forced['mjd'])]):
			if j == 0 and np.abs(date_to_mjd(followup_flag_date)-lf['mjd']) < 1: disc_point = 1
			else: disc_point = 0

			if lf['cal_psf_mag'] == 'None':
				forced_mag,forced_mag_err = None,None
			else:
				forced_mag,forced_mag_err = lf['cal_psf_mag'],lf['psf_inst_mag_sig']

			# forced photometry
			phot_upload_dict = {'obs_date':mjd_to_date(lf['mjd']),
								'band':lf['filter'],
								'groups':['YSE'],
								'mag':forced_mag,
								'mag_err':forced_mag_err,
								'flux':fluxToMicroJansky(lf['psf_inst_flux'],27.0,lf['zero_pt'])*10**(0.4*(27.5-23.9)),
								'flux_err':fluxToMicroJansky(lf['psf_inst_flux_sig'],27.0,lf['zero_pt'])*10**(0.4*(27.5-23.9)),
								'data_quality':0,
								'forced':1,
								'flux_zero_point':27.5,
								'discovery_point':disc_point,
								'diffim':1}
			photometrydict['photdata']['%s_%i'%(mjd_to_date(lf['mjd']),j)] = phot_upload_dict

	TransientDictAll = {name:TransientDict}
	PhotUploadAll['PS1'] = photometrydict
	TransientDictAll[name]['transientphotometry'] = PhotUploadAll
	UploadTransients(TransientDictAll,_dblogin,_dbpassword,dburl='https://ziggy.ucolick.org/yse')
	#dburl='http://127.0.0.1:8000')
	
def UploadTransients(TransientUploadDict,dblogin,dbpassword,dburl='https://ziggy.ucolick.org/yse'):
	url = f'{dburl}/add_transient/'
	try:
		r = requests.post(url = url, data = json.dumps(TransientUploadDict),
						  auth=HTTPBasicAuth(dblogin,dbpassword))

		try: print('YSE_PZ says: %s'%json.loads(r.text)['message'])
		except: print(r.text)
	except Exception as e: raise RuntimeError(e)
	print("Process done.")

if __name__ == "__main__":
	candid,name,ra,dec = '1160220371271425900','2020qhn',240.58495,27.24052
	rb_factor_image = 0.89
	sherlock_specz = None
	sherlock_object_id,sherlock_host_ra,sherlock_host_dec = 'SDSS J160220.45+271426.4',240.585223331, 27.240680682
	followup_flag_date = '2020-07-18T00:00:00'
	sherlockClassification = 'SN'
	send_to_yse_pz(candid,name,ra,dec,rb_factor_image,sherlock_specz,
				   sherlock_object_id,sherlock_host_ra,sherlock_host_dec,
				   followup_flag_date,sherlockClassification)

[genghisken]

OK... Interesting... About time we incorporated the dustmap code!

So... What I'll do is wrap this in some daemon code. The daemon can sit and listen to port N (to be defined) and fire off this request to yze_pz. All it needs is the object ID - I can pass that to the daemon from the webserver code, then collect all the other pertinent info from the database. I have a horrible mixture of python 2 and 3 code running everywhere. All the web stuff is still in python 2. (I'm in the long process of doing conversions, but I can't roll the code out until it's all converted.) The daemon allows me to run whatever python environment I want on a separate server and all it needs to do is listen for an object ID.

Apologies for very slow progress on many fronts - too many things going on (ATLAS fixes, LSST, reviews, substantial machine migrations), and I was also sick last week, so also playing catchup...

[djones1040]

thanks Ken, nice that I can submit some code for once instead of just asking for stuff and not providing any help!

[djones1040]

should add that there's almost certainly a better way to do the forced photometry once this code is on the server side, but if pinging the URL works for you then it's fine with me