Investigate number of incomplete science data requests

[samaloney]

Incomplete in this context means an incomplete packet sequence. A quick look seems to indicate it is a very low number approx 11 out of 19936 or 0.05%

# ipython magic
lb_files = !find . \( -name '*21-6-20*.fits' -o -name '*21-6-21*.fits' -o -name '*21-6-22*.fits' -o -name '*21-6-23*.fits' -o -name '*21-6-24*.fits' \)
lb_uids = []
for f in lb_files:
    *dirs, name = f.split('\\')
    parts = name.split('_')
    uid, tcref = parts[-1].split('-')
    lb_uids.append(uid)

# ipython magic
l0_files = !find . \( -name '*xray-rpd*.fits' -o -name '*xray-cpd*.fits' -o -name '*xray-scpd*.fits' -o -name '*xray-vis*.fits' -o -name '*xray-spec*'  \)
l0_uids = []
for f in l0_files:
    *dirs, name = f.split('\\')
    parts = name.split('_')
    uid, tcref = parts[-1].split('-')
    l0_uids.append(uid)


set(lb_uids).difference(set(l0_uids))

Result

{'0074661888',
 '0074661904',
 '2112090002',
 '2203151617',
 '2204028914',
 '2209279737',
 '2209280787',
 '2302120212',
 '2302257105',
 '2304131472',
 '2305104430'}

[samaloney]

Curiously extending to L1 we find one less, 2305104430, which I don't fully understand ...

l1_files = !find . \( -name '*xray-rpd*.fits' -o -name '*xray-cpd*.fits' -o -name '*xray-scpd*.fits' -o -name '*xray-vis*.fits' -o -name '*xray-spec*'  \)

l1_uids = []
for f in l1_files:
   *dirs, name = f.split('\\')
    parts = name.split('_')
    uid, tcref = parts[-1].split('-')
    l1_uids.append(uid)

set(lb_uids).difference(set(l1_uids))
{'0074661888',
 '0074661904',
 '2112090002',
 '2203151617',
 '2204028914',
 '2209279737',
 '2209280787',
 '2302120212',
 '2302257105',
 '2304131472'}

[samaloney]

So a further inspection it looks like some (4/10) of the request don't get to level L0 for other reasons the remaining (6/10) are really incomplete packets missing the closing packet.

Valid sets would be {3}, {1, 2}, {0, 1, 2}

for uid in lb_uids:
    file = !find * -name '*{uid}*'
    hdul = fits.open(file[0])
    print(file, set(hdul[1].data['sequence_flag']))

['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2302120212-60797.fits'] {0, 1}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2209280787-63265.fits'] {0, 1}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2302257105-61109.fits'] {0, 1}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2203151617-62310.fits'] {0, 1, 2}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_0074661904-00000.fits'] {0, 1}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2304131472-49403.fits'] {0, 1}
['6/24/solo_LB_stix-21-6-24_0000000000-9999999999_V01_2209279737-62645.fits'] {0, 1, 2}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2112090002-51143.fits'] {0, 1, 2}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2305104430-51218.fits'] {0, 1, 2}
['6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2204028914-63370.fits'] {0, 1}
['6/20/solo_LB_stix-21-6-20_0000000000-9999999999_V01_0074661888-00000.fits'] {0, 1}

[samaloney]

As a result of above I realised I was combining incomplete sequences and problems processing from LB to L0 so to focus on incomplete sequences I ran this scrip.

Flag Value	Flag Name	Description
0	Continuation	This packet belongs to a longer sequence of TM packets.
1	First	This is the first packet of a longer TM sequence. Must be followed by either a continuation or last sequence flag.
2	Last	This is the last packet of a longer TM sequence. Must have been preceded by a continuation or fist sequence flag.
3	Stand-alone	This packet stands for itself.

for file in lb_files:
    hdul = fits.open(file)
    seq_flag = hdul[1].data['sequence_flag']
    seq_flag_set = set(seq_flag)
    if len(seq_flag) == 1 and seq_flag[0] == 3:  # standalone
        pass
    elif len(seq_flag) == 2 and seq_flag_set == {1, 2}:  # seq with no cont.
        pass
    elif len(seq_flag) > 2 and seq_flag_set == {0, 1, 2}:  # seq with cont.
        pass
    else:
        print(file, seq_flag_set)  # incomplete

./6/24/solo_LB_stix-21-6-24_0000000000-9999999999_V01_2201270032-59177.fits {0, 2}
./6/24/solo_LB_stix-21-6-24_0000000000-9999999999_V01_2201260047-59172.fits {0, 2}
./6/24/solo_LB_stix-21-6-24_0000000000-9999999999_V01_2201270033-59178.fits {0, 2}
./6/20/solo_LB_stix-21-6-20_0000000000-9999999999_V01_0074661888-00000.fits {0, 1}
./6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2204028914-63370.fits {0, 1}
./6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2209280787-63265.fits {0, 1}
./6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_0074661904-00000.fits {0, 1}
./6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2302120212-60797.fits {0, 1}
./6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2302257105-61109.fits {0, 1}
./6/21/solo_LB_stix-21-6-21_0000000000-9999999999_V01_2304131472-49403.fits {0, 1}

Complete requests missing L0 files

incomplete = {'0074661888', '0074661904', '2201260047', '2201270032', '2201270033', '2204028914', '2209280787', '2302120212', '2302257105', '2304131472'}
missing_l0 = set(lb_uids).difference(set(l1_uids))
missing_l0.difference(incomplete)
{'2112090002', '2203151617', '2209279737'}

[FredSchuller]

Is there any chance to recover these data at some point?
For example, I was recently looking for the file with UID 2302120212 and I could not find the L1 file, although the L1A exists (but the content seems to be incomplete, according to the online quick preview)