stixdcpy.net.JSONRequest silently fails for some queries

[settwi]

Sometimes fetching a flare list given a date range silently fails and returns the incorrect number of flares.

Consider the following code (with sort='LC2'):

from stixdcpy.net import JSONRequest as jsreq
flare_list = jsreq.fetch_flare_list('2020-01-01T00:00:00', '2023-01-07T00:00:00', sort='LC2')
print(len(flare_list))

Output:

5000

Now consider this code that loops one month at a time:

import datetime
start_time = datetime.datetime.fromisoformat('2021-12-31T00:00:00')
end_time = datetime.datetime.fromisoformat('2023-01-07T00:00:00')
dt = datetime.timedelta(weeks=4)

num_flares = 0
t = start_time
while t < end_time:
    print('query start', t)
    flare_list = jsreq.fetch_flare_list(
        begin_utc=t.isoformat(),
        end_utc=(t + dt).isoformat(),
        sort='LC0'
    )
    num_flares += len(flare_list)
    t += dt

print('number in list', num_flares)

Output:

query start 2021-12-31 00:00:00
query start 2022-01-28 00:00:00
query start 2022-02-25 00:00:00
query start 2022-03-25 00:00:00
query start 2022-04-22 00:00:00
query start 2022-05-20 00:00:00
query start 2022-06-17 00:00:00
query start 2022-07-15 00:00:00
query start 2022-08-12 00:00:00
query start 2022-09-09 00:00:00
query start 2022-10-07 00:00:00
query start 2022-11-04 00:00:00
query start 2022-12-02 00:00:00
query start 2022-12-30 00:00:00
number in list 16942

The output is the same with sort=None.

Now consider this code which prints an error (with sort='goes'):

flare_list = jsreq.fetch_flare_list('2020-01-01T00:00:00', '2023-01-07T00:00:00', sort='goes')

Output

Executor error during find command :: caused by ::
Sort operation used more than the maximum 33554432 bytes of RAM.
Add an index, or specify a smaller limit.,
full error:
{'ok': 0.0, 
'errmsg': 
'Executor error during find command :: caused by ::
Sort operation used more than the maximum 33554432 bytes of RAM. 
Add an index, or specify a smaller limit.', 
'code': 96,
'codeName': 'OperationFailed'}