Quickscan

[jreadey]

This change addresses issue #255 - verbose info not correct.

For the verbose param to return the current info for domain or datasets, you'll need to do a PUT / with rescan and flush params set. See the checkVerbose function in value_test.py to see how this works.

[mattjala]

If last_action is set to the current time before the scanning loop is performed/at the end of each scan loop, won't this always be true?

[jreadey]

If there are no root_ids, last_action doesn't get updated, so eventually the if case will get executed.

[mattjala]

What's the idea behind varying the time between scans based on how long the scan itself takes? Or is last_action tracking something else?

[jreadey]

If there's nothing going on, use a longer sleep and save cpu cycles. It's not how long the scan takes but rather the last time there was a domain to scan.

[jreadey]

Here's a quick check using the value_test.py test case.
I modified the testPut1DDataset to do a sequence of flushes and rescans:

    # check values we should get from a verbose query
    import time
    ts = time.time()
    for i in range(10):
        expected = {"num_chunks": 1, "allocated_size": 40}
        self.checkVerbose(dset_id, headers=headers, expected=expected)
        now = time.time()
        elapsed = now - ts
        print(f"flush {elapsed:.2f}")
        ts = now

Output is:

      $ python value_test.py ValueTest.testPut1DDataset
      testPut1DDataset /home/test_user1/hsds_test/valuetest/20231010T154859_160988Z
      flush 2.54
      flush 0.12
      flush 0.12
      flush 0.13
      flush 0.13
      flush 0.13
      flush 0.12
      flush 0.12
      flush 0.13
      flush 0.13

The first round takes a couple of seconds because apparently the bucketScan task is in long sleep mode. After the first flush, it "wakes up" and each update only takes about 0.1s.

[mattjala]

Why the change in automatic scan frequency? I thought the idea was to allow certain requests to force a scan, in which case making automatic scans this frequent would be unnecessary.

[jreadey]

The buckeScan task is still running independently, so the sleep is there to not unnecessarily consume cpu.
It might be better to use something like asyncio.Event (see https://docs.python.org/3/library/asyncio-sync.html#asyncio.Event).

[jreadey]

I'll leave using signals as a future enhancement since what we have should be good enough for this non-critical path.

[mattjala]

This enabled H5Dget_storage_size to work from the REST VOL, when it requests a flush and rescan first. I don't fully understand the reasoning behind all the scan changes, but we can iron those out later. Approved.