Skip to content

Commit

Permalink
Special case ingest to use "replace" for record insertion in some sit…
Browse files Browse the repository at this point in the history
…uations

Until we have a coherent way of handling record clashes, for ingest
assume that if UUIDv5 IDs are being used and the file is not
going to be transferred into the datastore, then replacing
existing records is allowable.
  • Loading branch information
timj committed Aug 16, 2023
1 parent 1714233 commit 6afc9ce
Showing 1 changed file with 11 additions and 2 deletions.
13 changes: 11 additions & 2 deletions python/lsst/daf/butler/datastores/fileDatastore.py
Original file line number Diff line number Diff line change
Expand Up @@ -1034,6 +1034,7 @@ def _finishIngest(
record_validation_info: bool = True,
) -> None:
# Docstring inherited from Datastore._finishIngest.
uses_uuid_v5 = True
refsAndInfos = []
progress = Progress("lsst.daf.butler.datastores.FileDatastore.ingest", level=logging.DEBUG)
for dataset in progress.wrap(prepData.datasets, desc="Ingesting dataset files"):
Expand All @@ -1046,8 +1047,16 @@ def _finishIngest(
record_validation_info=record_validation_info,
)
refsAndInfos.extend([(ref, info) for ref in dataset.refs])

self._register_datasets(refsAndInfos, insert_mode="insert")
for ref in dataset.refs:
if ref.id.version != 5:
uses_uuid_v5 = False

insert_mode = "insert"
if uses_uuid_v5 and transfer == "direct":
# Datasets are immutable, external and use well-defined UUID.
# Re-ingest is allowed (use most recent information).
insert_mode = "replace"

Check warning on line 1058 in python/lsst/daf/butler/datastores/fileDatastore.py

View check run for this annotation

Codecov / codecov/patch

python/lsst/daf/butler/datastores/fileDatastore.py#L1058

Added line #L1058 was not covered by tests
self._register_datasets(refsAndInfos, insert_mode=insert_mode)

def _calculate_ingested_datastore_name(
self,
Expand Down

0 comments on commit 6afc9ce

Please sign in to comment.