Hex Mirror Sync Progress Markers

[asummers]

The way the code is currently written, a sync is only marked as succeeded once it succeeds fully. Given that we progress package by package, we should be able to save checkpoint intermediate progress if the program crashes for some reason.

An alternative implementation that worked for me locally in such a scenario was:

deleted = sync_deleted_packages(mirror, config, diff)
mirror = update_in(mirror.registry, fn registry -> Map.drop(registry, deleted) end)
RegistryBackup.save(mirror)

chunk_size = 5

mirror =
  diff.releases
  |> Enum.chunk_every(chunk_size)
  |> Enum.reduce(mirror, fn packages, mirror ->
    updated = sync_releases(mirror, config, packages)
    mirror = update_in(mirror.registry, fn registry -> Map.merge(registry, updated) end)
    RegistryBackup.save(mirror)

    mirror
  end)

  mirror =
  diff.packages.created
  |> Enum.chunk_every(chunk_size)
  |> Enum.reduce(mirror, fn packages, mirror ->
    created = sync_created_packages(mirror, config, packages)
    mirror = update_in(mirror.registry, fn registry -> Map.merge(registry, created) end)
    RegistryBackup.save(mirror)

    mirror
  end)

and changing the sync functions to accept packages and to not dig into the diff map contents anymore. Would you be interested in such a change?

Additionally, in the delete part, the code in master is currently calling Map.delete/2 with a list instead of Map.drop/2 with the list of names to drop which is a bug.

That list is likely to be much smaller, but could be given the same treatment as create/update for consistency:

mirror =
  diff.packages.deleted
  |> Enum.chunk_every(chunk_size)
  |> Enum.reduce(mirror, fn packages, mirror ->
    deleted = sync_deleted_packages(mirror, config, packages)
    mirror = update_in(mirror.registry, fn registry -> Map.drop(registry, deleted) end)
    RegistryBackup.save(mirror)

    mirror
  end)