Skip to content

DeltaRepo

Jump to bottom Edit New page
Tomas Mlcoch edited this page Dec 3, 2013 · 52 revisions

Set of tools that generate/merges differences between an old and a new version of a repodata.

Idea

  • Repodata could be pretty big (dozens or hundreds of megabytes).
  • Changes between two versions of repodata could be very small (a single deleted package).
  • Let's make a tool that could detect changes between two repodata, generate its delta (diff) + tool that could apply this delta on the old repodata.

Design ideas

  • Maximal compatibility.
  • Avoid significant changes in the repomd.xml or other repodata files.
  • Repodelta delta itself is a repository.
  • Plugins - Generic DeltaRepo can create and apply delta files for primary, filelists and other. Delta from other repodata files (groupfile, prestodelta, ..) could be done via plugins.

DeltaRepo

File structure of a single delta repo looks like:

repodata/
  |-primary.xml.gz
  |-filelists.xml.gz
  |-other.xml.gz
  |-deltametadata.xml.xz
  |-repomd.xml

Where primary, filelists and other are in classical format but their content is composed only from changed or added packages.

deltametadata.xml will be described bellow.

deltametata.xml.gz

This file contains records for each used plugin. It is considered as a persistent storage for delta plugins, which could store there important configuration and data which are necessary for further application.

Example:

<?xml version='1.0' encoding='UTF-8'?>
<deltametadata>
  <usedplugins>
    <plugin name="MainDeltaPlugin" version="1">
      <removedpackage location_href="../packages/fake_bash-1.1.1-1.x86_64.rpm"/>
      <metadata database="1" type="filelists"/>
      <metadata database="1" original="1" type="other"/>
      <metadata database="1" type="primary"/>
    </plugin>
  </usedplugins>
</deltametadata>

What a plugin stores in the deltametadata file depends on its needs! There is no prescription about what has to be stored by the plugin!

Plugins cannot read/write/modify the deltametadata file on its own! This file is parsed by deltarepo and accessed via the PluginBundle object.

Description of the above mentioned example:

As you can see, a plugin with name "MainDeltaPlugin" (the core plugin for metadata of type: primary, primary_db, filelists, filelists_db, other and other_db) stores there a list of removed packages and list of notes about each metadata file which was processed.

Each metadata element contains information like:

  • Type of metadata on which this record is applied.
  • Should be database generated (database attribute).
  • Is the file in the delta repository a real delta or just copy of the original file (original attribute)

TODO: Relax NG schema (?)

repomd.xml

Classical repomd.xml, but with DeltaRepoId (Where to store RepoId/DeltaRepoId in repomd is examined bellow).

DeltaRepoId

DeltaRepoId is an identifier of a DeltaRepo.

repoid_of_old_repo-repoid_of_new_repo

Eg: 5a8e6bbb940b151103b3970a26e32b8965da9e90a798b1b80ee4325308149d8d-b8d60e74c38b94f255c08c3fe5e10c166dcb52f2c4bfec6cae097a68fdd75e74

RepoId

RepoId is an identifier of a every non-delta repository.

It is a SHA256 hash calculated from all packages in repository.

Calculation algorithm:

pkgids = []
for pkg in repo:
  # Assume that pkg.pkgId and pkg.location_href are never empty (has a None value)
  pkgids.append("%s%s%s", pkg.pkgId, pkg.location_href, pkg.location_base or '')
repoid = hashlib.new("sha256")
for pkgid in pkgids.sort():
  repoid.update(pkgid)
return repoid.hexdigest()

Extra repodata files

Extra repodata files are files except of:

  • primary.xml
  • filelists.xml
  • other.xml
  • primary.sqlite
  • filelists.sqlite
  • other.sqlite
  • repomd.xml

Example of extra repodata files:

  • comps.xml
  • deltainfo
  • pkgorigins
  • prestodelta.xml
  • ...

For this files no delta is generated!

If extra file were removed, it will be listed in removed.xml.

If extra file were added, it will be included in DeltaRepo.

If extra file were changed, it will be listed in removed.xml and new version included in DeltaRepo.

TODO: In future, maybe use diff tool and only include diff, insted of full file.

Integration to the current repodata

Tree structure:

mirror/
  +-deltarepos
  |   +-ei7as764ly-043fds4red
  |   |   |-primary.xml
  |   |   |-filelists.xml
  |   |   |-other.xml
  |   |   |-removed.xml
  |   |   |-repomd.xml
  |   |
  |   +-0w78as1r9r-043fds4red
  |   |   |- ...
  |   |
  |   |-deltarepo.xml
  |
  +-Packages
  |   |- ...
  |
  +-repodata
      |-primary.sqlite.bz2
      |-primary.xml.gz
      |-filelists.sqlite.bz2
      |-filelists.xml.gz
      |-other.sqlite.bz2
      |-other.xml.gz
      |-repomd.xml

Notes:

  • deltarepos/ dir is outside of repodata/ intentionally. (Main reason is to prevent downloading of unwanted deltas. Eg. when someone is using wget do recursively download current repodata.)

deltarepo.xml

<?xml version="1.0" encoding="UTF-8"?>
<deltarepos>
  <deltarepo id-type="sha256" from="ei7as764ly" to="043fds4red">
    <location href="deltarepos/ei7as764ly-043fds4red" />
    <size>15432</size>
  </deltarepo>
  <deltarepo id-type="sha256" from="0w78as1r9r" to="043fds4red">
    <location href="deltarepos/0w78as1r9r-043fds4red" />
    <size>7869</size>
  </deltarepo>
</deltarepos>

Needed changes in the current state of repodata

repomd.xml

We need to store current RepoId/DeltaRepoId.

Add new element "repoid"

<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1355393568</revision>
  <repoid type="sha256">5a8e6bbb940b151103b3970a26e32b8965da9e90a798b1b80ee4325308149d8d</repoid>
  <data type="primary">
    ....

Use cases

./deltarepo repo1 repo2

  • Create dir which has a DeltaRepoId as name and contain repository delta information.

./deltarepo --apply delta repo

  • Applies delta to the repository

Options:

  • -h, --help - Show help.
  • --version - Show version
  • -q, --quiet - Quiet mode
  • -v, --verbose - Verbose mode
  • -l, --list-datatypes - List datatypes for which delta is supported.
  • -o, --outputdir - Output dir
  • Options for delta generation (default):
  • -s, --skip=DATATYPE - Don't do delta of the datatype. Could be specified multiple times. (E.g., --skip=comps)
  • -d --do-only=DATATYPE - Do delta only of the listed datatypes. Could be specified multiple times. (E.g., --do-only=primary)
  • -t, --id-type=HASHTYPE - Hash function for the ids (RepoId and DeltaRepoId). Default is sha256.
  • Options for delta applying:
  • -a, --apply - Enable delta application mode.
  • -d, --database - Gen database.
Add a custom footer

Wiki

External links

Clone this wiki locally