Skip to content

DeltaRepo

Jump to bottom Edit New page
Tomas Mlcoch edited this page Feb 5, 2014 · 52 revisions

Set of tools that generate/merges differences between an old and a new version of a repodata.

Idea

  • Repodata could be pretty big (dozens or hundreds of megabytes).
  • Changes between two versions of repodata could be very small (a single deleted package).
  • Let's make a tool that could detect changes between two repodata, generate its delta (diff) + tool that could apply this delta on the old repodata.

Design ideas

  • Maximal compatibility.
  • Avoid significant changes in the repomd.xml or other repodata files.
  • Repodelta delta itself is a repository.
  • Plugins - Generic DeltaRepo can create and apply delta files for primary, filelists and other. Delta from other repodata files (groupfile, prestodelta, ..) could be done via plugins.

DeltaRepo

File structure of a single delta repo looks like:

repodata/
  |-primary.xml.gz
  |-filelists.xml.gz
  |-other.xml.gz
  |-deltametadata.xml.xz
  |-repomd.xml

Where primary, filelists and other are in classical format but their content is composed only from changed or added packages.

deltametadata.xml will be described bellow.

deltametata.xml.gz

This file contains records for each used plugin. It is considered as a persistent storage for delta plugins, which could store here an important configuration and data which are necessary for further application.

Example:

<?xml version='1.0' encoding='UTF-8'?>
<deltametadata>
  <revision src="123" dst="456"> <!-- Revision -->
  <contenthash src="abc" dst="bcd"> <!-- Content hash -->
  <timestamp src="120" dst="450"> <!-- The highest timestamp from metadata -->
  <usedplugins>
    <plugin name="MainDeltaPlugin" version="1" src_contenthash="abc" dst_contenthash="bcd" contenthash_type="sha256">
      <removedpackage location_href="../packages/fake_bash-1.1.1-1.x86_64.rpm"/>
      <metadata database="1" type="filelists"/>
      <metadata database="1" original="1" type="other"/>
      <metadata database="1" type="primary"/>
    </plugin>
  </usedplugins>
</deltametadata>

Each <plugin> element should have name and version attribute. Other attributes can added by plugin with no limitation. What a plugin stores in the deltametadata file depends on its needs. There is no prescription about what has to be stored by the plugin.

Plugins must not read/write/modify the deltametadata file on its own! This file is parsed by deltarepo and accessed via the PluginBundle object.

Description of the above mentioned example:

As you can see, a plugin with name "MainDeltaPlugin" (the core plugin for metadata of type: primary, primary_db, filelists, filelists_db, other and other_db) stores there a list of removed packages and list of notes about each metadata file which was processed.

Each metadata element contains information like:

  • Type of metadata on which this record is applied.
  • Should be database generated (database attribute).
  • Is the file in the delta repository a real delta or just copy of the original file (original attribute)

TODO: Relax NG schema (?)

repomd.xml

Classical repomd.xml, but with contenthash (Examined bellow).

Contenthash of deltarepo

Contenthash is case of Deltarepo is a string:

contenthash_of_old_repo-contenthash_of_new_repo

Eg: 5a8e6bbb940b151103b3970a26e32b8965da9e90a798b1b80ee4325308149d8d-b8d60e74c38b94f255c08c3fe5e10c166dcb52f2c4bfec6cae097a68fdd75e74

Contenthash of a repo

Contenthash is an identifier based on content of repo's primary.xml.

It is a hash calculated from all packages listed in primary.xml.

Calculation algorithm:

pkgids = []
for pkg in repo:
  # Assume that pkg.pkgId and pkg.location_href are never empty (has a None value)
  pkgids.append("%s%s%s", pkg.pkgId, pkg.location_href, pkg.location_base or '')
contenthash = hashlib.new("sha256")
for pkgid in pkgids.sort():
  contenthash.update(pkgid)
return contenthash.hexdigest()

"Extra" repodata files

"Extra" repodata files are metadata files for which a sophisticated delta function is not currently provided.

DeltaRepo currently could do a sophisticated deltas of base metadata files like:

  • primary.xml
  • filelists.xml
  • other.xml
  • primary.sqlite
  • filelists.sqlite
  • other.sqlite
  • repomd.xml

Example of extra repodata files:

  • comps.xml
  • deltainfo
  • pkgorigins
  • prestodelta.xml
  • ...

For this "unsupported" files the algorithm is used:

  • If a file in the new repo is the same as the one in the old repo, just make a note to the deltametadata.xml that the old file should be used during delta application and don't include the file in to the deltarepo.
  • If a file in the new repo is different than the one in the old repo. Include copy of the new file to the deltarepo. (If file is not compressed, compress it first to save some space).

Integration to the current repodata

Tree structure:

mirror/
  +-deltarepos
  |   +-ei7as764ly-043fds4red
  |   |   +-repodata
  |   |      |-primary.xml
  |   |      |-filelists.xml
  |   |      |-other.xml
  |   |      |-removed.xml
  |   |      |-repomd.xml
  |   |
  |   +-0w78as1r9r-043fds4red
  |   |   +-repodata
  |   |      |- ...
  |   |
  |   |-deltarepos.xml.xz
  |
  +-Packages
  |   |- ...
  |
  +-repodata
      |-primary.sqlite.bz2
      |-primary.xml.gz
      |-filelists.sqlite.bz2
      |-filelists.xml.gz
      |-other.sqlite.bz2
      |-other.xml.gz
      |-repomd.xml

Notes:

  • deltarepos.xml
  • Compressed by xz because of its great compression ratio.
  • Present in repodata/ directory in order not to break "verification chain" of repodatas. (checksum of repomd.xml is available from metalink, checksum of deltarepos.xml is available from the repomd.xml, checksum of each repomd.xml of available deltarepos is available from the deltarepos.xml)

deltarepos.xml

<?xml version="1.0" encoding="UTF-8"?>
<deltarepos>
  <deltarepo>
    <location href="deltarepos/ei7as764ly-043fds4red" />
    <revision src="1387077214" dst="1387087288" />
    <size>15432</size>
    <highest-timestamp src="1387075214" dst="1387086288" />
    <plugin name="MainDeltaPlugin" version="1" contenthash_type="sha256" src_contenthash="ei7as764ly" dst_contenthash="043fds4red">
  </deltarepo>
  <deltarepo>
    <location href="deltarepos/0w78as1r9r-043fds4red" />
    <revision src="1387077745" dst="1387087288" />
    <size>7869</size>
    <highest-timestamp src="1387075417" dst="1387086288" />
    <plugin name="MainDeltaPlugin" version="1" contenthash_type="sha256" src_contenthash="0w78as1r9r" dst_contenthash="043fds4red">
  </deltarepo>
</deltarepos>

Needed changes in the current state of repodata

repomd.xml

We need to store contenthash.

Add new element "contenthash"

<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1355393568</revision>
  <contenthash type="sha256">5a8e6bbb940b151103b3970a26e32b8965da9e90a798b1b80ee4325308149d8d</contenthash>
  <data type="primary">
    ....

Use cases

./deltarepo repo1 repo2

  • Create dir which has a contenthash as name and contain repository delta information.

./deltarepo --apply repo1 delta

  • Applies delta to the repository

Options:

$ ./deltarepo.py --help
usage: deltarepo.py [options] <first_repo> <second_repo>
       deltarepo.py --apply <repo> <delta_repo>

Gen/Apply delta on yum repository.

positional arguments:
  path1                 First repository
  path2                 Second repository or delta repository

optional arguments:
  -h, --help            show this help message and exit
  --version             Show version number and quit.
  -q, --quiet           Run in quiet mode.
  -v, --verbose         Run in verbose mode.
  -o DIR, --outputdir DIR
                        Output directory.
  -d, --database        Force database generation
  --ignore-missing      Ignore missing metadata files. (The files that are
                        listed in repomd.xml but physically doesn't exists)

Delta generation:
  -t HASHTYPE, --id-type HASHTYPE
                        Hash function for the ids (contenthash).
                        Default is sha256.

Delta application:
  -a, --apply           Enable delta application mode.
Add a custom footer

Wiki

External links

Clone this wiki locally