dprms

Introduction

DeltaRPM

DeltaRPM is a tool that generates RPMs that contains the difference between an old and a new version of an RPM. This makes it possible to recreate the new RPM from the deltarpm and the old one. You don't have to have a copy of the old RPM, as it can also work with installed RPMs. The package also contains tools for creating and applying delta ISOs.

Upstream: ftp://ftp.suse.com/pub/projects/deltarpm/ (repo: https://gitorious.org/deltarpm/ (?))

DeltaRPM and createrepo

Original createrepo supports DeltaRPM (.drpm) generation, createrepo_c not yet.

Related options

• --deltas
• Default value: False
• --oldpackagedirs=OLDPACKAGE_PATHS paths to look for older pkgs to delta against
• Can be specified multiple times!
• Default value: []
• --num-deltas=NUM_DELTAS the number of older versions to make deltas against
• Default value: 1
• --read-pkgs-list=READ_PKGS_LIST output the paths to the pkgs actually read useful with --update
• --max-delta-rpm-size=MAX_DELTA_RPM_SIZE max size of an rpm that to run deltarpm against (in bytes)
• Default value: 100000000

createrepo behavior

# ./ is empty directory
createrepo --delta .

empty drpms/ directory is created

empty prestodelta metadata are generated (and present in repomd.xml)

---

# "./" is empty directory
# "../myold/" contains sos-1.8-[69].fc11.noarch.rpm
createrepo --delta --oldpackagedirs ../myold/ .

empty drpms/ directory is created

empty prestodelta metadata are generated (and present in repomd.xml)

---

# "./" contains sos-1.8-10.fc11.noarch.rpm
# "../myold/" contains sos-1.8-[69].fc11.noarch.rpm
createrepo --delta --oldpackagedirs ../myold/ .

drpms/ directory contains sos-1.8-9.fc11_1.8-10.fc11.noarch.drpm

prestodelta contains 1 record for the file in drpms/

---

# "./" contains sos-1.8-10.fc11.noarch.rpm
# "../myold/" contains sos-1.8-[69].fc11.noarch.rpm
createrepo --delta --oldpackagedirs ../myold/ --num-deltas=2 .

drpms/ directory contains sos-1.8-6.fc11_1.8-10.fc11.noarch.drpm and sos-1.8-9.fc11_1.8-10.fc11.noarch.drpm

prestodelta contains 2 records for the files in drpms/

---

# "./" contains sos-1.8-10.fc11.noarch.rpm
# "drpms/" contains sos-1.8-[69].fc11_1.8-10.fc11.noarch.drpm from previous run
# "../myold/" contains sos-1.8-[69].fc11.noarch.rpm
createrepo --delta --oldpackagedirs ../myold/ --num-deltas=1 .

drpms/ directory contains sos-1.8-6.fc11_1.8-10.fc11.noarch.drpm and sos-1.8-9.fc11_1.8-10.fc11.noarch.drpm

prestodelta contains 2 records for the files in drpms/

---

# "./" contains ONLY "drpms/" directory and no files
# "drpms/" contains sos-1.8-[69].fc11_1.8-10.fc11.noarch.drpm from previous run
# "../myold/" contains sos-1.8-[69].fc11.noarch.rpm
createrepo --delta --oldpackagedirs ../myold/ --num-deltas=1 .

drpms/ directory contains sos-1.8-6.fc11_1.8-10.fc11.noarch.drpm and sos-1.8-9.fc11_1.8-10.fc11.noarch.drpm

prestodelta contains 2 records for the files in drpms/

primary.xml contains no packages

---

# "./" contains "sos-1.8-14.fc11.noarch.rpm"
# "drpms/" contains sos-1.8-[69].fc11_1.8-10.fc11.noarch.drpm from previous run
# "../myold/" contains sos-1.8-[69].fc11.noarch.rpm
createrepo --delta --oldpackagedirs ../myold/ --num-deltas=1 .

drpms/ directory contains sos-1.8-6.fc11_1.8-10.fc11.noarch.drpm and sos-1.8-9.fc11_1.8-10.fc11.noarch.drpm and sos-1.8-9.fc11_1.8-14.fc11.noarch.drpm

New prestodelta metadata contains 3 records for the files in drpms/

The old prestodelta file is still available in repodata/ directory, but not referenced by repomd.xml any more. <- Createrepo bug

createrepo drpms handling implementation

Important functions

main() (./genpkgmetadata.py)
+-- MetaDataGenerator.doPkgMetadata() (./createrepo/__init__.py)
   +-- MetaDataGenerator.writeMetadataDocs (./createrepo/__init__.py)
      +-- MetaDataGenerator._do_delta_rpm_package() (./createrepo/__init__.py)
   +-- MetaDataGenerator.closeMetadataDocs (./createrepo/__init__.py)
      +-- MetaDataGenerator.generate_delta_xml() (./createrepo/__init__.py)

Simplified workflow

drpms generation

• For listed oldpackagedirs prepare list of available (old) rpm files [_get_old_package_dict]
• Skip files that are bigger then max_delta_rpm_size (int(fpstat[stat.ST_SIZE]) > self.conf.max_delta_rpm_size)
• Of course skip files that are no rpm packages
• The result is dict where keys are the paths set as --oldpackagedir and values are list of paths to packages (the path used as a key + rpm filename)

During iteration over packages that are to be processed, for each package:

• Note do this for each oldpackage_paths
• If the package is bigger then max_delta_rpm_size skip it (size_installed > max_delta_rpm_size)
• Try to find matching old packages in the list created during previous step (candidates are searched by name - by prefix of basefilename)
• Skip "old packages" that are newer or the same as the current one.
• Skip "old packages" if name or arch doesn't match the current one.
• Sort the candidates and create several deltas (--num-deltas) by deltarpms.create_drpm
• Create drpm and place it to the deltadir

Candidate:

• name must match
• arch must match
• EVR must be lesser
• Must not be exactly the same package ()

prestodelta metadata generation

See: generate_delta_xml in createrepo/__init__.py.

• Iterate over the .drpms in the deltadir
• Generate XML chunk for each package NOTE: These chunks must be grouped by the "target" (new) package.
• Gen the whole prestodelta metadata file

Notes:

• New rpm files bigger then max_delta_rpm_size are skiped
• Old rpm files bigger then max_delta_rpm_size are skiped
• To be candidate for delta, the new and old package must:
• have the same name and arch
• not be exactly the same
• the old package must has got less version (NVR)
• For valid packages deltarpms.create_drpm is called

Parallel creation

Author: Ian Mcleod

Message on mailing list: http://lists.baseurl.org/pipermail/yum-devel/2014-February/010580.html

Branch: https://github.com/imcleod/createrepo/tree/feature/parallel_deltas_full

Related commit: https://github.com/imcleod/createrepo/commit/fa0520ffbdcfe517ad775101c2b31ceb5b87e19e

Simplified workflow:

• List of candidate packages sort by payload size
• Done by multiprocessing
• Each process loads a candidate package and get its payload size
• After it is done, and payload sizes of all packages are known, the list is sorted
• Delta repo creation
• Manager process is started
• This process create bunch of workers
• It selects a package by package's payload size to match the free memory and pass it to the worker pool
• Do the delta

prestodelta.xml.gz

<?xml version="1.0" encoding="UTF-8"?>
<prestodelta>
  <newpackage name="crypto-utils" epoch="0" version="2.4.1" release="44.fc20" arch="x86_64">
     <delta oldepoch="0" oldversion="2.4.1" oldrelease="39.fc19">
      <filename>drpms/crypto-utils-2.4.1-39.fc19_2.4.1-44.fc20.x86_64.drpm</filename>
      <sequence>crypto-utils-2.4.1-39.fc19-105e34f910b5c6ff569edb7d589cf4e21112d1</sequence>
      <size>61556</size>
      <checksum type="sha256">9dcac0d12f7cf61d767e885dc89666894b69f15da4b7081b8dce8c21dfc895ec</checksum>
    </delta>
    </newpackage>
   <newpackage name="createrepo" epoch="0" version="0.10.3" release="1.fc20" arch="noarch">
     <delta oldepoch="0" oldversion="0.9.9" oldrelease="22.fc20">
      <filename>drpms/createrepo-0.9.9-22.fc20_0.10.3-1.fc20.noarch.drpm</filename>
      <sequence>createrepo-0.9.9-22.fc20-9223d4ec911b1a67abd82abf97442f1ad4</sequence>
      <size>24808</size>
      <checksum type="sha256">73dac2ede6af9e065dce0c3d90a970c2adf98eabebe5d6ec6828ae9ea4333304</checksum>
    </delta>
     <delta oldepoch="0" oldversion="0.9.9" oldrelease="21.fc19">
      <filename>drpms/createrepo-0.9.9-21.fc19_0.10.3-1.fc20.noarch.drpm</filename>
      <sequence>createrepo-0.9.9-21.fc19-89e65ed1e245d6d8b608a64fe42971f2d4</sequence>
      <size>25004</size>
      <checksum type="sha256">ce560729b44d1525c0ae820e94ddceda34f4a7942ffdea3f0969c18fcb52686b</checksum>
    </delta>
    </newpackage>

</prestodelta>