Installer options such as require PPAs

[NikolausDemmel]

Here is a proposal for how to realize this:
http://xylem.readthedocs.org/en/latest/design.html#installer-options

Feel free to comment / discuss in this issue.

[NikolausDemmel]

cc @wjwwood @tfoote @dirk-thomas @esteve

[tfoote]

I like having the installer_options for the top of the file. I'm not sure if supporting rule specific ppa requirements is good for overall complexity of maintenance of the rules files. if each PPA requirement is at the top you can capture two PPAs in with half the lines by using two files instead of interleaving the required ppa for each entry.

The other question I have is about how to express the PPAs. You're currently implying that the PPA is some sort of reference into a config file, I'd much rather see the PPA parameterization captured in the rules file header. For example you may want to add the GPG keys for a custom repo in addition to the url.

[NikolausDemmel]

I agree that in general rules files are best split up to have at most one ppa per file and put that at the top, which should be the documented best practice. However, do we need to prevent people from the possibility of using per-key PPA requirements? I'm not saying this is needed for the rules files, but I like the idea of having these requirements at the key-level in the expande rules file, such that only the requirements for keys actually to be installed are checked. Expanding the rules files such that these keys are attached to each key is the simplest way to achieve that IMO. Any alternative would require to track during resolution where each rule was coming from.

I was actually not implying a reference to a config file and I agree that the info in the rules file should be self contained. I was only considering ubuntu launchpad PPAs which can be added with apt-add-repository ppa:user/repo, where the key is automatically added from the ubuntu keyserver. But this is indeed not sufficient even for the ROS repos, let alone custom robot vendor PPAs. Instead of just a string, PPAs should therefore be specified by a dict with the following keys:

• sourceline: source line added to the apt sources config TYPE URL DISTRIBUTION COMPONENTS; possibly allow the same shortcuts as apt-add-repository
  • TYPE: binary (deb) or source (deb-src); defaults to binary
  • URL: url of the package repository; ppa:user/repo shorthand for Ubuntu launchpad PPA's is allowed
  • DISTRIBUTION: usually the OS version, but can be something like precise-updates or stable; we need some way to substitute the current OS version here, maybe $OS_VERSION-updates; defaults to current os version
  • COMPONENTS: components of the PPA; defaults to main
• fingerprint: fingerprint of the key; optional for ubuntu launchpad ppa:user/repo style PPAsl; other-wise used to check if the key is already installed and to confirm the downloaded key
• keyserver: url of the keyserver where the key can be retrieved from; default to apt sources default keyserver
• key: direct url to the key; can be used instead of keyserver

We keep talking of PPAs, but I guess strictly speakin that is referring to Ubuntu launchpad PPAs only? Maybe better term this "apt repository". It probably makes sense to have the rule-file-wide repositry definition depend on os and verison. Maybe like this:

_any_key:
  ubuntu:
    any_version:
      apt:
        repositories:
          sourceline: "deb http://packages.ros.org/ros/ubuntu $OS_VERSION main"
          key: https://raw.githubusercontent.com/ros/rosdistro/master/ros.key
          fingerprint: 1234567890ABCDEF

[tfoote]

I agree using the repositories keyword is better and reusing the apt-add-repository keywords is a good idea.

For the availability of per rule repository definitions makes everything more complicated. Unless there's a use case that is not met, I believe that adding the complexity to the file, and the parsing, and for the maintainer will simply create more work at every level. And I think that since we can have as many source files as required the rules can be maintained in separate files anyway.

[NikolausDemmel]

Ok, I can understand your point. I agree that there is no use case as of now that can not be served well with per-rules-file-only apt-repository information. My main argument for anchoring that information at the rule level is that it makes sense from the implementation perspective, i.e. I suggest to transfer this information to every key upon rule file expansion such that during lookup the repositories are checked/installed if and only if one of the xylem keys from that file is being installed and has resolved to an apt-rule.

Independent from the implementation, what do you suggest how xylem should use the repository information. At what point should xylem check/install a repository defined in a rules file?

[tfoote]

I think that xylem should check at load time not recheck on every evaluation. There's some level of caching possible for the code path.

I would suggest that xylem be able to automatically add sources only when explicitly asked for by an option like --auto-add-sources

Case 1: Repository declared and missing and --auto-add set
Check at beginning and add if necessary, evaluate with rules file

Case 2: Repository declared and missing
Ignore the rules file for which repos are not satisfied (with clear warning) Option --strict to force an error.

Case 3: Repository declared and missing --ignore-missing-repositories set
Evaluate with all rules file, it will likely fail to install but could be used for debugging.

Case 4: Repository declared and present
Operate as normal, evaluate with rules file.

Case 5: Repository not declared
Operate as normal.

[NikolausDemmel]

What do you mean when you say load time? What is evaluation? I don't understand your sentence about "caching possible for the code path."

Just to make sure we are on the same page. The workflow in xylem is the following for rules files and the database:

• during update the rules files are downloaded and expaneded. Expansion simply means removing any syntactic shortcuts to get a more verbose representation that is better suited for rule lookup. This expanded rules dictionary is pickled.
• during commands like resolve or check or install the cached rules dictionaries are loaded from disk. This comprises the database, which can now be queried for indivdual rules.
  What part of the above constitutes the "evaluation" of rules files that you refer to?

I assume you mean the repositories should be checked when the database is loaded (not on update) at the start of an invocation like xylem install ...? I assume you mean to check only the repositories specified for the current os and version.

To me, that doesn't make much sense.

• What if I load the database to do some rule lookup and don't actually install anything?
• What if I am on OS X and do --os ubuntu:precise. I assume your case 2 would not apply here, i.e. all rules are loaded. Does that mean that lookup on ubuntu precise without the repositories installed will give me fewer resolutions than on OS X with --os ubuntu:precise?
• What about repository information for installers I never intend to use at all? E.g. a homebrew user on OS X should not get warnings about macports rules for which some repository is not present.

Is there a specific reason you do not want to perform this check just before an install command would be issued? I think this is not only closer to what I as a user would want, but also much simpler to implement. With my suggestion, only the installer plugin needs to worry at all about this information. It does not receive any kind of special treatment during rules file expansion, loading of the database, rule lookup / resolution, ... None of that code needs to know anything about apt-repositories. No heuristics at "rules file evaluation" about whether or not certain repositories should be checked given the detected/overwritten OS. I don't even need to load any installer plugins at all just to load the rule database into memory and lookup rools (although I do need the installer plugins for resolve and of course install & friends). Keep in mind also that apt-repositories should not be in any way more special that e.g. homebrew taps or any equivalent for RPM etc...

Note that with my proposal the check happens at most once for every xylem install command issued. The workflow for install with respect to that would be:

• All requested xylem keys are resolved (including dependency resolution) given the detected/requested OS and version. This gives an ordered list of "resolutions", one for each xylem key, which each are associated to exactly one installer. The resolution is the "installer rule". The installer rule is a dictionary which often only has a packages entry, but might have other installer specific entries such as options for the homebrew installer, or -- as I propose -- information about requried repositories for the apt installer. The installer plugins can convert these "installer rules" to install commands, reinstall commands, commands checking for successful installation, etc...
  • Before the installer commands for those resolutions are executed in order, the installer plugins get a chance to inspect all "their" resolutions and check for prerequisites. In case of the apt installer, it would pick up any repository information, perform the necessary checks, optionally offer to install the repos or issue a warning/error.
  • Another "prerequisite" to be checked at this point is whether the installer is actually present / executable. However this prerequisite does not depend on the actual content of the resolutions.
  • If everything is fine, or if the user requested to skip the tests / ignore failures, xylem proceeds to issue the installer commands.

[tfoote]

As long as the check is only done once per invocation and not once per package that will work. That's the main think I want to avoid.

[NikolausDemmel]

Ok great, I fully agree about only one invocation per xylem install ... call. This ensured by what I am proposing.

[trainman419]

Is it possible to stretch this a little bit to include enabling Ubuntu's universe and multiverse repositories? I continue to see users on ROS answers that have dependency problems because they missed that step in the installation guide.

If you think this is better addressed elsewhere, that is fine as well.

[NikolausDemmel]

@trainman419: Is there anything special in the way these "source lines" should be added to /etc/apt/sources.list. Can it just be appended like any other new repo, or do we need to find the existing line and uncomment it?

[tfoote]

I would suggest that once this is fully supported that we just add these as sources in our standard rosdep rules. If they've already uncommented it, that should not be a problem. And if they haven't appending shouldn't be a problem. Note for the implementation we should probably be working in our own file in the sources.list.d/ directory.

[NikolausDemmel]

Sounds goog. So no special case required.