CKAN 2.9 images embed CKAN 2.10 schema

[themowski]

I've noticed a discrepancy between the Makefiles and the Dockerfiles in regards to how the CKAN version is handled. As a result, the CKAN 2.9 versions of the images end up built with the wrong schemas.

The solrN/Makefile supports the CKAN_VERSION variable. It expects something of the form X.Y, and creates the ckan/ckan-solr:X.Y-solrN Docker image.

The solrN/Dockerfile and solrN/Dockerfile.spatial use a Docker ARG called CKAN_BRANCH that is hardcoded with the default value dev-v2.10.

Importantly, the build argument in the Dockerfiles is not integrated with the corresponding Makefile. What this means is that, regardless of the value of CKAN_VERSION, the dev-v2.10 branch is always used to pull the Solr schema.

Demonstration

To demonstrate, I ran these steps:

# Build against CKAN 2.9
$ make build CKAN_VERSION=2.9
(... output truncated ...)

# Create a container called solr using the image that was built
$ docker run -d --rm --name solr ckan/ckan-solr:2.9-solr9
(... output truncated ...)

# Print out the schema and look for the "schema name" block
$ docker exec solr cat /opt/solr/server/solr/configsets/ckan/conf/managed-schema | grep 'schema name'
<schema name="ckan-2.10" version="1.6">

As the output of the final command shows, the schema is the ckan-2.10 schema, even though I supposedly built a Solr image for CKAN 2.9. As further confirmation, here is the raw schema from the ckan-2.9.9 tag:

https://raw.githubusercontent.com/ckan/ckan/ckan-2.9.9/ckan/config/solr/schema.xml

The relevant line (below) does not match what is installed in the ckan-solr:2.9-solr9 image:

<schema name="ckan" version="2.9">

Possible Fix 1

I think as an easy solution, you could introduce CKAN_BRANCH in the Makefile and explicitly pass it via --build-arg when you invoke docker build, so that users can explicitly select the branch (or tag) from which the Solr schema will be pulled:

CKAN_VERSION="2.10"
CKAN_BRANCH="dev-v2.10"

(...)

build:
    docker build --build-arg CKAN_BRANCH=$(CKAN_BRANCH) -t $(TAG_NAME) -f Dockerfile .
    docker build --build-arg CKAN_BRANCH=$(CKAN_BRANCH) -t $(TAG_NAME) -f Dockerfile.spatial .

I think allowing the CKAN_BRANCH override is important, because it lets people choose to build based on a specific tag. For my team's use-case, we need to be able to specify a specific tagged version of CKAN to build from, for reproducibility.

# Build based on the the ckan-2.10.1 tag in ckan/ckan
make build CKAN_BRANCH=ckan-2.10.1

That said, the CKAN_BRANCH option is still disjoint from the CKAN_VERSION argument, so users would need to specify both values to correctly build a CKAN 2.9-compatible image. For example:

# Build based on the ckan-2.9.9 tag in ckan/ckan
make build CKAN_VERSION=2.9 CKAN_BRANCH=ckan-2.9.9

This is somewhat confusing, but with proper documentation / examples it is probably not a huge deal.

Possible Fix 2

As an alternative option, I am wondering if a better strategy is to adopt a strategy more similar to what ckan/ckan-docker-base does, where directories map to explicit versions of CKAN:

ckan-solr/
  ckan-2.10/
    solr8/
    solr9/
  ckan-2.9/
    solr8/
    solr9/

There is more code duplication this way, but it's a lot more clear what you're getting with each Makefile, and theoretically you won't ever need to update these once they're created. This would also allow you to have better control over which version combinations of Solr & CKAN are supported.

If this approach were to be implemented, I would still want the ability to override the CKAN_BRANCH value at build time so that I can guarantee that I'm building something compatible with CKAN 2.10.1 or CKAN 2.9.8 (for example). The advantage of this approach over the other one is that we would not need to worry about CKAN_VERSION, because that would be set correctly for the Makefile based on the directory we are in.