As a developer, I want to improve the dockerize.sh

[epag]

---

Author Name: James (James)
Original Redmine Issue: 104957, https://vlab.noaa.gov/redmine/issues/104957
Original Date: 2022-05-19

---

Given a @dockerize.sh@ that automates the tagging of an image to reduce errors
When I see some frictions in that script
Then I want to remove them

[epag]

---

Original Redmine Comment
Author Name: James (James)
Original Date: 2022-05-19T15:13:33Z

---

One improvement is to validate the @DOCKER_REGISTRY@ env variable for absence of an http(s) scheme, which will produce an invalid tag name (#104715-50).

Another is for the script to use a designated jenkins build number, rather than the latest build (because the latest build is often not the RC).

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-19T15:26:51Z

---

Reiterating what I stated in the other ticket, the change I made to @dockerize.sh@ was to add a check around the activities that occur if the DOCKER_REGISTRY environment variable is defined. The check is as follows:

    if [[ $DOCKER_REGISTRY == http* ]]

If "http" is present at the beginning of the variable, a message is output saying that the scheme should not be included and the registry stuff is skipped. The registry activities are then within the else portion of that if-statement.

I'll push the change to the repository later this afternoon.

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T12:57:45Z

---

Implemented James's if-clause and it seems to work:

if [[ $DOCKER_REGISTRY =~ ^https?:// ]]; then
    echo "fubared!" 
fi

Well, the message is more meaningful than "fubared", but whatever.

I'm going to take a quick look to see how difficult it would be for the script to download the .zips from the build instead of the active work space.

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T13:07:17Z

---

I think all the artifacts that need to be downloaded can be found in the Jenkins build area for a given build/revision, which is identified by a number. For example, I believe this corresponds to 6.4:

https://vlab.noaa.gov/jenkins/job/Verify_OWP_WRES/4418/artifact/

Currently, the script builds the URLs from the Jenkins workspace:

jenkins_workspace=$JOB_URL/ws
core_url=$jenkins_workspace/build/distributions/$wres_core_file
worker_url=$jenkins_workspace/wres-worker/build/distributions/$worker_shim_file
tasker_url=$jenkins_workspace/wres-tasker/build/distributions/$tasker_file
vis_url=$jenkins_workspace/wres-vis/build/distributions/$vis_file

$JOB_URL is the Jenkins URL, https://vlab.noaa.gov/jenkins/job/Verify_OWP_WRES.

Is there an automated way to identify the build number in Jenkins? If not, is it reasonable to expect us to identify that number and pass it in as an argument to @dockerize.sh@?

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T13:09:41Z

---

And will the output from @versions.sh@ match the versions of the .zips in the Jenkins build?

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T13:13:28Z

---

The version is identified and built manually in @versions.sh@ based on git:

1. Accepts any number of directories arguments, e.g. of modules.
1. Returns the most recent WRES version auto-detected from git.
1. When the Dockerfile has uncommitted modifications, flag "-dev" (dirty).
function get_ver
{
    last_commit_hash=$( git log --format="%h" -n 1 -- build.gradle gradle $@ )
    last_commit_date=$( git log --date=iso8601 --format="%cd" -n 1 -- build.gradle gradle $@ )
    last_commit_date_utc=$( date --date "${last_commit_date}" --iso-8601 --utc )
    last_commit_date_short=$( echo ${last_commit_date_utc} | sed 's/\-//g' - )
    potential_version=${last_commit_date_short}-${last_commit_hash}

    # Look for the git status of the dockerfile for the directory passed
    dockerfile_modified=$( git status --porcelain -- $1/Dockerfile | grep "^ M" )

    if [ "$dockerfile_modified" != "" ]
    then
        echo ${potential_version}-dev
    else
    echo ${potential_version}
    fi
}

I have no clue if that will properly correspond to the zip versions found in the Jenkins build, because its not talking to Jenkins. I don't have time for a full rewrite of that section, but maybe in the future something less brittle can be built.

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T13:17:11Z

---

The URL for the Jenkins build includes the revision:

https://vlab.noaa.gov/jenkins/job/Verify_OWP_WRES/4418/

So if you know the build number, you know the revision.

Can we go in the opposite direction? Is there a way to identify the build number for a revision?

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T13:26:33Z

---

Quick internet search implies it would either require a plug-in or require a build search (i.e., looping over builds until we find the revision).

Proposal: As part of the deployment process we identify a revision. We can also identify the Jenkins build number. We can then pass that build number in to @dockerize.sh@ as the first argument and use it to locate the .zips to download instead of assume "ws" in the URL.

Thoughts?

This would be a small change that I can make now, but we would need to remember the added step when we next deploy. Perhaps I can change the Subject of the 6.4 deployment ticket to include the Jenkins build number, 4418, so that can serve as a reminder.

Hank

[epag]

---

Original Redmine Comment
Author Name: James (James)
Original Date: 2022-05-20T13:54:12Z

---

My thoughts are that we should supply a build number to the script, because we nominate a build to deploy, i.e., every RC starts with a given build that succeeded and the identity of that build is what we should supply when creating the images.

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T14:00:10Z

---

Let me put that change in, test, and push,

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T14:18:55Z

---

Added a required first argument, the Jenkins build number, to the @dockerize.sh@. Confirmed that the URL identified, at least for the core WRES build, is correct and the @curl@ command recommended in the script output works. The full test of the change will happen when deploying 6.5

Given the generic nature of this ticket, this will not resolve the ticket. I also don't think this resolves #99827. Yes, moving to using the build number instead of the workspace helps, but the versions and names of the .zip are still being identified independently of Jenkins, hopefully using the same process, but without a check. Thus, I think there is still risk that a .zip identified is not correct.

Let me push my change against both tickets.

Hank

[epag]

---

Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2022-05-20T14:22:40Z

---

commit:f26e223358f5d65bdc9476c54756ab837b09f186

Hank