Add support for downloading artifacts from matrix builds #26

manics · 2014-01-17T15:30:46Z

Examples:

omego download --branch OMERO-5.1-latest --skipunzip matlab -v # Defaults to ICE=3.5
omego download --branch OMERO-5.1-latest --skipunzip matlab --labels=ICE=3.X -v

Multiple labels in any order (not much practical use at present):

omego download --branch OMERO-5.1-latest --skipunzip matlab --labels=label=ome-c6100-3,ICE=3.3

Must have only one match, otherwise should fail:

omego download --branch OMERO-5.1-latest --skipunzip matlab --labels='' -v
omego download --branch OMERO-5.1-latest --skipunzip matlab --labels='X=nonexistent' -v

Also --dry-run goes a bit further (downloads job XML), and an invalid job name is trapped:

omego download --branch OMERO-5.1-latest --skipunzip matlab -vn
omego download --branch NON-EXISTENT-JOB --skipunzip matlab -vn

Non-matrix jobs should still work

omego download --branch OMERO-4.4-merge-ice35 --skipunzip matlab -v

And omego upgrade should work too.

This means its possible to get a better idea of what would happen

manics · 2014-01-19T22:30:45Z

I'll fixup the tests...

AFAICT the only reason this previously worked was that calling omero.cli.CLI() or self.cli.loadplugins() in Upgrade.setup_script_environment() auto-creates a default config.xml (see comment in Upgrade.__init__()). Why has it broken now? No idea... maybe it's to do with importing multiple versions of the CLI in the nose test harness. Previously only one server was being upgraded in the test run.... perhaps when importing the CLI for the second time it isn't properly imported?

manics · 2014-01-21T18:25:16Z

Travis is now happy, though note commit message 34cf59c

pwalczysko · 2014-01-22T12:29:50Z

Meged this branch locally, and then, using the routine, where the omega command is replaced by the python execution of main.py in my local ome-setup repo like this:

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab -v

I started to execute the commands. See the output below.

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab -v
2014-01-22 12:26:52,229 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:26:52,229 [omego.artifa] DEBUG               branch => OMERO-5.1-latest
2014-01-22 12:26:52,229 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/
2014-01-22 12:26:52,229 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:26:52,229 [omego.artifa] DEBUG               labels => ICE=3.5
2014-01-22 12:26:52,229 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:26:52,247 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:26:52,254 [omego.artifa] DEBUG Searching for matrix runs matching: set(['ICE=3.5'])
2014-01-22 12:26:52,288 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.5,label=ome-c6100-3/19/api/xml code:200
2014-01-22 12:26:52,300 [omego.artifa] INFO  Downloading http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.5,label=ome-c6100-3/19/artifact/src/target/OMERO.matlab-5.0.0-rc1-108-3a67ad1-ice35-b19.zip
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:31:02Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab --labels=ICE=3.3 -v
2014-01-22 12:30:42,076 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:30:42,077 [omego.artifa] DEBUG               branch => OMERO-5.1-latest
2014-01-22 12:30:42,077 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/
2014-01-22 12:30:42,077 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:30:42,077 [omego.artifa] DEBUG               labels => ICE=3.3
2014-01-22 12:30:42,077 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:30:42,101 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:30:42,108 [omego.artifa] DEBUG Searching for matrix runs matching: set(['ICE=3.3'])
2014-01-22 12:30:42,123 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.3,label=ome-c6100-3/19/api/xml code:200
2014-01-22 12:30:42,136 [omego.artifa] INFO  Downloading http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.3,label=ome-c6100-3/19/artifact/src/target/OMERO.matlab-5.0.0-rc1-108-3a67ad1-ice33-b19.zip
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:31:39Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab --labels=ICE=3.4 -v
2014-01-22 12:31:21,306 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:31:21,306 [omego.artifa] DEBUG               branch => OMERO-5.1-latest
2014-01-22 12:31:21,307 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/
2014-01-22 12:31:21,307 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:31:21,307 [omego.artifa] DEBUG               labels => ICE=3.4
2014-01-22 12:31:21,307 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:31:21,325 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:31:21,332 [omego.artifa] DEBUG Searching for matrix runs matching: set(['ICE=3.4'])
2014-01-22 12:31:21,353 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.4,label=ome-c6100-3/19/api/xml code:200
2014-01-22 12:31:21,362 [omego.artifa] INFO  Downloading http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.4,label=ome-c6100-3/19/artifact/src/target/OMERO.matlab-5.0.0-rc1-108-3a67ad1-ice34-b19.zip
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:35:01Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab --labels='' -v
2014-01-22 12:34:38,280 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:34:38,280 [omego.artifa] DEBUG               branch => OMERO-5.1-latest
2014-01-22 12:34:38,280 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/
2014-01-22 12:34:38,280 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:34:38,280 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:34:38,297 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:34:38,301 [omego.artifa] DEBUG Searching for matrix runs matching: set([])
2014-01-22 12:34:38,301 [omego.artifa] ERROR Found 3 matching matrix build runs: ['http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.3,label=ome-c6100-3/19/', 'http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.4,label=ome-c6100-3/19/', 'http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.5,label=ome-c6100-3/19/']
Expected one matching run, found 3
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:42:34Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab --labels='X=nonexistent' -v
2014-01-22 12:41:59,926 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:41:59,926 [omego.artifa] DEBUG               branch => OMERO-5.1-latest
2014-01-22 12:41:59,926 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/
2014-01-22 12:41:59,926 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:41:59,926 [omego.artifa] DEBUG               labels => X=nonexistent
2014-01-22 12:41:59,927 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:42:00,286 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:42:00,294 [omego.artifa] DEBUG Searching for matrix runs matching: set(['X=nonexistent'])
2014-01-22 12:42:00,294 [omego.artifa] ERROR Found 0 matching matrix build runs: []
Expected one matching run, found 0
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:48:17Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch OMERO-5.1-latest --skipunzip matlab -vn
2014-01-22 12:46:09,771 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:46:09,771 [omego.artifa] DEBUG               branch => OMERO-5.1-latest
2014-01-22 12:46:09,771 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/
2014-01-22 12:46:09,771 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:46:09,771 [omego.artifa] DEBUG               labels => ICE=3.5
2014-01-22 12:46:09,771 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:46:09,785 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:46:09,792 [omego.artifa] DEBUG Searching for matrix runs matching: set(['ICE=3.5'])
2014-01-22 12:46:09,806 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-5.1-latest/ICE=3.5,label=ome-c6100-3/19/api/xml code:200
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:50:00Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py download --branch NON-EXISTENT-JOB --skipunzip matlab -vn
2014-01-22 12:49:39,570 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:49:39,570 [omego.artifa] DEBUG               branch => NON-EXISTENT-JOB
2014-01-22 12:49:39,570 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/NON-EXISTENT-JOB/lastSuccessfulBuild/
2014-01-22 12:49:39,570 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:49:39,570 [omego.artifa] DEBUG               labels => ICE=3.5
2014-01-22 12:49:39,570 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:49:39,575 [omego.artifa] ERROR Failed to get CI XML (HTTP Error 404: Not Found)
Job lookup failed, is the job name correct?
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T12:51:10Z

[petr@LS25165 ~/Work/omero5-server]$ python ~/Work/omero-setup/omego/main.py  download --branch OMERO-4.4-merge-ice35 --skipunzip matlab -v
2014-01-22 12:50:50,164 [omego.artifa] DEBUG             artifact => matlab
2014-01-22 12:50:50,164 [omego.artifa] DEBUG               branch => OMERO-4.4-merge-ice35
2014-01-22 12:50:50,164 [omego.artifa] DEBUG                build => http://ci.openmicroscopy.org/job/OMERO-4.4-merge-ice35/lastSuccessfulBuild/
2014-01-22 12:50:50,164 [omego.artifa] DEBUG                   ci => ci.openmicroscopy.org
2014-01-22 12:50:50,164 [omego.artifa] DEBUG               labels => ICE=3.5
2014-01-22 12:50:50,164 [omego.artifa] DEBUG                unzip => unzip
2014-01-22 12:50:50,312 [omego.artifa] DEBUG Fetching xml from http://ci.openmicroscopy.org/job/OMERO-4.4-merge-ice35/lastSuccessfulBuild/api/xml code:200
2014-01-22 12:50:50,324 [omego.artifa] INFO  Downloading http://ci.openmicroscopy.org/job/OMERO-4.4-merge-ice35/lastSuccessfulBuild/artifact/src/target/OMERO.matlab-4.4.10-3-19c9a1e-ice35-b181.zip
[petr@LS25165 ~/Work/omero5-server]$

pwalczysko · 2014-01-22T15:49:16Z

The omego upgrade worked too as expected. All seems fine here.
After discussion with @manics I am creating a github issue on one-setup repo for simplifying the options on omego. See #27

sbesson · 2014-01-27T08:21:16Z

@manics: any additional commit?

manics · 2014-01-27T09:03:38Z

@sbesson No.

Add support for downloading artifacts from matrix builds

manics added 8 commits January 17, 2014 14:21

Fetch artifacts from child runs of matrix builds

a0ab627

Add label matching for matrix build runs

3a14c6f

Set the default label value to ICE=3.5

f1e5893

Trap HTTPError when fetching CI XML

52ec7b1

Discard empty labels

cd8506f

Log an ERROR if incorrect number of matching matrix runs found

084de6d

--dry-run only stops the final download stage

fc3df7f

This means its possible to get a better idea of what would happen

Test upgrade to a matrix build

f02b6c3

manics added 2 commits January 21, 2014 17:38

Fix line length

17c033a

pwalczysko mentioned this pull request Jan 22, 2014

Simplify the command line options in omego #27

Open

sbesson added a commit that referenced this pull request Jan 27, 2014

Merge pull request #26 from manics/artifacts_matrix

3c630c0

Add support for downloading artifacts from matrix builds

sbesson merged commit 3c630c0 into ome:master Jan 27, 2014

manics deleted the artifacts_matrix branch January 28, 2014 10:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for downloading artifacts from matrix builds #26

Add support for downloading artifacts from matrix builds #26

manics commented Jan 17, 2014

manics commented Jan 19, 2014

manics commented Jan 21, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

sbesson commented Jan 27, 2014

manics commented Jan 27, 2014

Add support for downloading artifacts from matrix builds #26

Add support for downloading artifacts from matrix builds #26

Conversation

manics commented Jan 17, 2014

manics commented Jan 19, 2014

manics commented Jan 21, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

pwalczysko commented Jan 22, 2014

sbesson commented Jan 27, 2014

manics commented Jan 27, 2014