Create a new extension revision only if submissiontype is experimental

[jcfr]

In case of nightly or continuous submission, there are no need to upload a new revision if extension metadata are matching.

Indeed, in that case the package are expected to be the same. In practice, a given extension generated for the same Slice revision/Extension revision/Arch/OS/... may be different if created a two different day. This could be explained by the fact year/month/day info are included in the package.

To prevent the upload of an extra extension revision when not needed, new item revision will be created only if submission type is experimental.

[aylward]

I propose that experimental uploads and continuous uploads should be
deleted 48 hours after they are uploaded.

That kind of garbage collection seems to be more generally useful as a
kitware technology - in cdash/midas.

Stephen

On Fri, Aug 16, 2013 at 2:19 PM, Jean-Christophe Fillion-Robin <
[email protected]> wrote:

In case of nightly or continuous submission, there are no need to upload
a new revision if extension metadata are matching.

Indeed, in that case the package are expected to be the same. In practice,
a given extension generated for the same Slice revision/Extension
revision/Arch/OS/... may be different if created a two different day. This
could be explained by the fact year/month/day info are included in the
package.

To prevent the upload of an extra extension revision when not needed, new
item revision will be created only if submission type is experimental.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3
.

Stephen R. Aylward, Ph.D.
Senior Director of Operations, North Carolina, Kitware, Inc.
http://www.kitware.com
http://www.aylward.org

(919) 969-6990 x300

[jcfr]

Currently there are below ~200 experimental extension uploads, don't think we should worry about them.

Let's also note that following Slicer/DashboardScripts@bbe4cb4 extension are always build/tested/packaged against the Slicer nightly build tree. In that case, instead of systematically deleting all continuous upload older than 48hrs we should probably keep the latest extension revision build against a given Slicer revision.

Doing so will ensure that every nightly package available will have a set of extensions associated to it.

[aylward]

I think having extensions for every nightly is extreme. After a few days,
they should be deleted.

On Fri, Aug 16, 2013 at 3:21 PM, Jean-Christophe Fillion-Robin <
[email protected]> wrote:

Currently there are below two hundreds experimental upload of extension,
don't think we should worry about them.

Let's also note that following Slicer/DashboardScripts@bbe4cb4Slicer/DashboardScripts@bbe4cb4cfde51f9121e7d2ff22f760de5a51f589extension are always build/tested/packaged against the Slicer nightly build
tree. In that case, instead of systematically deleting all continuous
upload older than 48hrs we should probably keep the latest extension
revision build against a given Slicer revision.

Doing so will ensure that every nightly package available will have a set
of extensions associated to it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-22787958
.

Stephen R. Aylward, Ph.D.
Senior Director of Operations, North Carolina, Kitware, Inc.
http://www.kitware.com
http://www.aylward.org

(919) 969-6990 x300

[jcfr]

@finetjul @aylward Since a lot of functionality moved into extension, I think it is useful to have Slicer nightly packages and associated extensions available.

I will quote @finetjul - email from june 2012 entitled 'slicer.kitware.com space saving':

The more I think about it the more I think that no package should be ever deleted, instead they
should  archived.
Keeping packages around is important because I doubt that it will be easy to build old 
versions of Slicer in the future. The Superbuild mechanism has the drawback to add 
dependency on repository we have no control over (PythonQt, DCMTK... ), I remember 
DCMTK recently rewrote their history and the SHA1 we have been using are no longer valid,
meaning it makes impossible to build 2 month old slicer anymore without some changes.
And even if we have control over those repo, I'm not sure they will be reachable
forever at the same address. 
[...]
If really we have to "save disk space" I think all the packages should be available for at
least 2 months, after that there should be no less than a set (Windows 32/64, mac and linux)
of packages per week. It might be a bit tricky as for some days, the set is not complete
(conf/build or upload issue).

Will be doing so manual cleaning in the Nighly folder of slicer.kitware.com and will discuss the different approach during next week developer hangout.

[aylward]

Meh ;)
On Aug 16, 2013 3:49 PM, "Jean-Christophe Fillion-Robin" <
[email protected]> wrote:

@finetjul https://github.com/finetjul @aylwardhttps://github.com/aylwardSince a lot of functionality moved into extension, I think it is useful to
have Slicer nightly packages and associated extensions available.

I will quote @finetjul https://github.com/finetjul - email from june
2012 entitled 'slicer.kitware.com space saving':

The more I think about it the more I think that no package should be ever deleted, instead they
should archived.
Keeping packages around is important because I doubt that it will be easy to build old
versions of Slicer in the future. The Superbuild mechanism has the drawback to add
dependency on repository we have no control over (PythonQt, DCMTK... ), I remember
DCMTK recently rewrote their history and the SHA1 we have been using are no longer valid,
meaning it makes impossible to build 2 month old slicer anymore without some changes.
And even if we have control over those repo, I'm not sure they will be reachable
forever at the same address.
[...]
If really we have to "save disk space" I think all the packages should be available for at
least 2 months, after that there should be no less than a set (Windows 32/64, mac and linux)
of packages per week. It might be a bit tricky as for some days, the set is not complete
(conf/build or upload issue).

Will be doing so manual cleaning in the Nighly folder of
slicer.kitware.com and will discuss the different approach during next
week developer hangout.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-22789427
.

[jcfr]

For reference - See http://massmail.spl.harvard.edu/public-archives/slicer-devel/2013/013526.html

[aylward]

Nice email!

s

On Fri, Aug 16, 2013 at 4:13 PM, Jean-Christophe Fillion-Robin <
[email protected]> wrote:

For reference - See
http://massmail.spl.harvard.edu/public-archives/slicer-devel/2013/013526.html

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-22790811
.

Stephen R. Aylward, Ph.D.
Senior Director of Operations, North Carolina, Kitware, Inc.
http://www.kitware.com
http://www.aylward.org

(919) 969-6990 x300