Remove preview file values (#1230)

* feature (upgrade): Introduce upgrade tools directory. * refactor (api-v1): Don't store preview images (ongoing). * test (api-v2): Fix tests. * test (sipi): Fix integration tests. * docs (knora-ontologies): Update docs. * test (api-v1): Fix test data.
dasch-swiss · Feb 28, 2019 · c069371 · c069371
1 parent 7c69279
commit c069371
Show file tree

Hide file tree

Showing 63 changed files with 396,527 additions and 476,952 deletions.
diff --git a/docs/src/paradox/00-release-notes/next.md b/docs/src/paradox/00-release-notes/next.md
@@ -4,17 +4,30 @@ Write any new release notes between releases into this file. They will be moved
 at the time of the release, because only then we will know the next release number.
 
 Also, please change the **HINT** to the appropriate level:
- - MAJOR: the changes introduced warrant a major number increase
- - FEATURE: the changes introduced warrant a minor number increase
- - FIX: the changes introduced warrant a bugfix number increase
+
+- MAJOR: the changes introduced warrant a major number increase
+
+- FEATURE: the changes introduced warrant a minor number increase
+
+- FIX: the changes introduced warrant a bugfix number increase
 
 
 ## HINT => MAJOR CHANGE
 
 - MAJOR: Use HTTP POST to mark resources and values as deleted (@github[#1203](#1203))
+
 - MAJOR: Reorganize user and project routes (@github[#1209](#1209))
+
 - FEATURE: Secure routes returning user informations (@github[#961](#961))
-- MAJOR: Change all `xsd:dateTimeStamp` to `xsd:dateTime` in the triplestore (@github[#1211](#1211)). Existing data must be updated.
+
+- MAJOR: Change all `xsd:dateTimeStamp` to `xsd:dateTime` in the triplestore (@github[#1211](#1211)).
+  Existing data must be updated; see `upgrade/1211-datetime` for instructions.
+
 - FIX: Ignore order of attributes when comparing standoff (@github[#1224](#1224)).
+
 - FEATURE: Query version history (@github[#1214](#1214))
+
 - FIX: Don't allow conflicting cardinalities (@github[#1229](#1229))
+
+- MAJOR: Remove preview file values (@github[#1230](#1230)). Existing data must be updated;
+  see `upgrade/1230-delete-previews` for instructions.
diff --git a/docs/src/paradox/02-knora-ontologies/knora-base.md b/docs/src/paradox/02-knora-ontologies/knora-base.md
@@ -185,8 +185,7 @@ to store files. The @ref:[Knora APIs](../03-apis/index.md) provide ways
 to create file values using Knora and Sipi.
 
 A resource that has a file value must belong to one of the subclasses of
-`kb:Representation`. Its subclasses, which are intended to be used directly in data,
-include:
+`kb:Representation`. Its subclasses include:
 
 `StillImageRepresentation`
 
@@ -213,12 +212,15 @@ include:
 :   A representation containing a document (such as a PDF file) that is
     not a text file.
 
+These classes can be used directly in data, but it is often better to make
+subclasses of them, to include metadata about the files being stored.
+
 The base class of all these classes is `Representation`, which is not intended to
 be used directly. It has this property, which its subclasses override:
 
-`hasFileValue` (1-n)
+`hasFileValue` (1)
 
-:   Points to one or more file values.
+:   Points to a file value.
 
 There are two ways for a project to design classes for representations.
 The simpler way is to create a resource class that represents a thing in
@@ -229,12 +231,11 @@ by still images, `ex:Painting` could be a subclass of
 `StillImageRepresentation`. This is the only approach supported in
 @ref:[Knora API v1](../03-apis/api-v1/index.md).
 
-The more flexible approach, which is supported by
-@ref:[Knora API v2](../03-apis/api-v2/index.md), is for each
-`ex:Painting` to use the `kb:hasRepresentation` property to point to
-other resources containing files that represent the painting. Each of
-these other resources can extend a different subclass of
-`Representation`. For example, a painting could have a
+The more flexible approach, which is supported by @ref:[Knora API
+v2](../03-apis/api-v2/index.md), is for each `ex:Painting` to link (using
+`kb:hasRepresentation` or a subproperty) to other resources containing files
+that represent the painting. Each of these other resources can extend a
+different subclass of `Representation`. For example, a painting could have a
 `StillImageRepresentation` as well as a `DDDrepresentation`.
 
 #### Standard Resource Classes

diff --git a/docs/src/paradox/03-apis/api-v2/editing-values.md b/docs/src/paradox/03-apis/api-v2/editing-values.md
@@ -187,7 +187,7 @@ provided.
 
 Knora supports the storage of certain types of data as files, using
 [Sipi](https://github.com/dhlab-basel/Sipi)
-(see @ref:[FileValue](../../02-knora-ontologies/knora-base.md#FileValue)).
+(see @ref:[FileValue](../../02-knora-ontologies/knora-base.md#filevalue)).
 Knora API v2 currently supports using Sipi to store image files. Support for
 other types of files will be added in the near future.
 
@@ -243,21 +243,37 @@ a JPG thumbnail image that is 150 pixels wide, you would add
 
 ### Submit A File Value to Knora
 
-After uploading a file to Sipi, you can submit a request to Knora to create
-a file value. The request submitted to Knora takes the same form as any other
-request to create a value. Instead of providing the file's complete metadata,
-you just provide the unique internal filename generated by Sipi. For example:
+A Knora `Representation` (i.e. a resource containing information about a
+file) must always have exactly one file value attached to it. (see
+@ref:[Representations](../../02-knora-ontologies/knora-base.md#representations)).
+Therefore, a request to create a new file value must always be submitted as part
+of a request to create a new resource (see
+@ref:[Creating a Resource](editing-resources.md#creating-a-resource)).
+You can also update a file value in an existing `Representation`; see
+@ref:[Updating a Value](#updating-a-value).
+
+Instead of providing the file's complete metadata to Knora, you just provide the
+unique internal filename generated by Sipi. Here is an example of a request to
+create a resource of class `anything:ThingPicture`, which is a subclass of
+`knora-api:StillImageRepresentation` and therefore has the property
+`knora-api:hasStillImageFileValue`:
 
 ```jsonld
 {
-  "@id" : "http://rdfh.ch/0001/a-thing-picture",
   "@type" : "anything:ThingPicture",
   "knora-api:hasStillImageFileValue" : {
     "@type" : "knora-api:StillImageFileValue",
     "knora-api:fileValueHasFilename" : "3UIsXH9bP0j-BV0D4sN51Xz.jp2"
   },
+  "knora-api:attachedToProject" : {
+    "@id" : "http://rdfh.ch/projects/0001"
+  },
+  "rdfs:label" : "test thing",
   "@context" : {
+    "rdf" : "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
     "knora-api" : "http://api.knora.org/ontology/knora-api/v2#",
+    "rdfs" : "http://www.w3.org/2000/01/rdf-schema#",
+    "xsd" : "http://www.w3.org/2001/XMLSchema#",
     "anything" : "http://0.0.0.0:3333/ontology/0001/anything/v2#"
   }
 }

diff --git a/knora-ontologies/knora-base.ttl b/knora-ontologies/knora-base.ttl
@@ -1168,18 +1168,6 @@
 
 
 
-###  http://www.knora.org/ontology/knora-base#isPreview
-
-:isPreview rdf:type owl:DatatypeProperty ;
-
-           rdfs:subPropertyOf :valueHas ;
-
-           :subjectClassConstraint :FileValue ;
-
-           :objectDatatypeConstraint xsd:boolean .
-
-
-
 ###  http://www.knora.org/ontology/knora-base#isRootNode
 
 :isRootNode rdf:type owl:DatatypeProperty ;
@@ -1263,18 +1251,6 @@
 
 
 
-###  http://www.knora.org/ontology/knora-base#qualityLevel
-
-:qualityLevel rdf:type owl:DatatypeProperty ;
-
-              rdfs:subPropertyOf :valueHas ;
-
-              :subjectClassConstraint :FileValue ;
-
-              :objectDatatypeConstraint xsd:integer .
-
-
-
 ###  http://www.knora.org/ontology/knora-base#valueHasComment
 
 :valueHasComment rdf:type owl:DatatypeProperty ;
@@ -1623,7 +1599,7 @@
                      rdfs:subClassOf :Representation  ,
                                 [ rdf:type owl:Restriction ;
                                   owl:onProperty :hasAudioFileValue ;
-                                  owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                  owl:cardinality "1"^^xsd:nonNegativeInteger
                                 ] ;
 
                      rdfs:label "Repräsentation (Audio)"@de ,
@@ -1672,7 +1648,7 @@
                    rdfs:subClassOf :Representation  ,
                                 [ rdf:type owl:Restriction ;
                                   owl:onProperty :hasDDDFileValue ;
-                                  owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                  owl:cardinality "1"^^xsd:nonNegativeInteger
                                 ] ;
 
                    rdfs:label "Repräsentation (3D)"@de ,
@@ -1740,7 +1716,7 @@
                         rdfs:subClassOf :Representation ,
                                 [ rdf:type owl:Restriction ;
                                   owl:onProperty :hasDocumentFileValue ;
-                                  owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                  owl:cardinality "1"^^xsd:nonNegativeInteger
                                 ] .
 
 
@@ -1789,10 +1765,6 @@
 :FileValue rdf:type owl:Class ;
 
            rdfs:subClassOf :Value ,
-                           [ rdf:type owl:Restriction ;
-                             owl:onProperty :isPreview ;
-                             owl:maxCardinality "1"^^xsd:nonNegativeInteger
-                           ] ,
                            [ rdf:type owl:Restriction ;
                              owl:onProperty :originalMimeType ;
                              owl:cardinality "1"^^xsd:nonNegativeInteger
@@ -2071,10 +2043,6 @@
                                         owl:onProperty :fps ;
                                         owl:cardinality "1"^^xsd:nonNegativeInteger
                                       ] ,
-                                      [ rdf:type owl:Restriction ;
-                                        owl:onProperty :qualityLevel ;
-                                        owl:cardinality "1"^^xsd:nonNegativeInteger
-                                      ] ,
                                       [ rdf:type owl:Restriction ;
                                         owl:onProperty :duration ;
                                         owl:cardinality "1"^^xsd:nonNegativeInteger
@@ -2091,7 +2059,7 @@
                            rdfs:subClassOf :Representation  ,
                                 [ rdf:type owl:Restriction ;
                                   owl:onProperty :hasMovingImageFileValue ;
-                                  owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                  owl:cardinality "1"^^xsd:nonNegativeInteger
                                 ] ;
 
                            rdfs:label "Repräsentation (Video)"@de ,
@@ -2149,15 +2117,15 @@
                 rdfs:subClassOf :Resource ,
                                 [ rdf:type owl:Restriction ;
                                   owl:onProperty :hasFileValue ;
-                                  owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                  owl:cardinality "1"^^xsd:nonNegativeInteger
                                 ] ;
 
                 rdfs:label "Repräsentation"@de ,
                            "Representation"@en ,
                            "Répresentation"@fr ,
                            "Rappresentazione"@it ;
 
-                rdfs:comment "A resource that can store one or more files"@en .
+                rdfs:comment "A resource that can store a file"@en .
 
 
 
@@ -2406,10 +2374,6 @@
                                        owl:onProperty :dimY ;
                                        owl:cardinality "1"^^xsd:nonNegativeInteger
                                      ] ,
-                                     [ rdf:type owl:Restriction ;
-                                       owl:onProperty :qualityLevel ;
-                                       owl:cardinality "1"^^xsd:nonNegativeInteger
-                                     ] ,
                                      [ rdf:type owl:Restriction ;
                                        owl:onProperty :dimX ;
                                        owl:cardinality "1"^^xsd:nonNegativeInteger
@@ -2426,15 +2390,15 @@
                           rdfs:subClassOf :Representation ,
                                 [ rdf:type owl:Restriction ;
                                   owl:onProperty :hasStillImageFileValue ;
-                                  owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                  owl:cardinality "1"^^xsd:nonNegativeInteger
                                 ] ;
 
                           rdfs:label "Repräsentation (Bild)"@de ,
                                      "Representation (Image)"@en ,
                                      "Répresentation (Image)"@fr ,
                                      "Rappresentazione (Imagine)"@it ;
 
-                          rdfs:comment "A resource that can contain two-dimensional still image files"@en .
+                          rdfs:comment "A resource that can contain a two-dimensional still image file"@en .
 
 
 
@@ -2455,15 +2419,15 @@
                     rdfs:subClassOf :Representation ,
                                     [ rdf:type owl:Restriction ;
                                       owl:onProperty :hasTextFileValue ;
-                                      owl:minCardinality "1"^^xsd:nonNegativeInteger
+                                      owl:cardinality "1"^^xsd:nonNegativeInteger
                                     ] ;
 
                     rdfs:label "Repräsentation (Text)"@de ,
                                                "Representation (Text)"@en ,
                                                "Répresentation (Texte)"@fr ,
                                                "Rappresentazione (testo)"@it ;
 
-                    rdfs:comment "A resource containing text files"@en .
+                    rdfs:comment "A resource containing a text file"@en .
 
 
 ### http://www.knora.org/ontology/knora-base#ForbiddenResource

diff --git a/sipi/scripts/convert_from_binaries.lua b/sipi/scripts/convert_from_binaries.lua
@@ -158,49 +158,11 @@ if mediatype == IMAGE then
         return
     end
 
-    --
-    -- create thumbnail (jpg)
-    --
-    success, thumbImg = SipiImage.new(sourcePath, { size = config.thumb_size })
-    if not success then
-        server.log("SipiImage.new failed: " .. thumbImg, server.loglevel.LOG_ERR)
-        return
-    end
-
-    success, thumbDims = thumbImg:dims()
-    if not success then
-        server.log("thumbImg:dims() failed: " .. thumbDims, server.loglevel.LOG_ERR)
-        return
-    end
-
-    thumbImgName = baseName .. '.jpg'
-
-    --
-    -- create new thumnail image file path with sublevels:
-    --
-    success, newThumbPath = helper.filename_hash(thumbImgName);
-    if not success then
-        server.sendStatus(500)
-        server.log(gaga, server.loglevel.error)
-        return false
-    end
-
-
-    success, errmsg = thumbImg:write(projectDir .. newThumbPath)
-    if not success then
-        server.log("thumbImg:write failed: " .. errmsg, server.loglevel.LOG_ERR)
-        return
-    end
-
     result = {
         mimetype_full = "image/jp2",
         filename_full = fullImgName,
         nx_full = fullDims.nx,
         ny_full = fullDims.ny,
-        mimetype_thumb = "image/jpeg",
-        filename_thumb = thumbImgName,
-        nx_thumb = thumbDims.nx,
-        ny_thumb = thumbDims.ny,
         original_mimetype = originalMimetype,
         original_filename = originalFilename,
         file_type = IMAGE

diff --git a/sipi/scripts/convert_from_file.lua b/sipi/scripts/convert_from_file.lua
@@ -159,27 +159,6 @@ if not success then
     return
 end
 
-
-thumbImgName = baseName .. '.jpg'
-
---
--- create new thumnail image file path with sublevels:
---
-success, newThumbPath = helper.filename_hash(thumbImgName);
-if not success then
-    server.sendStatus(500)
-    server.log("Unable to generate filename hash for " .. thumbImgName, server.loglevel.LOG_ERR)
-    return false
-end
-
-fullThumbPath = projectDir .. newThumbPath
-success, errmsg = thumbImg:write(fullThumbPath)
-if not success then
-    server.log("Unable to write " .. fullThumbPath, server.loglevel.LOG_ERR)
-    return
-end
-
-
 --
 -- delete tmp and preview files
 --
@@ -188,22 +167,13 @@ if not success then
     server.log("server.fs.unlink failed: " .. errmsg, server.loglevel.LOG_ERR)
     return
 end
-success, errmsg = server.fs.unlink(config.imgroot .. '/thumbs/' .. filename .. ".jpg")
-if not success then
-    server.log("server.fs.unlink failed: " .. errmsg, server.loglevel.LOG_ERR)
-    return
-end
 
 result = {
     status = 0,
     mimetype_full = "image/jp2",
     filename_full = fullImgName,
     nx_full = fullDims.nx,
     ny_full = fullDims.ny,
-    mimetype_thumb = "image/jpeg",
-    filename_thumb = thumbImgName,
-    nx_thumb = thumbDims.nx,
-    ny_thumb = thumbDims.ny,
     original_mimetype = originalMimetype,
     original_filename = originalFilename,
     file_type = 'image'