feat(api-v1): Change API v1 file uploads to work like API v2 (DSP-41, PR 3) (#1722)

Author: Benjamin Geer

docs/03-apis/api-v1/adding-resources.md

@@ -47,67 +47,53 @@ The request header's content type has to be set to `application/json`.
 
 ## Adding Resources with Image Files
 
-Certain resource classes can have attached image files. There are two ways to
-attach a file to a resource: Either by submitting directly the binaries of the file in a
-an HTTP Multipart request, or by indicating the location of the file. The two cases are referred to
-as non-GUI case and GUI case (see [Sipi and Knora](../../07-sipi/sipi-and-knora.md)).
-
-### Including the binaries (non-GUI case)
-
-In order to include the binaries, a HTTP Multipart request has to be
-sent. One part contains the JSON (same format as described for
-[Adding Resources Without Images Files](#adding-resources-without-a-digital-representation))
-and has to be named `json`. The other part contains the file's name, its binaries, and its mime type
-and has to be named `file`. The following example illustrates how to
-make this type of request using Python 3:
-
-```python
-#!/usr/bin/env python3
-
-import requests, json
-
-# a Python dictionary that will be turned into a JSON object
-resourceParams = {
-   'restype_id': 'http://www.knora.org/ontology/test#testType',
-   'properties': {
-       'http://www.knora.org/ontology/test#testtext': [
-           {'richtext_value': {'utf8str': "test"}}
-       ],
-       'http://www.knora.org/ontology/test#testnumber': [
-           {'int_value': 1}
-       ]
-   },
-   'label': "test resource",
-   'project_id': 'http://rdfh.ch/projects/testproject'
-}
-
-# the name of the file to be submitted
-filename = "myimage.jpg"
+The first step is to upload an image file to Sipi, using a
+`multipart/form-data` request, where `sipihost` represents the host and
+port on which Sipi is running:
 
-# a tuple containing the file's name, its binaries and its mimetype
-file = {'file': (filename, open(filename, 'rb'), "image/jpeg")} # use name "file"
+```
+HTTP POST to http://sipihost/upload?token=TOKEN
+```
 
-# do a POST request providing both the JSON and the binaries
-r = requests.post("http://host/v1/resources",
-                  data={'json': json.dumps(resourceParams)}, # use name "json"
-                  files=file,
-                  auth=('user', 'password'))
+The `TOKEN` is the `sid` returned by Knora in response to the
+client's login request (see [Authentication](authentication.md)).
+The request must contain a body part providing the file as well as a parameter
+`filename`, providing the file's original filename, which both Knora and Sipi will
+store; these filenames can be descriptive and need not be unique.
+
+Sipi will then convert the uploaded image file to JPEG 2000 format and store
+it in a temporary location. If this is successful, it will return a JSON
+response that looks something like this:
+
+```json
+{
+  "uploadedFiles": [{
+    "originalFilename": "manuscript-1234-page-1.tiff",
+    "internalFilename": "3UIsXH9bP0j-BV0D4sN51Xz.jp2",
+    "temporaryBaseIIIFUrl": "http://sipihost/tmp"
+  }]
+}
 ```
 
-Please note that the file has to be read in binary mode (by default it
-would be read in text mode).
+This provides:
 
-### Indicating the location of a file (GUI case)
+- the `originalFilename`, which we submitted when uploading the file
+- the unique `internalFilename` that Sipi has randomly generated for the file
+- the `temporaryBaseIIIFUrl`, which we can use to construct a IIIF URL for
+  previewing the file
 
-This request works similarly to
-[Adding Resources Without Image Files](#adding-resources-without-a-digital-representation). The JSON format is described
-in the TypeScript interface `createResourceWithRepresentationRequest` in
-module `createResourceFormats`. The request header's content type has to
-set to `application/json`.
+The client may now wish to get a thumbnail of the uploaded image, to allow
+the user to confirm that the correct files have been uploaded. This can be done
+by adding the filename and IIIF parameters to `temporaryBaseIIIFUrl`. For example, to get
+a JPG thumbnail image whose width and height are at most 128 pixels wide, you would request
+`http://sipihost/tmp/3UIsXH9bP0j-BV0D4sN51Xz.jp2/full/!128,128/0/default.jpg`.
 
-In addition to [Adding Resources Without Image Files](#adding-resources-without-a-digital-representation), the
-(temporary) name of the file, its original name, and mime type have to
-be provided (see [GUI Case](../../07-sipi/sipi-and-knora.md#gui-case)).
+The request to Knora works similarly to
+[Adding Resources Without Image Files](#adding-resources-without-image-files),
+with the addition of `file`, whose value is the `internalFilename` that Sipi returned.
+See the TypeScript interface `createResourceWithRepresentationRequest` in
+module `createResourceFormats` for details. The request header's content type must be
+set to `application/json`.
 
 ## Response to a Resource Creation
 
@@ -122,7 +108,7 @@ The JSON format of the response is described in the TypeScript interface
 ## Changing a Resource's Label
 
 A resource's label can be changed by making a PUT request to the path
-segments `resources/label`. The resource's Iri has to be provided in the
+segments `resources/label`. The resource's IRI has to be provided in the
 URL (as its last segment). The new label has to submitted as JSON in the
 HTTP request's body.
 
@@ -150,18 +136,24 @@ Only system or project administrators may use the bulk import.
 The procedure for using this feature is as follows
 (see the [example below](#bulk-import-example)).
 
-1.  Make an HTTP GET request to Knora to [get XML schemas](#1-get-xml-schemas) describing
-    the XML to be provided for the import.
-2.  [Generate an XML import document](#2-generate-xml-import-document) representing the
-    data to be imported, following the Knora import schemas that were generated in step 1.
-    You will probably want to write a script to do this. Knora is not involved in this step.
-    If you are also importing image files, this XML document needs to
-    [contain the filesystem paths](#bulk-import-with-image-files) of those files.
-3.  [Validate your XML import document](#3-validate-xml-import-document), using an XML schema validator such as
-    [Apache Xerces](http://xerces.apache.org) or [Saxon](http://www.saxonica.com), or an
-    XML development environment such as [Oxygen](https://www.oxygenxml.com). This will
-    help ensure that the data you submit to Knora is correct. Knora is not involved in this step.
-4.  [Submit the XML import document to Knora](#4-submit-xml-import-document-to-knora).
+1. Make an HTTP GET request to Knora to [get XML schemas](#1-get-xml-schemas) describing
+   the XML to be provided for the import.
+    
+2. If you are importing image files, [upload files to Sipi](#2-upload-files-to-sipi).
+
+3. [Generate an XML import document](#3-generate-xml-import-document) representing the
+   data to be imported, following the Knora import schemas that were generated in step 1.
+   You will probably want to write a script to do this. Knora is not involved in this step.
+   If you are also importing image files, this XML document needs to
+   [contain the filenames](#bulk-import-with-image-files) that Sipi returned
+   for the files you uploaded in step 2.
+
+4. [Validate your XML import document](#4-validate-xml-import-document), using an XML schema validator such as
+   [Apache Xerces](http://xerces.apache.org) or [Saxon](http://www.saxonica.com), or an
+   XML development environment such as [Oxygen](https://www.oxygenxml.com). This will
+   help ensure that the data you submit to Knora is correct. Knora is not involved in this step.
+
+5. [Submit the XML import document to Knora](#5-submit-xml-import-document-to-knora).
 
 In this procedure, the person responsible for generating the XML import
 data need not be familiar with RDF or with the ontologies involved.
@@ -206,7 +198,12 @@ containing three files:
 
 - `knoraXmlImport.xsd`: The standard Knora XML import schema, used by all XML imports.
 
-#### 2. Generate XML Import Document
+#### 2. Upload Files to Sipi
+
+See [Upload Files to Sipi](../api-v2/editing-values.md#upload-files-to-sipi) in
+the Knora API v2 documentation.
+
+#### 3. Generate XML Import Document
 
 We now convert our existing data to XML, probably by writing a custom
 script. The resulting XML import document could look like this:
@@ -305,18 +302,6 @@ This illustrates several aspects of XML imports:
 
   - The type of each value must be specified using the attribute
     `knoraType`.
-    
-  - If a property has `knoraType="date_value"`, the date value must have the following format: 
-  
-    ```
-    (GREGORIAN|JULIAN|ISLAMIC):\d{1,4}(-\d{1,2}(-\d{1,2})?)?( BC| AD| BCE| CE)?(:\d{1,4}(-\d{1,2}(-\d{1,2})?)?( BC| AD| BCE| CE)?)?
-    ```
-    
-    E.g. an exact date like `GREGORIAN:2015-12-03` or a period like `GREGORIAN:2015-12-03:2015-12-04`.
-    Dates may also have month or year precision, e.g. `ISLAMIC:1407-02` (the whole month of december) or `JULIAN:1330` 
-    (the whole year 1330). An optional ERA indicator term (`BCE`, `CE`, or `BC`, `AD`) can be added to the date, when no 
-    era is provided the default era `AD` will be considered. Era can be given as `GREGORIAN:1220 BC` or in range as 
-    `GREGORIAN:600 BC:480 BC`.
 
   - A link to another resource described in the XML import is
     represented as a child element of a property element, with
@@ -351,7 +336,7 @@ This illustrates several aspects of XML imports:
   - A text value can have a `lang` attribute, whose value is an ISO 639-1
     code specifying the language of the text.
 
-#### 3. Validate XML Import Document
+#### 4. Validate XML Import Document
 
 You can use an XML schema validator such as [Apache Xerces](http://xerces.apache.org) or
 [Saxon](http://saxon.sourceforge.net/), or an XML development environment
@@ -364,7 +349,7 @@ For example, using Saxon:
 java -cp ./saxon9ee.jar com.saxonica.Validate -xsd:p0801-biblio.xsd -s:data.xml
 ```
 
-#### 4. Submit XML Import Document to Knora
+#### 5. Submit XML Import Document to Knora
 
 To create these resources in Knora, make an HTTP post request with the XML import document
 as the request body. The URL must specify the (URL-encoded) IRI of the project in which
@@ -421,8 +406,8 @@ contains the IRI of the target resource.
 
 To attach an image file to a resource, we must provide the
 element `knoraXmlImport:file` before the property elements. In this
-element, we must give the absolute filesystem path to the file that
-should be attached to the resource, along with its MIME type:
+element, we must provide a `filename` attribute, containing the `internalFilename`
+that Sipi returned for the file in [2. Upload Files to Sipi](#2-upload-files-to-sipi).
 
 ```xml
 <?xml version="1.0" encoding="UTF-8"?>
@@ -437,7 +422,7 @@ should be attached to the resource, along with its MIME type:
     </incunabula:book>
     <incunabula:page id="test_page">
         <knoraXmlImport:label>a page with an image</knoraXmlImport:label>
-        <knoraXmlImport:file path="/usr/local/share/import-images/incunabula/12345.tiff" mimetype="image/tiff"/>
+        <knoraXmlImport:file filename="67SEfNU1wK2-CSf5abe2eh3.jp2"/>
         <incunabula:origname knoraType="richtext_value">Chlaus</incunabula:origname>
         <incunabula:pagenum knoraType="richtext_value">1a</incunabula:pagenum>
         <incunabula:partOf>
@@ -448,6 +433,5 @@ should be attached to the resource, along with its MIME type:
 </knoraXmlImport:resources>
 ```
 
-During the processing of the bulk import, Knora will
-communicate the location of file to Sipi, which will convert it to JPEG 2000
-for storage.
+During the processing of the bulk import, Knora will ask Sipi for the rest of the
+file's metadata, and store that metadata in a file value attached to the resource.

docs/03-apis/api-v1/changing-values.md

@@ -56,52 +56,21 @@ has to be used in order to create a new value (all these TypeScript interfaces a
 
 ## Modifying a File Value
 
-In order to exchange a file value (digital representation of a
-resource), the path segment `filevalue` has to be used. The IRI of the
-resource whose file value is to be exchanged has to be appended:
+To change a file value, the client first uploads the new file to
+Sipi, following the procedure described in
+[Adding Resources with Image Files](adding-resources.md#adding-resources-with-image-files).
+
+Then the client sends a request to Knora, using this following route:
 
 ```
 HTTP PUT to http://host/filevalue/resourceIRI
 ```
 
-Please note that the resource IRI has to be URL encoded.
-
-There are two ways to change a file of a resource: Either by submitting
-directly the binaries of the file in a HTTP Multipart request or by
-indicating the location of the file. The two cases are referred to as
-non-GUI case and GUI case (TODO: add a link to "Sipi and Knora").
-
-### Including the binaries (non-GUI case)
-
-Here, a HTTP MULTIPART request has to be made simply providing the
-binaries (without JSON):
-
-```python
-#!/usr/bin/env python3
-
-import requests, json, urllib
-
-# the name of the file to be submitted
-filename = 'myimage.tif'
-
-# a tuple containing the file's name, its binaries and its mimetype
-files = {'file': (filename, open(filename, 'rb'), "image/tiff")}
-
-resIri = urllib.parse.quote_plus('http://rdfh.ch/xy')
-
-r = requests.put("http://host/filevalue/" + resIri,
-                 files=files)
-```
-
-Please note that the file has to be read in binary mode (by default it
-would be read in text mode).
-
-### Indicating the location of a file (GUI case)
-
-Here, simply the location of the new file has to be submitted as JSON.
-The JSON format is described in the TypeScript interface
-`changeFileValueRequest` in module `changeValueFormats`. The request
-header's content type has to set to `application/json`.
+Here, `resourceIRI` is the URL-encoded IRI of the resource whose file value is
+to be changed. The body of the request is a JSON object described in the TypeScript
+interface `changeFileValueRequest` in module `changeValueFormats`, and contains
+`file`, whose value is the `internalFilename` that Sipi returned. The request header's
+content type must be set to `application/json`.
 
 ## Response on Value Change
 

docs/03-apis/api-v2/editing-values.md

@@ -262,11 +262,11 @@ Sipi then returns a JSON response that looks something like this:
   "uploadedFiles": [{
     "originalFilename": "manuscript-1234-page-1.tiff",
     "internalFilename": "3UIsXH9bP0j-BV0D4sN51Xz.jp2",
-    "temporaryBaseIIIFUrl": "http://sipihost/tmp/3UIsXH9bP0j-BV0D4sN51Xz.jp2"
+    "temporaryBaseIIIFUrl": "http://sipihost/tmp"
   }, {
     "originalFilename": "manuscript-1234-page-2.tiff",
     "internalFilename": "2RvJgguglpe-B45EOk0Gx8H.jp2",
-    "temporaryBaseIIIFUrl": "http://sipihost/tmp/2RvJgguglpe-B45EOk0Gx8H.jp2"
+    "temporaryBaseIIIFUrl": "http://sipihost/tmp"
   }]
 }
 ```

docs/05-internals/design/api-v2/sipi.md

@@ -41,7 +41,7 @@ are described below.
 The `upload.lua` script is available at Sipi's `upload` route. It processes one
 or more file uploads submitted to Sipi. It converts uploaded images to JPEG 2000
 format, and stores them in Sipi's `tmp` directory. The usage of this script is described in
-[Creating File Values](../../../03-apis/api-v2/editing-values.md#creating-file-values).
+[Upload Files to Sipi](../../../03-apis/api-v2/editing-values.md#upload-files-to-sipi).
 
 Each time `upload.lua` processes a request, it also deletes old temporary files
 from `tmp` and (recursively) from any subdirectories. The maximum allowed age of
@@ -94,7 +94,7 @@ If it encounters an error, it returns `SipiException`.
    to create or change a file value. The request includes Sipi's internal filename.
 3. During parsing of this JSON-LD request, a `StillImageFileValueContentV2`
    is constructed to represent the file value. During the construction of this
-   object, a `GetImageMetadataRequestV2` is sent to `SipiConnector`, which
+   object, a `GetFileMetadataRequestV2` is sent to `SipiConnector`, which
    uses Sipi's built-in `knora.json` route to get the rest of the file's
    metadata.
 4. A responder (`ResourcesResponderV2` or `ValuesResponderV2`) validates

docs/07-sipi/setup-sipi-for-knora.md

@@ -45,11 +45,10 @@ Whenever a file is requested from Sipi (e.g. a browser trying to
 dereference an image link served by Knora), a preflight function is
 called. This function is defined in `sipi.init-knora.lua` present in the
 Sipi root directory. It takes three parameters: `prefix`, `identifier`
-(the name of the requested file), and `cookie`. File links created by
-Knora use the prefix `knora`, e.g.
-`http://localhost:1024/knora/incunabula_0000000002.jp2/full/2613,3505/0/default.jpg`.
+(the name of the requested file), and `cookie`. The prefix is the shortcode
+of the project that the resource containing the file value belongs to.
 
-Given these information, Sipi asks Knora about the current's users
+Given this information, Sipi asks Knora about the current's users
 permissions on the given file. The cookie contains the current user's
 Knora session id, so Knora can match Sipi's request with a given user
 profile and determine the permissions this user has on the file. If the
@@ -60,8 +59,8 @@ refuses to serve the file. However, all of this behaviour is defined in
 the preflight function in Sipi and not controlled by Knora. Knora only
 provides the permission code.
 
-See [Sharing the Session ID with Sipi](sipi-and-knora.md#sharing-the-session-id-with-sipi) for more
-information about sharing the session id.
+See [Authentication of Users with Sipi](sipi-and-knora.md#authentication-of-users-with-sipi) for more
+information about sharing the session ID.
 
 ## Using Sipi in Test Mode
 
@@ -76,42 +75,11 @@ $ docker run --rm -it --add-host webapihost:$DOCKERHOST -v $PWD/config:/sipi/con
 ```
 
 Then always the same test file will be served which is included in Sipi. In test mode, Sipi will
-not aks Knora about the user's permission on the requested file.
-
-## Using Sipi in production behind a proxy
-
-For SIPI to work with Salsah1 (non-angular) GUI, we need to define an additional set of
-environment variables if we want to run SIPI behind a proxy:
-
-- `SIPI_EXTERNAL_PROTOCOL=https`
-- `SIPI_EXTERNAL_HOSTNAME=iiif.example.org`
-- `SIPI_EXTERNAL_PORT=443`
-
-These variables are only used by `make_thumbnail.lua`:
-
-```lua
-    server.log("make_thumbnail - external_protocol: " .. get_external_protocol(), server.loglevel.LOG_DEBUG)
-
-    server.log("make_thumbnail - external_hostname: " .. get_external_hostname(), server.loglevel.LOG_DEBUG)
-
-    server.log("make_thumbnail - external_port: " .. get_external_port(), server.loglevel.LOG_DEBUG)
-
-    answer = {
-        nx_thumb = dims.nx,
-        ny_thumb = dims.ny,
-        mimetype_thumb = 'image/jpeg',
-        preview_path =  get_external_protocol() .. "://" .. get_external_hostname() .. ":" .. get_external_port() .."/thumbs/" .. thumbName .. "/full/max/0/default.jpg",
-        filename = tmpName, -- make this a IIIF URL
-        original_mimetype = submitted_mimetype.mimetype,
-        original_filename = filename,
-        file_type = 'IMAGE'
-    }
-```
-
+not ask Knora about the user's permission on the requested file.
 
 ## Additional Sipi Environment Variables
 
-Additionaly, these environment variables can be used to further configure sipi:
+Additionally, these environment variables can be used to further configure Sipi:
 
 - `SIPI_WEBAPI_HOSTNAME=localhost`: overrides `knora_path` in Sipi's config
 - `SIPI_WEBAPI_PORT=3333`: overrides `knora_port` in Sipi's config