Skip to content

Latest commit

 

History

History
745 lines (549 loc) · 16.8 KB

README.rst

File metadata and controls

745 lines (549 loc) · 16.8 KB

CDS

This is the CERN Document Server source code overlay.

Powered by Invenio

CDS is a small layer on top of Invenio, a ​free software suite enabling you to run your own ​digital library or document repository on the web.

Table of Contents

Prerequisites

Ensure that the following dependencies are installed with the specified versions:

  1. Python 3.9
  2. Node.js v18
  3. FFmpeg v5.0
  4. Docker v2 or later: If Docker is not already installed, download and install Docker Desktop from the official Docker website.

Update dependencies

To update Python dependencies you need to run npm install in the target deployment environment:

$ docker run -it --platform="linux/amd64" --rm -v $(pwd):/app -w /app \
    registry.cern.ch/inveniosoftware/almalinux:1 \
    sh -c "dnf install -y openldap-devel && pip install -e . && pip freeze > requirements.new.txt"

Installation and Setup

1. Clone the Repository

Begin by cloning this repository and navigating to the project directory:

git clone https://github.com/CERNDocumentServer/cds-videos.git
cd cds-videos

2. Start Docker

Use Docker Compose to start the required containers in detached mode:

docker compose up -d

3. Run Setup Scripts

The scripts folder contains the necessary setup scripts to initialize and configure your instance.

1. Bootstrap Script

Initialize the environment by running the bootstrap script:

./scripts/bootstrap

Troubleshooting:

These are the macOS solutions using brew for installation.

If you encounter the error pg_config executable not found, you may need to install PostgreSQL and update the PATH:

brew install postgresql@14
export PATH=$PATH:/opt/homebrew/opt/postgresql@14/bin

For errors related to missing cmake and ninja tools ERROR: Command errored out with exit status 1 ... "cmake>=3.14" "ninja>=1.5":

Install cmake and ninja with the following command:

brew install cmake ninja

If you encounter errors with cryptography and OpenSSL, ensure that OpenSSL version 3 is installed:

brew install openssl@3
2. Setup Script

Run the setup script to finalize the installation and configuration:

./scripts/setup

Troubleshooting: If you encounter the error connection to server at "localhost", port 5432 failed: FATAL: role ".." does not exist, it may indicate an issue with the database role or a port conflict. To diagnose:

  1. First, connect to the Docker database container and verify that the expected role exists and the database is working correctly.

    docker exec -it <db_container_name> psql -U <username> -d <database>
  2. If the role is present and the database is functional, check for port conflicts on port 5432:

    lsof -i :5432
Terminate any conflicting process if found, and restart Docker.

4. Local Development

To facilitate local development, open multiple terminal sessions and run the following commands separately:

  • Start Web Server This command launches the web server:

    ./scripts/server
  • Start Celery Workers Celery workers are required for background task processing:

    ./scripts/celery
  • Watch Frontend Code This command watches frontend code for changes and rebuilds assets as needed:

    ./scripts/assets-watch

Testing

Running the tests are as simple as:

python setup.py test

or (to also show test coverage)

source run-tests.sh

Publish Video through REST API

Generate a Personal Access Token

  • Navigate to the CDS Videos platform.
  • Click on your user info in the top-right corner.
  • Go to Applications and create a new Personal Access Token.
  • Copy the token and store it securely.

Using Bruno

If you'd like to use the pre-configured REST API collection in Bruno, ensure you have the application installed. Follow the steps below to set up and use the collection:

  1. Install Bruno:

    Visit the official Bruno documentation or repository and install the application.

  2. Import the Collection:

    • Download this Bruno collection.
    • Open Bruno and import downloaded collection.
    • Create an environment for the collection.
    • Configure the environment by adding a variable named baseURl. Set its value to your API base URL (e.g., http://localhost:5000).
  3. Configure Authentication in Bruno:

    • In Bruno, open the Collection Settings.
    • Go to Auth and set the Bearer Token to your Personal Access Token.

Step 1: Create a Project

Request:

POST {{baseURL}}/api/deposits/project/

Headers:

  • content-type: application/vnd.project.partial+json

Parameters:

Name Type Location Description Required/Optional
$schema string body Schema URL for the project creation. Required
category string body Category of the project. Required
type string body Type of the project. Required
_access json body Access options for the project. Optional
contributors array<object> body List of contributors, including their details. Optional
description string body Description of the project. Optional
title json body Title of the project. Optional
keywords list<json> body Keywords related to the project. Optional

Body:

To restrict the project, add _access/read:

{
   "$schema": "https://localhost:5000/schemas/deposits/records/videos/project/project-v1.0.0.json",
   "_access": {
         "update": [
         "[email protected]",
         "[email protected]"
      ],
      "read": [
            "[email protected]"
      ]
   },
   "category": "ATLAS",
   "type": "VIDEO",
   "contributors": [
         {
            "name": "Surname, Name",
            "ids": [
                  {
                     "value": "cern id",
                     "source": "cern"
                  }
            ],
            "email": "[email protected]",
            "role": "Co-Producer"
         }
      ],
   "title":
      {
      "title":"project title"
      },
   "keywords":[
      {
            "name": "keyword",
            "value": {
               "name": "keyword"
            }
      },
      {
            "name": "keyword2",
            "value": {
               "name": "keyword2"
            }
      }
      ],
   "description": "Description"
}

Response:

Created project JSON.

Step 2: Create a Video

Request:

POST {{baseURL}}/api/deposits/video/

Headers:

  • content-type: application/vnd.video.partial+json

Parameters:

Name Type Location Description Required/Optional
$schema string body Schema URL for video creation. Required
_project_id string body ID of the project. Required
title string body Title of the video. Required
_access json body Access details for the video. Optional
vr boolean body Indicates if the video is 360. Optional
contributors array<object> body List of contributors, including their details. Required
description string body Description of the video. Required
date string (date) body Date in YYYY-MM-DD format. Required
language string body Language of the video. Optional
featured boolean body Whether the video is featured. (Available for members of VIDEOS_EOS_PATH_EGROUPS) Optional
keywords list<json> body Keywords related to the video. Optional
related_links list<json> body Links related to the video. Optional

Body:

To restrict the video, add _access/read. The _access/update will be the same as the project:

{
   "$schema":"https://localhost:5000/schemas/deposits/records/videos/video/video-v1.0.0.json",
   "_project_id":"{{project_id}}",
   "title":
      {
         "title":"217490_medium"
      },
   "_access": {
      "read": [
            "[email protected]"
      ]
   },
   "vr": false,
   "featured": false,
   "language": "en",
   "contributors": [
         {
            "name": "Surname, Name",
            "ids": [
               {
                     "value": "cern id",
                     "source": "cern"
               }
            ],
            "email": "[email protected]",
            "role": "Co-Producer"
         }
   ],
   "description": "Description",
   "date": "2024-11-12",
   "keywords":[
      {
         "name": "keyword",
         "value": {
               "name": "keyword"
         }
      },
      {
         "name": "keyword2",
         "value": {
               "name": "keyword2"
         }
      }
   ],
   "related_links":[
      {
         "name": "related link",
         "url": "https://relatedlink"
      }
   ]
}

Response:

Created video JSON.

Step 3: Upload the Video

Request:

PUT {{baseURL}}/api/files/{{bucket_id}}/{{video_name}}

Headers:

  • content-type: video/mp4
  • Accept: application/json, text/plain, */*
  • Accept-Encoding: gzip, deflate, br, zstd

Parameters:

Name Type Location Description
bucket_id string path Bucket ID.
video_name string path Name of the video file.
file object body Video file.

Response:

Uploaded video JSON.

Step 4: Create a Flow

Request:

POST /api/flows/

Headers:

  • content-type: application/vnd.project.partial+json

Parameters:

Name Type Location Description
version_id string body Version ID from the uploaded video response.
key string body Video key from the uploaded video response.
bucket_id string body Bucket ID from the Create Video response.
deposit_id string body Deposit ID from the Create Video response.

Body:

{
  "version_id": "{{main_file_version_id}}",
  "key": "{{video_key}}",
  "bucket_id": "{{bucket_id}}",
  "deposit_id": "{{video_id}}"
}

Response:

Created flow JSON.

Step 5: (Optional) Upload Additional File

Request:

PUT {{baseURL}}/api/files/{{bucket_id}}/{{additional_file}}

Parameters:

Name Type Location Description
bucket_id string path ID of the bucket to upload the file.
file_name string path Name of the file.
file file body The file to be uploaded.

Response:

Uploaded additional file JSON.

Optional: Update the Access of the Video

Request:

PUT {{baseURL}}/api/deposits/video/{{video_id}}

Headers:

  • content-type: application/vnd.video.partial+json

Parameters:

Name Type Location Description
video_id string path ID of the video.

Body:

To restrict the video, add _access/read. If you want to change the access/update permissions, replace the email addresses in the update field accordingly.

{
 "_access": {
    "update": [
      "[email protected]",
      "[email protected]"
    ],
    "read": [
          "[email protected]"
    ]
 }
}

Response:

Updated video JSON.

Step 6: Get Project to Check the Flow Status

Request:

GET {{baseURL}}/api/deposits/project/{{project_id}}

Headers:

  • content-type: application/vnd.project.partial+json

Parameters:

Name Type Location Description
project_id string path ID of the project.

Response:

Updated project JSON with flow status as state:

{
  "id": "b320568fc1264dda90a8f459be42892e",
  "_cds": {
    "state": {
      "file_transcode": "STARTED",
      "file_video_extract_frames": "SUCCESS",
      "file_video_metadata_extraction": "SUCCESS"
    }
  }
}

Step 7: Publish Video

Before publishing the video, ensure that the workflow is complete.

Request:

POST {{baseURL}}/api/deposits/video/{{video_id}}/actions/publish

Headers:

  • content-type: application/json

Parameters:

Name Type Location Description
video_id string path ID of the video to publish.

Response:

Published video deposit JSON.

License

Copyright (C) 2013-2024 CERN.

CDS is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

CDS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CDS; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

In applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.