CDS

This is the CERN Document Server source code overlay.

Powered by Invenio

CDS is a small layer on top of Invenio, a ​free software suite enabling you to run your own ​digital library or document repository on the web.

Table of Contents

• Prerequisites
• Update Dependencies

• Installation and Setup

  • 1. Clone the Repository
  • 2. Start Docker
  • 3. Run Setup Scripts
  • 4. Local Development

• Testing

• Publish Video through REST API

  • Generate a Personal Access Token
  • Step 1: Create a Project
  • Step 2: Create a Video
  • Step 3: Upload the Video
  • Step 4: Create a Flow
  • Step 5: (Optional) Upload Additional File
  • Optional: Update the Access of the Video
  • Step 6: Get Project to Check the Flow Status
  • Step 7: Publish Video

• License

Prerequisites

Ensure that the following dependencies are installed with the specified versions:

1. Python 3.9
2. Node.js v18
3. FFmpeg v5.0
4. Docker v2 or later: If Docker is not already installed, download and install Docker Desktop from the official Docker website.

Update dependencies

To update Python dependencies you need to run npm install in the target deployment environment:

$ docker run -it --platform="linux/amd64" --rm -v $(pwd):/app -w /app \
    registry.cern.ch/inveniosoftware/almalinux:1 \
    sh -c "dnf install -y openldap-devel && pip install -e . && pip freeze > requirements.new.txt"

Installation and Setup

1. Clone the Repository

Begin by cloning this repository and navigating to the project directory:

git clone https://github.com/CERNDocumentServer/cds-videos.git
cd cds-videos

2. Start Docker

Use Docker Compose to start the required containers in detached mode:

docker compose up -d

3. Run Setup Scripts

The scripts folder contains the necessary setup scripts to initialize and configure your instance.

1. Bootstrap Script

Initialize the environment by running the bootstrap script:

./scripts/bootstrap

Troubleshooting:

These are the macOS solutions using brew for installation.

If you encounter the error pg_config executable not found, you may need to install PostgreSQL and update the PATH:

brew install postgresql@14
export PATH=$PATH:/opt/homebrew/opt/postgresql@14/bin

For errors related to missing cmake and ninja tools ERROR: Command errored out with exit status 1 ... "cmake>=3.14" "ninja>=1.5":

Install cmake and ninja with the following command:

brew install cmake ninja

If you encounter errors with cryptography and OpenSSL, ensure that OpenSSL version 3 is installed:

brew install openssl@3

2. Setup Script

Run the setup script to finalize the installation and configuration:

./scripts/setup

Troubleshooting: If you encounter the error connection to server at "localhost", port 5432 failed: FATAL: role ".." does not exist, it may indicate an issue with the database role or a port conflict. To diagnose:

1. First, connect to the Docker database container and verify that the expected role exists and the database is working correctly.

   docker exec -it <db_container_name> psql -U <username> -d <database>

2. If the role is present and the database is functional, check for port conflicts on port 5432:

   lsof -i :5432

Terminate any conflicting process if found, and restart Docker.

4. Local Development

To facilitate local development, open multiple terminal sessions and run the following commands separately:

• Start Web Server This command launches the web server:

  ./scripts/server

• Start Celery Workers Celery workers are required for background task processing:

  ./scripts/celery

• Watch Frontend Code This command watches frontend code for changes and rebuilds assets as needed:

  ./scripts/assets-watch

Testing

Running the tests are as simple as:

python setup.py test

or (to also show test coverage)

source run-tests.sh

Publish Video through REST API

Generate a Personal Access Token

• Navigate to the CDS Videos platform.
• Click on your user info in the top-right corner.
• Go to Applications and create a new Personal Access Token.
• Copy the token and store it securely.

Using Bruno

If you'd like to use the pre-configured REST API collection in Bruno, ensure you have the application installed. Follow the steps below to set up and use the collection:

1. Install Bruno:

   Visit the official Bruno documentation or repository and install the application.

2. Import the Collection:

   • Download this Bruno collection.
   • Open Bruno and import downloaded collection.
   • Create an environment for the collection.
   • Configure the environment by adding a variable named baseURl. Set its value to your API base URL (e.g., http://localhost:5000).

3. Configure Authentication in Bruno:

   • In Bruno, open the Collection Settings.
   • Go to Auth and set the Bearer Token to your Personal Access Token.

Step 1: Create a Project

Request:

POST {{baseURL}}/api/deposits/project/

Headers:

• content-type: application/vnd.project.partial+json

Parameters:

Name	Type	Location	Description	Required/Optional
$schema	string	body	Schema URL for the project creation.	Required
category	string	body	Category of the project.	Required
type	string	body	Type of the project.	Required
_access	json	body	Access options for the project.	Optional
contributors	array<object>	body	List of contributors, including their details.	Optional
description	string	body	Description of the project.	Optional
title	json	body	Title of the project.	Optional
keywords	list<json>	body	Keywords related to the project.	Optional

Body:

To restrict the project, add _access/read:

{
   "$schema": "https://localhost:5000/schemas/deposits/records/videos/project/project-v1.0.0.json",
   "_access": {
         "update": [
         "[email protected]",
         "[email protected]"
      ],
      "read": [
            "[email protected]"
      ]
   },
   "category": "ATLAS",
   "type": "VIDEO",
   "contributors": [
         {
            "name": "Surname, Name",
            "ids": [
                  {
                     "value": "cern id",
                     "source": "cern"
                  }
            ],
            "email": "[email protected]",
            "role": "Co-Producer"
         }
      ],
   "title":
      {
      "title":"project title"
      },
   "keywords":[
      {
            "name": "keyword",
            "value": {
               "name": "keyword"
            }
      },
      {
            "name": "keyword2",
            "value": {
               "name": "keyword2"
            }
      }
      ],
   "description": "Description"
}

Response:

Created project JSON.

Step 2: Create a Video

Request:

POST {{baseURL}}/api/deposits/video/

Headers:

• content-type: application/vnd.video.partial+json

Parameters:

Name	Type	Location	Description	Required/Optional
$schema	string	body	Schema URL for video creation.	Required
_project_id	string	body	ID of the project.	Required
title	string	body	Title of the video.	Required
_access	json	body	Access details for the video.	Optional
vr	boolean	body	Indicates if the video is 360.	Optional
contributors	array<object>	body	List of contributors, including their details.	Required
description	string	body	Description of the video.	Required
date	string (date)	body	Date in YYYY-MM-DD format.	Required
language	string	body	Language of the video.	Optional
featured	boolean	body	Whether the video is featured. (Available for members of VIDEOS_EOS_PATH_EGROUPS)	Optional
keywords	list<json>	body	Keywords related to the video.	Optional
related_links	list<json>	body	Links related to the video.	Optional

Body:

To restrict the video, add _access/read. The _access/update will be the same as the project:

{
   "$schema":"https://localhost:5000/schemas/deposits/records/videos/video/video-v1.0.0.json",
   "_project_id":"{{project_id}}",
   "title":
      {
         "title":"217490_medium"
      },
   "_access": {
      "read": [
            "[email protected]"
      ]
   },
   "vr": false,
   "featured": false,
   "language": "en",
   "contributors": [
         {
            "name": "Surname, Name",
            "ids": [
               {
                     "value": "cern id",
                     "source": "cern"
               }
            ],
            "email": "[email protected]",
            "role": "Co-Producer"
         }
   ],
   "description": "Description",
   "date": "2024-11-12",
   "keywords":[
      {
         "name": "keyword",
         "value": {
               "name": "keyword"
         }
      },
      {
         "name": "keyword2",
         "value": {
               "name": "keyword2"
         }
      }
   ],
   "related_links":[
      {
         "name": "related link",
         "url": "https://relatedlink"
      }
   ]
}

Response:

Created video JSON.

Step 3: Upload the Video

Request:

PUT {{baseURL}}/api/files/{{bucket_id}}/{{video_name}}

Headers:

• content-type: video/mp4
• Accept: application/json, text/plain, */*
• Accept-Encoding: gzip, deflate, br, zstd

Parameters:

Name	Type	Location	Description
bucket_id	string	path	Bucket ID.
video_name	string	path	Name of the video file.
file	object	body	Video file.

Response:

Uploaded video JSON.

Step 4: Create a Flow

Request:

POST /api/flows/

Headers:

• content-type: application/vnd.project.partial+json

Parameters:

Name	Type	Location	Description
version_id	string	body	Version ID from the uploaded video response.
key	string	body	Video key from the uploaded video response.
bucket_id	string	body	Bucket ID from the Create Video response.
deposit_id	string	body	Deposit ID from the Create Video response.

Body:

{
  "version_id": "{{main_file_version_id}}",
  "key": "{{video_key}}",
  "bucket_id": "{{bucket_id}}",
  "deposit_id": "{{video_id}}"
}

Response:

Created flow JSON.

Step 5: (Optional) Upload Additional File

Request:

PUT {{baseURL}}/api/files/{{bucket_id}}/{{additional_file}}

Parameters:

Name	Type	Location	Description
bucket_id	string	path	ID of the bucket to upload the file.
file_name	string	path	Name of the file.
file	file	body	The file to be uploaded.

Response:

Uploaded additional file JSON.

Optional: Update the Access of the Video

Request:

PUT {{baseURL}}/api/deposits/video/{{video_id}}

Headers:

• content-type: application/vnd.video.partial+json

Parameters:

Name	Type	Location	Description
video_id	string	path	ID of the video.

Body:

To restrict the video, add _access/read. If you want to change the access/update permissions, replace the email addresses in the update field accordingly.

{
 "_access": {
    "update": [
      "[email protected]",
      "[email protected]"
    ],
    "read": [
          "[email protected]"
    ]
 }
}

Response:

Updated video JSON.

Step 6: Get Project to Check the Flow Status

Request:

GET {{baseURL}}/api/deposits/project/{{project_id}}

Headers:

• content-type: application/vnd.project.partial+json

Parameters:

Name	Type	Location	Description
project_id	string	path	ID of the project.

Response:

Updated project JSON with flow status as state:

{
  "id": "b320568fc1264dda90a8f459be42892e",
  "_cds": {
    "state": {
      "file_transcode": "STARTED",
      "file_video_extract_frames": "SUCCESS",
      "file_video_metadata_extraction": "SUCCESS"
    }
  }
}

Step 7: Publish Video

Before publishing the video, ensure that the workflow is complete.

Request:

POST {{baseURL}}/api/deposits/video/{{video_id}}/actions/publish

Headers:

• content-type: application/json

Parameters:

Name	Type	Location	Description
video_id	string	path	ID of the video to publish.

Response:

Published video deposit JSON.

License

Copyright (C) 2013-2024 CERN.

CDS is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

CDS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CDS; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

In applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.