initial instructions for use by a Test Engineer performing the Staging testing

[jpiesing]

Here is my current draft for my action "(2023-04-19) Mr. Piesing to write initial instructions for use by a Test Engineer
performing the Staging testing. E.g. where to find the information, what tests are to be run, etc. – Ongoing".

Here is my current draft. This issue will be edited - text in italics is editors notes, things to be done and so on.

1. Obtain the following;
   1.1 a camera capable of recording 120 frames / second for minutes at a time without overheating - see https://github.com/cta-wave/device-observation-framework#camera-requirements
   1.2 a PC with Linux on it (windows with WSL is not excluded but requires a significantly more experienced person doing the installation) and with the relevant hardware to read recordings from the camera (e.g. an SD card reader if the camera records onto SD cards)
   1.3 a tripod or similar to hold the camera steady when pointing at the device being tested
2. Find a room to do the tests where the camera can be pointed at the device being tested where there are no reflections on the screen of the device - see https://github.com/cta-wave/device-observation-framework#recording-environment-set-up
3. Install (or have installed for you) various software on the PC as follows;
   3.1 docker, docker-compose and python3. Note that the commercial "Docker Desktop" product is not required. You can obtain the individual pieces, docker and docker-compose separately from the commercial product.
   3.2 download the dpctf-deploy github repo to the PC, either as a .zip file or 'git clone https://github.com/cta-wave/dpctf-deploy.git'
   3.3 follow the instructions on https://www.github.com/cta-wave/dpctf-deploy including the headings below (noting carefully the requirements for root / administrator access);
   3.4 download and install the required libraries for the Observation Framework
   3.5 Download the observation framework
   3.6 Install the observation framework python packages
4. Create and run the docker image on the PC
   4.1 "Create Image" with <commit-id/branch/tag> as "master" and <image-version> as "latest"
   4.2 Ensure that the results directory is owned by 1000.1000 and is writable
   4.3 Obtain the IP address of the PC and add it to the config.json file using host_override
   4.4 "Running the created image in a container" Note that the docker-compose.yml file may already be correct.
5. Download a local copy of the test content including;
   5.1 Agree to EULA
   5.2 Import content
6. Start the container
7. If the browser on the device being tested supports direct URL entry then follow the instructions under "Running tests" and "[Run on separate DUT (TV, Mobile, etc.)] (https://github.com/cta-wave/dpctf-deploy#run-on-separate-dut-tv-mobile-etc)" to so that the device under test is showing the landing page
8. If the device being tested supports HbbTV and does not support direct URL entry then;
   8.1 update the transport stream to include the IP address of the PC where the tests are installed
   8.2 play that transport stream through a modulator into the device being tested and select that service so that the device under test is showing the landing page
9. Note the token from the landing page however you got to that page
10. Continue with the steps under https://github.com/cta-wave/dpctf-deploy#run-on-separate-dut-tv-mobile-etc to enter the token from the landing page into the web page of the test runner
11. Validate that the basic operation of the test runner is correct by running one of the simplest tests without the camera
    11.1. Choose sequential track playback -t1
    11.2
12. Once you can reliably run one test without the camera, run the same test with the camera recording;
    12.1 Point the camera at the device under test
    12.2 Ensure you have a good recording environment - see https://github.com/cta-wave/device-observation-framework#recording-environment-set-up
    12.3 Start the camera recording
    12.4 Run the test the same as you did previously
    12.5 Stop the camera recording
    12.6 Download the recording from the camera to the test runner PC (typically remove an SD card from the camera and plug it into an SD card reader on the PC)
    12.7 Copy the recording file onto the test runner PC
    12.8 Run the observation framework on the recording - see https://github.com/cta-wave/device-observation-framework#how-to-verify-the-camera-setup

The above text resulted in the following issues that can be considered blockers to release - but not to using this at the HbbTV plugfest assuming someone from Fraunhofer is participating to hold people's hands.

#40
#39

The following was raised but is not a blocker.

cta-wave/Test-Content#50

[gitwjr]

I'd include the link to the OF ReadMe in your instructions for "install the OF" and spell out Observation Framework on first use instead of OF: https://github.com/cta-wave/device-observation-framework
The OF instructions are not as clear on running in Linux vs Windows environment. The initial installation paragraph notes that "The Observation Framework must be installed outside of Docker or Docker Desktop and run outside of the docker from the windows command line." That would seem to imply running it under Windows.

[jpiesing]

I'd include the link to the OF ReadMe in your instructions for "install the OF" and spell out Observation Framework on first use instead of OF: https://github.com/cta-wave/device-observation-framework The OF instructions are not as clear on running in Linux vs Windows environment. The initial installation paragraph notes that "The Observation Framework must be installed outside of Docker or Docker Desktop and run outside of the docker from the windows command line." That would seem to imply running it under Windows.

Addressed by later edits.

[louaybassbouss]

Thanks @jpiesing as just discussed in the Testing call, we can put the documentation in the readme of this repo https://github.com/cta-wave/dpc-test-suite (or in the corresponding github.io page https://cta-wave.github.io/dpc-test-suite/). wdyt?

[jpiesing]

Thanks @jpiesing as just discussed in the Testing call, we can put the documentation in the readme of this repo https://github.com/cta-wave/dpc-test-suite (or in the corresponding github.io page https://cta-wave.github.io/dpc-test-suite/). wdyt?

What is that repo for? Is this the repo for staging & the repo that we will communicate to people?

[gitwjr]

@jpiesing @louaybassbouss
Has this been addressed by the latest updates to the Quick Start.md or are there elements that still need to be addressed in the dpctf-deploy and Observation Framework .md instructions/code? dpctf-deploy #39 and #40 are closed and cta-wave/Test-Content#50 has been deferred.

[jpiesing]

@jpiesing @louaybassbouss Has this been addressed by the latest updates to the Quick Start.md or are there elements that still need to be addressed in the dpctf-deploy and Observation Framework .md instructions/code? dpctf-deploy #39 and #40 are closed and cta-wave/Test-Content#50 has been deferred.

I'm going through the Quick Start instructions myself right now.

[jpiesing]

Here are a few other comments from doing a clean install following QUICKSTART on Linux.

1. Consider renaming the current README to something else & renaming QUICKSTART to README.
2. build.sh should check that the permissions are correct to read the docker socket and generate a helpful error message if not, e.g. referring people to the docker page for 'Linux post-install'. If you're not either 1) running build.sh with sudo or 2) added the the user to the docker group then the error message you get is accurate but unhelpful to newbies.
3. The Windows instructions assume you're running without WSL. This means you're using Docker Desktop which is a commercial (paid) product for large companies. If someone in a large company wants to run Docker on Windows for free then they have to use WSL. For WSL, you have to follow the Linux instructions. Please add something at the start about WSL.
   WSL is also complex as the WSL subsystem is behind a NAT and port forwarding has to be setup to forward incoming connections to the host to the WSL subsystem. See https://jwstanly.com/blog/article/Port+Forwarding+WSL+2+to+Your+LAN/ .
4. Do something about the README files in the other repos.

• Rename them?
• Insert a warning at the top pointing people at what's currently called QUICKSTART in this repo unless they really, really know what they're doing?
• Something else?

5. The examples in QUICKSTART.md are confusing with respect to port numbers.
   
   The example of the output generated when the test runner is started has port numbers 8443 + 36161.
   The example URLs has the app running on port 8000.
   In the installation I've just done, port 8000 works and the port number printed when the test runner started does not work so it may be helpful to add a note that the port numbers of 8443 and 36161 are to be ignored.
6. The text relating to TLS server certificates should be moved to the "Host machine requirements" section at the start as this is the section that corporate IT people will see or at least will focus on. Perhaps @mlasak 's text in improve HbbTV.md re domain #43 (comment) could be re-used in part?
7. In the description of phase 2, perhaps add some text suggesting people start with just sequential track playback t1 and get that working before trying more tests. Perhaps add a screen dump showing what things look like in the UI with just that selected?
8. Include a link to "Recording Environment Setup" from the OF README - https://github.com/cta-wave/device-observation-framework#recording-environment-set-up on the end of "position video recording device (e.g. smartphone with 120fps using AVC codec) in front of the display of DUT".

[louaybassbouss]

@FritzHeiden FYI

[gitwjr]

@louaybassbouss @FritzHeiden @jpiesing
I deleted my set up and ran the new Quick Start step by step. After running @ ./build-dof.sh the new container dpctf-dof appears but the "device-observation-framework-main" folder does is not created in folder "dpctf-deploy". Further the entire command finishes in 0.6 or 0.7 seconds.
Here is the run:
brose@Bill-Lenovo-Laptop:/mnt/c/Users/brose/dpctf-deploy$ ./build-dof.sh
[+] Building 0.7s (15/15) FINISHED docker:default
=> [internal] load .dockerignore 0.0s
=> => transferring context: 75B 0.0s
=> [internal] load build definition from Dockerfile.dof 0.0s
=> => transferring dockerfile: 810B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8-bookworm 0.5s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [ 1/10] FROM docker.io/library/python:3.8-bookworm@sha256:22b8bf2be7a50eeb14b01322e580485681017032a720604baab 0.0s
=> CACHED [ 2/10] RUN apt-get update && apt-get install -y git libzbar0 netcat-openbsd libgl1 ffmpeg python3-pya 0.0s
=> CACHED [ 3/10] WORKDIR /usr/app 0.0s
=> CACHED [ 4/10] RUN git clone https://github.com/cta-wave/device-observation-framework.git 0.0s
=> CACHED [ 5/10] WORKDIR /usr/app/device-observation-framework 0.0s
=> CACHED [ 6/10] RUN ./install.sh 0.0s
=> CACHED [ 7/10] RUN cd audio_mezzanine && ./import_audio_mezzanine.sh && cd .. 0.0s
=> CACHED [ 8/10] RUN mkdir /usr/app/recordings 0.0s
=> CACHED [ 9/10] RUN echo "echo "python observation_framework.py --input "/usr/app/recordings/$RECORDING_FILEN 0.0s
=> CACHED [10/10] RUN echo "python observation_framework.py --input "/usr/app/recordings/$RECORDING_FILENAME" "$ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:4623a2e742b0c7187d9f1ec0595a52fd1c3367303e6014a4def2259ad28e1f3d 0.0s
=> => naming to docker.io/library/dpctf-dof:latest

[louaybassbouss]

@gitwjr can you try to clear docker cache https://docs.docker.com/engine/reference/commandline/builder_prune/ and try again

[gitwjr]

@louaybassbouss I deleted the dpctf-dof image and ran docker builder prune --all. I then ran ./build-dof.sh. It took time and listed all of the steps but still did not create the device-observation-framework-main in dpctf-deploy. A search of the entire c-drive shows no such folder exists. Other ideas? I am not building the OF using the Observation Framework README instructions first (i.e. cloning the OF). The Quick Start implies that building the -dof image handles that. Is that correct?

[FritzHeiden]

@gitwjr I am not quiet sure what that directory is you mention. When running the dof docker container using the analyse-recording.sh there are two directories that are created and contain data generated by the dof: The observation-results directory containing the results and the observation-logs containing the logs.

The recent update of the quick start guide only included changes for phase 1. I will improve the observation part (phase 3) very soon to be as elaborate as possible on how to work with the dof docker image and container using the related scripts

[gitwjr]

@FritzHeiden Thanks for the clarification. When reading the Quick Start it led me to believe the OF is included in the -dof files as it says . "Following steps should be sufficient to get started with dockerized version, more details at https://github.com/cta-wave/device-observation-framework" and there is no other mention anywhere in the QS of loading the OF. I assumed the OF was now included in the Quick Start when building the -dof image. I suggest making that more clear e.g. At the beginning of phase 3 instruct the user to how to build the OF following the instructions in the OF README. Would be even better if the instructions were included in the QS guide. The OF README immediately follows cloning the OF with installing Python which I do believe is not needed anymore. I thought the necessary files are in -dof as well as loading the Mezzanine though I'd like confirmation on that.

[jpiesing]

I've been through this a couple of times and addressed the remaining issues with PR #68 and new issue #67.