BAA container is getting killed on startup

[sivaraman-g]

Hi Team,
We are observing an issue of a BAA container getting killed on startup. We have checked in the "docker inspect" log and we could see that the baa is exited with error code "137" as below.

"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2021-12-27T12:41:39.547588173Z",
"FinishedAt": "2021-12-27T12:42:01.959250913Z"
},

BAA Logs:

root@root:/home/#docker-compose -f docker-compose-baa.yml up
Creating network "alpha_docker-capg-network" with driver "bridge"
Creating baa ... done
Attaching to baa
baa    | Run command is : exec ./karaf debug
baa    | Listening for transport dt_socket at address: 5005
baa    |         __ __                  ____
baa    |        / //_/____ __________ _/ __/
baa    |       / ,<  / __ `/ ___/ __ `/ /_
baa    |      / /| |/ /_/ / /  / /_/ / __/
baa    |     /_/ |_|\__,_/_/   \__,_/_/
baa    |
baa    |   Apache Karaf (4.0.4)
baa    |
baa    | Hit '<tab>' for a list of available commands
baa    | and '[cmd] --help' for help on a specific command.
baa    | Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown Karaf.
baa    |
baa    | karaf@root()> 'org.apache.felix.eventadmin.impl.adapter.LogEventAdapter$1@6f6e8803' is removed as a LogListener, since it threw an exception.
baa    | Killed

We have done the following things but still faced the same issue.

1. Uninstalled, cleaned up the caches, and re-installed the docker.
2. Increased the memory for the baa container inside the docker-compose file.
3. Checked with the latest broadbandforum/baa:R4.1.1.
4. Installed the latest version of docker and docker-compose.

Also, we have started the BAA docker with /bin/bash as a startup script. and inside the docker container, we have manually tried to bring up the "/baa/baa-dist/bin/karaf". but that process is getting killed. We are suspecting that the java process "/usr/lib/jvm/java-1.8-openjdk/bin/java" which is started by karaf is somehow getting killed.

Note: This docker kill issue is not observed with other docker images ex: control-relay

For reference: Below is the specification of the PC and BAA which we are using.

OS = Ubuntu
Version = 18.04.5 LTS (Bionic Beaver)
ID_LIKE = debian
Kernel Version = 5.4.0-91-generic
Architecture =  x86_64
CPU op-mode(s) =  32-bit, 64-bit
Byte Order = Little Endian
CPU(s) = 6
RAM=  128Gb
Hard Disk = 2TB
Docker Version = 19.03.6
Docker-compose version = 1.24.0
BAA Version : R4.0.0

[selvaitech]

Hi @sivaraman-g, Can you please attach the karaf.log file present in container baa in path /baa/baa-dist/data/log/karaf.log?

We do not get the actual issue by seeing the logs mentioned in the issue?

[sivaraman-g]

Hi Selva,

I have attached the Karaf log.
karaf.log

Thanks,

[selvaitech]

OS - Ubuntu 18.04
Docker version : 19.03.15

I have started the BAA with R4.1.1 on my development environment and I see that BAA can be started without any issues.

Attaching the log for your reference,

[
karaf.log
karaf1.log
]

Can you please check and stop other processes running in your machine before starting BAA?

[sivaraman-g]

Hi Selva,

Can you please let us know which process can affect the BAA container?
So, that we will be stopping those and checking the baa kill issue.

We have attached the running process and the top command output for reference.
process.log
top.log

Thanks

[selvaitech]

Processes I meant, if there is any other docker containers using most of the system memory or so. Can you try to upgrade the docker version to 19.03.15 and re-attempt the docker-compose up command? Ensure that no other docker containers are running before starting the obbaa containers.

[sivaraman-g]

Hi Selva,

We have tried to run baa docker alone with docker version 19.03.15, but still we have observed the same issue.
Attached karaf.log file.
karaf.log

Console LOGS:

root@root:/home/# docker -v
Docker version 19.03.15, build 99e3ed8919
root@root:/home/# docker version
Client: Docker Engine - Community
 Version:           19.03.15
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        99e3ed8919
 Built:             Sat Jan 30 03:16:51 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.15
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       99e3ed8919
  Built:            Sat Jan 30 03:15:20 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
  
  
root@root:/home/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES


root@root:/home/# docker-compose -f docker-compose-baa.yml up
Starting baa ... done
Attaching to baa
baa    | Run command is : exec ./karaf debug
baa    | Listening for transport dt_socket at address: 5005
baa    | The specified feature: 'wrap' version '0.0.0' is already installed
baa    |         __ __                  ____
baa    |        / //_/____ __________ _/ __/
baa    |       / ,<  / __ `/ ___/ __ `/ /_
baa    |      / /| |/ /_/ / /  / /_/ / __/
baa    |     /_/ |_|\__,_/_/   \__,_/_/
baa    |
baa    |   Apache Karaf (4.0.4)
baa    |
baa    | Hit '<tab>' for a list of available commands
baa    | and '[cmd] --help' for help on a specific command.
baa    | Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown Karaf.
baa    |
baa    | karaf@root()> Killed
baa exited with code 137
root@root:/home/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@root:/home/# docker ps -a
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                            PORTS               NAMES
db4c51aad3f5        broadbandforum/baa:R4.0.0   "/bin/sh -c /baa/baa…"   2 hours ago         Exited (137) About a minute ago                       baa
root@root:/home/# docker-compose -v
docker-compose version 1.24.0, build 0aa59064

[selvaitech]

Hi Sivaraman,

Exit Code : 137, seems to be thrown when not enough memory for the Docker to Run container(s). We were able to start BAA setup with 2 vCPUs, 4 GB RAM and 40 GB HDD in 64bit Ubuntu 18.04(for the BBWF 2021 Demo).

Instead of starting R4.1.1 can you please try to use the image docker pull broadbandforum/baa:R4.1.0 and check if you are able to start the BAA container without any issue(s)? I think it must be something to do with the environmental issue.

[sivaraman-g]

Hi Selva,

Thanks for the response.
We have tried with R4.1.0 Version, but still, we are facing the same issue of the baa container getting killed.

[selvaitech]

Hi Sivaraman,

I have gone through the logs which was attached to this ticket earlier. I did not found abnormal logs which could caused the container to exit.

Request you to reproduce the issue on different server and check if you are able to start the BAA?