Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues when deploying collector as a sidecar in AWS ECS Fargate #118

Open
frossi85 opened this issue Dec 20, 2023 · 1 comment
Open

Issues when deploying collector as a sidecar in AWS ECS Fargate #118

frossi85 opened this issue Dec 20, 2023 · 1 comment

Comments

@frossi85
Copy link

I am trying to run the collector as a sidecar of a main service using AWS ECS Fargate, whch works correctly when deploying it without the collectot, but I am not able to do it due to some errors that are not clear to me.

I created a custom Dockerfile:

FROM sumologic/collector
ADD file-sumo-sources.json /etc/sumo-sources.json

file-sumo-sources.json:

{
  "api.version": "v1",
  "sources": [
    {
      "sourceType" : "LocalFile",
      "name": "my-service-local-log",
      "pathExpression": "/var/lib/docker/containers/**/*.log",
      "multilineProcessingEnabled": false,
      "automaticDateParsing": true,
      "forceTimeZone": false,
      "category": "file-docker"
    },
    {
        "name": "my-service-local-stats",
        "category": "docker",
        "automaticDateParsing": true,
        "forceTimeZone": false,
        "cutoffTimestamp": 0,
        "uri": "unix:///var/run/docker.sock",
        "specifiedContainers": ["my-service", "logging-service"],
        "allContainers": false,
        "multilineProcessingEnabled": false,
        "certPath": "",
        "sourceType": "DockerStats",
        "pollInterval": 5000
    }
	]
}

It works locally using docker-compose and from ECS it creates the collector in sumologic dashboard but when I deploy it to ECS it fails with messages like this:

Full log:

2023-12-20T17:15:18.376-03:00 Running SumoLogic Collector...
2023-12-20T17:15:36.734-03:00 wrapper | JVM exited unexpectedly.
2023-12-20T17:15:43.249-03:00 wrapper | The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
2023-12-20T17:15:43.249-03:00 wrapper | --> Wrapper Started as Console
2023-12-20T17:15:43.250-03:00 �]0;Sumo Logic Collector�wrapper | Java Service Wrapper Standard Edition 64-bit 3.5.49
2023-12-20T17:15:43.250-03:00 wrapper | Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
2023-12-20T17:15:43.250-03:00 wrapper | http://wrapper.tanukisoftware.com
2023-12-20T17:15:43.250-03:00 wrapper | Licensed to Sumo Logic Inc. for Collector
2023-12-20T17:15:43.250-03:00 wrapper |
2023-12-20T17:15:43.250-03:00 wrapper | Launching a JVM...
2023-12-20T17:15:43.250-03:00 jvm 1 | WrapperManager: Initializing...
2023-12-20T17:15:43.250-03:00 jvm 1 | . . . . . . . . .
2023-12-20T17:15:43.250-03:00 jvm 1 | .+'|=|+. .+'| |+. .+'|=|+.=|+. .+'|=|`+.
2023-12-20T17:15:43.250-03:00 jvm 1 | | | +.| | | | | | | + | `+ | | | | |
2023-12-20T17:15:43.250-03:00 jvm 1 | | | . | | | | | | | | | | | | | |
2023-12-20T17:15:43.250-03:00 jvm 1 | +.|=|+. | | | | | | | | | | | | | |
2023-12-20T17:15:43.250-03:00 jvm 1 | . | | | | | | | | | | | | | | | |
2023-12-20T17:15:43.250-03:00 jvm 1 | |`+. | | | | | | | | | | | | | | | |
2023-12-20T17:15:43.250-03:00 jvm 1 | +.|=|.+' +.|=|.+' +.| |.| |+' +.|=|.+'
2023-12-20T17:15:43.250-03:00 jvm 1 | Sumo Logic Collector Version 19.467-2
2023-12-20T17:15:43.250-03:00 jvm 1 | Sumo Logic Build Hash 54d48e135565
2023-12-20T17:15:43.251-03:00 jvm 1 | current folder:/opt/SumoCollector
2023-12-20T17:15:43.251-03:00 jvm 1 | * See /opt/SumoCollector/./logs for more details.
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.251-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.252-03:00 jvm 1 | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
2023-12-20T17:15:43.252-03:00 jvm 1 | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
2023-12-20T17:15:43.252-03:00 jvm 1 | (oslib.linux.CommandContainsType) cmd=lsb_release -sd > null
2023-12-20T17:15:43.252-03:00 jvm 1 | * Connecting to https://collectors.sumologic.com.
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.253-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.253-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.253-03:00 wrapper | JVM received a signal SIGKILL (9).
2023-12-20T17:15:43.253-03:00 wrapper | JVM process is gone.
2023-12-20T17:15:43.253-03:00 wrapper | JVM process exited with a code of 1, setting the Wrapper exit code to 1.
2023-12-20T17:15:43.253-03:00 wrapper | Reloading Wrapper configuration...
2023-12-20T17:15:43.253-03:00 wrapper | The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
2023-12-20T17:15:43.253-03:00 �]0;Sumo Logic Collector�wrapper | JVM process is gone.

My task definition definition as a sidecar in terraform is:

resource "aws_ecs_task_definition" "service_task_definition" {
  family                   = "${var.env}-${var.name}-task-definition"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = var.task_execution_role

  //https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html
  cpu                      = 2048 //var.cpu +  256
  memory                   = 4096 //var.memory + 256

  runtime_platform {
    cpu_architecture        = "X86_64"
    operating_system_family = "LINUX"
  }

  volume {
    name = "service-logs-storage"  
  }

  volume {
    name = "service-sock-storage"  
  }

  container_definitions = jsonencode([
    {
      name      = var.container_name
      image     = var.docker_image
      repositoryCredentials = {
        credentialsParameter = var.docker_repository_credentials
      },
      cpu       = var.cpu
      memory    = var.memory
      essential = true
      networkMode = "awsvpc",

      secrets = [
        for key in var.secrets_names :
        {
          name      = key
          valueFrom = "${var.secrets_arn}:${key}::"
        }
      ]
      
      environment = [
          {
              "name": "ECS_AVAILABLE_LOGGING_DRIVERS",
              "value": "[\"json-file\",\"awslogs\"]"
          }
      ]

      healthCheck = {
        command = [
            "CMD-SHELL",
            "curl -f http://localhost/healthcheck || exit 1"
        ],
        interval = 300,
        timeout = 5,
        retries = 3,
        startPeriod = 120
      }

      logConfiguration = {
        logDriver = "awslogs",
        options   = {
          awslogs-create-group  = "true",
          awslogs-group         = "/ecs/${var.env}/${var.name}-task",
          awslogs-region        = "us-west-1",
          awslogs-stream-prefix = "ecs"
        }
      }

      portMappings = [
        {
          containerPort = 80
          hostPort      = 80
          protocol      = "tcp"
          appProtocol   = "http"
        }
      ]

      mountPoints = [
        {
            sourceVolume: "service-logs-storage",
            containerPath: "/var/lib/docker/containers/logs",
            readOnly: false
        },
        {
            sourceVolume: "service-sock-storage",
            containerPath: "/var/run/docker.sock",
            readOnly: false
        }      
      ]
    },
    {
      name      = "${var.container_name}-logs-collector"
      image     = var.logs_collector_docker_image
      repositoryCredentials = {
        credentialsParameter = var.docker_repository_credentials
      },
      cpu       = 256
      memory    = 512
      essential = true
      networkMode = "awsvpc",

      secrets = [
        for key in var.secrets_names :
        {
          name      = key
          valueFrom = "${var.secrets_arn}:${key}::"
        }
      ]
      
      environment = [
          {
              "name": "SUMO_COLLECTOR_NAME",
              "value": "${var.env}"
          },
          {
              "name": "SUMO_COLLECTOR_NAME_PREFIX",
              "value": ""
          },
          {
              "name": "SUMO_COLLECTOR_FIELDS",
              "value": "env=local,another=none"
          },
          {
              "name": "SUMO_ACCESS_ID",
              "value": "...."
          },
          {
              "name": "SUMO_ACCESS_KEY",
              "value": "..."
          },
      ]

      logConfiguration = {
        logDriver = "awslogs",
        options   = {
          awslogs-create-group  = "true",
          awslogs-group         = "/ecs/${var.env}/${var.name}-task",
          awslogs-region        = "us-west-1",
          awslogs-stream-prefix = "ecs"
        }
      }

      mountPoints = [
        {
            sourceVolume: "service-logs-storage",
            containerPath: "/var/lib/docker/containers/logs",
            readOnly: false
        },
        {
            sourceVolume: "service-sock-storage",
            containerPath: "/var/run/docker.sock",
            readOnly: false
        }
      ]
    }
  ])
}

I could not find any useful information on what could be the issue or how to deploy it on ECS Fargate without using an HTTP source.
@frossi85
Copy link
Author

I could solve it by using my own Dockerfile installing what my error said was missing in the original image:

FROM sumologic/collector
ADD file-sumo-sources.json /etc/sumo-sources.json

RUN apt-get update --quiet && \
  apt-get install -y lsb-core && \
  apt-get reinstall --quiet -y lsb-release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@frossi85