Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nomad Logs doesn't return logs on short running containers. #2457

Closed
sprutner opened this issue Mar 20, 2017 · 6 comments · May be fixed by #10910
Closed

Nomad Logs doesn't return logs on short running containers. #2457

sprutner opened this issue Mar 20, 2017 · 6 comments · May be fixed by #10910

Comments

@sprutner
Copy link
Contributor

sprutner commented Mar 20, 2017

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Nomad v0.5.4

Operating system and Environment details

Ubuntu 16.04 LTS on AWS

Issue

Fast running docker containers (a few seconds long) do not leave logs accessible with the nomad logs command. Also, the container never appears in a docker ps -a on the node. The container in question (myena/consul-backinator) just runs a simple command. The logs are, however, accessible on the node at /var/lib/nomad/alloc/[allocation-id]/alloc/logs/consul-backinator.stderr.0

Reproduction steps

Run the nomad job file below:

job "consul-backinator" {

  datacenters = ["dc1"]


  type = "service"


  update {

    stagger = "10s"


    max_parallel = 1
  }


  group "consul-backinator" {

    count = 1


    restart {
      # The number of attempts to run the job within the specified interval.
      # Runs the job every hour, adjust depending on your needs.
      
      attempts = 1
      interval = "1h"

      # The "delay" parameter specifies the duration to wait before restarting
      # a task after it has failed.
      
      delay = "1h"


      mode = "delay"
    }


    ephemeral_disk {

      size = 300
    }

  #task for consul-backinator
  task "consul-backinator" {

      driver = "docker"

      # The "config" stanza specifies the driver configuration, which is passed
      # directly to the driver to start the task. The details of configurations
      # are specific to each driver, so please see specific driver
      # documentation for more information.
      
      config {
        image = "myena/consul-backinator"

        args = [
        "backup", "-file", "s3://name-of-your-bucket/backups"
        ]

      }

      env {

        "AWS_ACCESS_KEY_ID"     = "AWS ACCESS KEY GOES HERE"
        "AWS_SECRET_ACCESS_KEY" = "AWS SECRET KEY GOES HERE"
        "AWS_REGION"            = "us-east-1"
        "CONSUL_HTTP_ADDR"      = "${NOMAD_IP_consulbackinator}:8500"
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
        network {
          mbits = 10
          port "consulbackinator" {}
        }
      }


    }
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Job file (if appropriate)

Quoted inline above

@dadgar
Copy link
Contributor

dadgar commented Mar 20, 2017

Hey I couldn't replicate this:

nomad alloc-status b9
ID                  = b9ce3119
Eval ID             = ea9c0ec8
Name                = example.cache[0]
Node ID             = 564daec4
Job ID              = example
Client Status       = complete
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created At          = 03/20/17 17:55:27 UTC

Task "redis" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
500 MHz  256 MiB  300 MiB  0     db: 127.0.0.1:44055

Recent Events:
Time                   Type        Description
03/20/17 17:55:27 UTC  Terminated  Exit Code: 0
03/20/17 17:55:27 UTC  Started     Task started by client
03/20/17 17:55:27 UTC  Task Setup  Building Task Directory
03/20/17 17:55:27 UTC  Received    Task received by client
vagrant@nomad-server01:/opt/gopath/src/github.com/hashicorp/nomad$ nomad fs b9 alloc/logs
Mode        Size  Modified Time          Name
-rw-r--r--  0 B   03/20/17 17:55:27 UTC  redis.stderr.0
-rw-r--r--  14 B  03/20/17 17:55:27 UTC  redis.stdout.0
vagrant@nomad-server01:/opt/gopath/src/github.com/hashicorp/nomad$ nomad logs b9
hello, world!

Job:

      config {
        image = "redis:3.2"
        command = "echo"
        args = ["hello, world!"]
      }

I believe you may just be missing nomad logs -stderr <id> (the stderr flag).

@dadgar dadgar closed this as completed Mar 20, 2017
@perkfly
Copy link

perkfly commented Jan 3, 2019

I can reproduce this, logs for short running param jobs are all missing

@chrissound
Copy link

Also seem to notice this behavior

@yurii-fisakov
Copy link

v 0.9.2 also affected

@Manicqin
Copy link

Manicqin commented Jan 8, 2020

same for 0.10.0-dev

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants