Programmatic access to full run failure logs, inc stdout/stderr

[genesis-gh-adekel]

Hi all. Is there any way to fetch the full stdout/stderr logs from a run on Dagster+. I've been trying to play with the GraphQL but honestly and I getting lost there and cannot get the result I'm looking for. Basically I'm trying to get from a run_id to the steps it ran (a dbt run in my case) to the stdout/stderr logs where I can see my full DBT logs. To be clear, I'm trying to do all this programmatically.

[cmpadden]

Hi @genesis-gh-adekel , if you open the developer tools of the browser, and go to the Network tab. You may notice that there's a GraphQL CapturedLogsMetadataQuery.

If you go to the Request tab, you will see the actual GraphQL query that is being performed.

Note that the stdout and stderr logs are stored on S3, so the response will give you the values to where that file resides with stdoutDownloadUrl and stderrDownloadUrl. You should be able to read those files to get the log content.

[garethbrickman]

Here are queries for the GraphQL API for getting a job's run ID filtered by date and using it to fetch run logs.

GetRunIdsForJobWithDateFilter.gql

query GetRunIdsForJobWithDateFilter($jobName: String!, $createdAfter: Float, $createdBefore: Float) {
  runsOrError(filter: { pipelineName: $jobName, createdAfter: $createdAfter, createdBefore: $createdBefore }) {
    ... on Runs {
      results {
        runId
      }
    }
  }
}

GetRunIdsForJobWithDateFilter_query_variables.json

{
  "jobName": "my_job_name",
  "createdAfter": 1726804801.00,
  "createdBefore": 1726891201.00
}

GetLogsForRun.gql

query GetLogsForRun($runId: ID!, $afterCursor: String, $limit: Int) {
  logsForRun(runId: $runId, afterCursor: $afterCursor, limit: $limit) {
    ... on EventConnection {
      events {
        __typename
        ... on LogMessageEvent {
          message
          timestamp
          level
          stepKey
        }
        ... on MaterializationEvent {
          assetKey{
            path
          }
	  message
          timestamp
          level
          stepKey
        }
        # You can add more event types here if needed
      }
      cursor
      hasMore
    }
    ... on RunNotFoundError {
      message
    }
    ... on PythonError {
      message
      stack
    }
  }
}

GetLogsForRun_query_variables.json

{
  "runId": "601002a0-e85b-4f75-b7f3-72eb0e3ab9ee"
}

[genesis-gh-adekel]

Thanks guys! This was very helpful.