Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

slog-to-otel does not handle slow vat termination #10405

Open
gibson042 opened this issue Nov 5, 2024 · 3 comments
Open

slog-to-otel does not handle slow vat termination #10405

gibson042 opened this issue Nov 5, 2024 · 3 comments
Assignees
Labels
bug Something isn't working

Comments

@gibson042
Copy link
Member

Describe the bug

Errors from https://github.com/Agoric/agoric-sdk/actions/runs/11692254542/job/32561236661?pr=10165 :

Unknown crank-start message.type cleanup-terminated-vat
Error tracing slog: {
  type: 'crank-start',
  crankType: 'delivery',
  crankNum: 4913n,
  message: {
    type: 'cleanup-terminated-vat',
    vatID: 'v16',
    budget: { default: 5, kv: 50 }
  },
  time: 1730840617.433475,
  monotime: 167.98721473999984
} (Error#1)
Error#1: Unknown crank-start message.type cleanup-terminated-vat

  at extractMessageAttrs (packages/telemetry/src/slog-to-otel.js:269:13)
  at slogSender (packages/telemetry/src/slog-to-otel.js:1035:46)
  at wrappedSlogSender (packages/telemetry/src/slog-to-otel.js:1082:7)
  at slogSender (packages/telemetry/src/make-slog-sender.js:154:11)
  at process.<anonymous> (packages/telemetry/src/slog-sender-pipe-entrypoint.js:116:15)
  at process.emit (node:events:517:28)
Unknown slog type: vat-cleanup
Error tracing slog: {
  type: 'vat-cleanup',
  vatID: 'v16',
  work: { kv: 8, snapshots: 1, transcripts: 2 },
  time: 1730840617.435071,
  monotime: 167.98881064499997
} (Error#2)
Error#2: Unknown slog type: vat-cleanup

  at slogSender (packages/telemetry/src/slog-to-otel.js:1075:13)
  at wrappedSlogSender (packages/telemetry/src/slog-to-otel.js:1082:7)
  at slogSender (packages/telemetry/src/make-slog-sender.js:154:11)
  at process.<anonymous> (packages/telemetry/src/slog-sender-pipe-entrypoint.js:116:15)
  at process.emit (node:events:517:28)
Unknown slog type: vat-cleanup-complete
Error tracing slog: {
  type: 'vat-cleanup-complete',
  vatID: 'v16',
  time: 1730840617.43517,
  monotime: 167.98891011099994
} (Error#3)
Error#3: Unknown slog type: vat-cleanup-complete

  at slogSender (packages/telemetry/src/slog-to-otel.js:1075:13)
  at wrappedSlogSender (packages/telemetry/src/slog-to-otel.js:1082:7)
  at slogSender (packages/telemetry/src/make-slog-sender.js:154:11)
  at process.<anonymous> (packages/telemetry/src/slog-sender-pipe-entrypoint.js:116:15)
  at process.emit (node:events:517:28)
@gibson042 gibson042 added the bug Something isn't working label Nov 5, 2024
@gibson042
Copy link
Member Author

Errors from ollinet upgrade to upgrade-18-rc2 :

  • Unknown slog type: cosmic-swingset-inject-kernel-upgrade-events
  • Unknown crank-start message.type upgrade-vat

@gibson042
Copy link
Member Author

Issues from mainfork testing:

  • A slog entry with jsonPayload including "message": { "type": "cleanup-terminated-vat", "vatID": "v156", "budget": { "default": null } } was followed by one with jsonPayload including "vatID": "v156", "type": "create-vat", which obviously didn't happen (as confirmed by direct slogfile inspection).

@mhofman
Copy link
Member

mhofman commented Jan 4, 2025

I'm tempted to say we should not bother fixing this until we work on #10799, especially if we disabled otel-trace in integration tests (#10798)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants