[opentelemetry-php-contrib]: OpenTelemetry Laravel Auto-Instrumentation is not detecting jobs

[Nelwhix]

Describe your environment Hi there, I set up laravel auto-instrumentation. and it is correctly detecting the http request, the calls to DB, but not the jobs.

Steps to reproduce
The entire code is in a docker container here

What is the expected behavior?
I expected to see some spans for the job

What is the actual behavior?

More Context: I have a separate container for the api and the queue worker. The jobs are detected when I set QUEUE_CONNECTION to sync but not when it is set to database

[Nelwhix]

@ChrisLightfootWild Hi Chris, Thanks for your work on the laravel auto instrumentation package, I am sure I am just misconfiguring something so please if you could point me in the right direction. I would appreaciate🙏.

[ChrisLightfootWild]

@ChrisLightfootWild Hi Chris, Thanks for your work on the laravel auto instrumentation package, I am sure I am just misconfiguring something so please if you could point me in the right direction. I would appreaciate🙏.

Hey @Nelwhix 👋

Thanks for raising the issue. Based on your example repo, it looks like the database driver is being used for your queue. I wonder if this relates to long-running queues again 🤔. If you make several requests to the orders endpoint over a few seconds, I wonder if you might see the batch processor emit some of the telemetry that should be captured.

I will try to find some time to test this.

[alexandrecruz]

@Nelwhix running queue with sync invokes the job synchronously with your code.

if you use Redis or database drive, the workers are handling the Job.
Since that's happening, the trace won't display the Job metrics (which you should check separately).

This behavior is also similar to other proprietary APMs like new Relic, Elastic & others.
I don't see any errors based on the provided description.

[Nelwhix]

Hi @alexandrecruz. So what if I want the job to display traces, how do I configure that?

[alexandrecruz]

@Nelwhix I don't think that's possible for an async job since it's not running within the request cycle.
Instead, it is running as a separate thread/process in the background.

There might be a way to associate them, but again, since the job is running outside of the request cycle I don't believe the trace will be displaying it or affecting its metrics (CPU, RAM, time, etc)

[alexandrecruz]

I did not particularly test this solution, but Spatie has a TraceAware job that seems to send the same trace ID when running the job to combine results. You can check the approach and source code at: https://spatie.be/docs/laravel-open-telemetry/v1/basic-usage/measuring-queued-jobs

IMO: if the job is not part of the request cycle itself, it should be measured separately.
It can be useful to see them together but in reality, they are two separate things.

[Nelwhix]

I actually want them separately, I tried the spatie package first..it wasn't working

[ChrisLightfootWild]

@Nelwhix running queue with sync invokes the job synchronously with your code.

if you use Redis or database drive, the workers are handling the Job. Since that's happening, the trace won't display the Job metrics (which you should check separately).

This behavior is also similar to other proprietary APMs like new Relic, Elastic & others. I don't see any errors based on the provided description.

My understanding may have been off then, but it should currently serialize the job payload when writing to the jobs table in the database.

I was planning on checking into that, but have not had any free time to pick it up recently (apologies for anybody waiting on me here!).

[gmhafiz]

@Nelwhix Instead of php artisan queue:work, try php artisan queue:listen and see if there is a difference.

I get one trace when I use queue:work and it looks like this

When I switch to queue:listen, I get a trace within one of those 'Artisan handler' trace.

I am using supervisor to run queue:listen. Maybe something in there could help? https://codeberg.org/gmhafiz/observability/src/commit/8a31169f3011d934e64d8581bd3db4f50811a3b4/laravel/server/supervisord.conf#L19

[Nelwhix]

Thanks @gmhafiz let me try

[Nelwhix]

queue:listen works!

[ChrisLightfootWild]

The difference between queue:work and queue:listen, I believe here, is that queue:work is essentially a longer running process and the outer trace is not being wrapped up and exported as desired.

There's a couple of issues around this, I've just not quite had the time to crack it yet!