Looking for help: not receiving any trace data from collector #1557
Comments
|
@mchiaramonte can you try by setting |
|
@serkan-ozal Yes, Let me give that a try. I had tried that at some point but not with this more minimal configuration. Give me a bit to try this out. Thanks for responding. |
|
Okay, I tried it with no change. Here's log output from when I call the lambda. There was an error regarding metric export, which I'm not sure is related all. I don't actually care about the metrics, so I'm only including it for completeness. |
|
Hmm, @mchiaramonte can you also try by setting |
|
@mchiaramonte, I think metrics are failing because, there is no metrics pipeline in the collector, but traces pipeline. So, no metrics endpoint should be available |
|
Here's the log info after adding debug logging and making 10 requests: |
|
@mchiaramonte Let's investigate step by step by starting from simplest form of configuations:
With these configurations, traces and metrics should be printed out to the stdout and so Cloudwatch logs. For example: Let's verify these configurations first and let us whether it works as expected. Then we can continue by using custom collector configuration with |
|
Okay- let me take a look and see if I can get this done today. Thanks! |
|
Keeping everything else the same, I removed all the environment variables except AWS_LAMBDA_EXEC_WRAPPER and OTEL_TRACES_SAMPLER and invoked the lambda via postman. This is what it printed in cloudwatch logs:
|
|
Also, not sure if it will matter, but just so you can see here are the layers that are part of this lambda:
|
|
@mchiaramonte it is really weird. No AWS Lambda instrumentation is in place. Is it possible to share the sample Lambda artifact zip file you upload to Lambda (along with handler name you set) with us so we can try to reproduce the issue. As far as I see, somehow, your handler is not automatically wrapped/instrumented. |
|
@mchiaramonte and in the meantime, can you also try with the previous layer version: |
|
@mchiaramonte and is your handler an ES (EcmaScript) module (generated handler file name ends with |
|
@serkan-ozal Yes, it's a .mjs file - it's the default file generated by Amazon when you create the lambda for node.js. I can try the older layer if you like. As for the artifact zip- we're not even there yet. This is just me working with it directly in the editor within Amazon. I was just trying to prove out that this even works. That code I initially provided is all that's there. It's just the index.mjs file. |
|
@mchiaramonte Ok, I think that explains everything. AFAIK, ES modules are not supported by OpenTelemetry AWS Lambda instrumentation. Is it possible to produce a common JS file ( |
|
@serkan-ozal Ah okay. Let me try that. |
|
@serkan-ozal Yes, there we go. That's what the problem was. Okay- did I just miss this in the documentation somewhere? Or is this not explicitly called out? |
|
@mchiaramonte I don't think it is mentioned at somewhere. I have just found it out from the AWS Lambda instrumentation source code here: https://github.com/open-telemetry/opentelemetry-js-contrib/blob/7633caee19a7f04c5cc5e191d1ae7745ac3739f1/plugins/node/opentelemetry-instrumentation-aws-lambda/src/instrumentation.ts#L119 Only |
|
Great, @serkan-ozal - thank you so much for all your help with this. It's greatly appreciated. |
8 tasks
|
Hi @mchiaramonte, Just FYI. I have a working solution for the automated wrapping of the ESM based handlers. So, I have sent the following PRs: |
|
Hi @serkan-ozal this is great. Thanks for the update. I really appreciate the help and responsiveness on this. One other thing I noticed while testing things out is it looks like the fetch API isn't being auto-instrumented. Is that correct? I saw an experimental project for instrumenting fetch here, so I assume this is expected. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I'll preface this by saying this is likely something I'm doing wrong, but basically the issue is I'm never able to receive traces from Lambda. If I don't use a collector, I see the metrics endpoint invoked from (presumably) the auto-instrumentation coming in via the open telemetry node layer. My lambda isn't even really code- it's just the default example Amazon provides:
And I've added the layers
opentelemetry-nodejs-0_10_0andopentelemetry-collector-amd64-0_11_0, as per the directions. I've setup only these two environment variables:And then my config.yaml is configured as follows:
(Yes I realize I'm not using batch, but I was trying it out at one point and it made no difference that I could tell)
I am not seeing anything from the collector coming in at all. As I mentioned, if I don't use the collector, I do see the
v1/metricsendpoint get invoked, but never thev1/tracesendpoint. I've tried many permutations of the above configuration but none of them seem to make any difference. I've spent several days trying out different things and nothing is working. I'm at a total loss at this point.The local software running that receives the traces is functioning properly, because if I run a standard node auto-instrumented app I see the data coming in. I can confirm that I can get external traffic from the address and port that are open.
Any help or advice that can be provided would be appreciated. Thanks so much!
The text was updated successfully, but these errors were encountered: