Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error=":excon is not registered on Faraday::Adapter" with v1.15.0 Docker and fluent-plugin-opensearch #338

Closed
brian-finisher opened this issue Jul 28, 2022 · 6 comments
Closed

error=":excon is not registered on Faraday::Adapter" with v1.15.0 Docker and fluent-plugin-opensearch #338

brian-finisher opened this issue Jul 28, 2022 · 6 comments

Comments

@brian-finisher
Copy link

Describe the bug

With FluentD v1.15.0 Docker image for linux/arm64 (tag: fluent/fluentd:v1.15.0-debian-arm64-1.0) I get the following recurring exception, and no records logged to opensearch as desired.

unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter" (see full error below)

To Reproduce

Generate a FluentD docker image from the fluent/fluentd:v1.15.0-debian-arm64-1.0 base image.
Install fluent-plugin-opensearch gem.
Try to send records to OpenSearch.

Expected behavior

Records record to opensearch

Your Environment

- Fluentd version: v1.15.0
- TD Agent version:
- Operating system: Linux Ubuntu 20.04 for Arm64 (Docker 20.10.12)
- Kernel version:

My customized Docker image uses fluent/fluentd:v1.15.0-debian-arm64-1.0 as a base image, and installs the fluent-plugin-opensearch gem (v1.0.7)

Your Configuration

(I redacted much below. I didn't test this exact configuration to confirm)

<source>
  @type forward
  port 24226
  bind 0.0.0.0
</source>
<filter **>
  @type record_transformer
  <record>
    (redacted)
  </record>
</filter>
<filter mylogs/**>
  @type         parser
  key_name      log
  reserve_time  true
  reserve_data  true
  <parse>
    @type      regexp
    (redacted)
  </parse>
</filter>
<match mylogs/**>
  @type copy
  # Store service logs to local OpenSearch instance
  <store>
    @type opensearch
    host localhost
    port 9200
    user ____
    password ____
    include_timestamp true
    logstash_format true
    logstash_prefix mylogs
    <buffer>
      flush_interval 1s
    </buffer>
  </store>
</match>

Your Error Log

Error:
2022-07-28 20:14:21 +0000 [error]: #0 unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter"
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.4.0/lib/faraday/middleware_registry.rb:57:in `lookup_middleware'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.4.0/lib/faraday/rack_builder.rb:113:in `adapter'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/lib/ruby/3.1.0/forwardable.rb:238:in `adapter'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:633:in `block in client'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.4.0/lib/faraday/connection.rb:91:in `initialize'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.4.0/lib/faraday.rb:98:in `new'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.4.0/lib/faraday.rb:98:in `new'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.0.1/lib/opensearch/transport/transport/http/faraday.rb:72:in `__build_connection'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.0.1/lib/opensearch/transport/transport/base.rb:166:in `block in __build_connections'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.0.1/lib/opensearch/transport/transport/base.rb:158:in `map'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.0.1/lib/opensearch/transport/transport/base.rb:158:in `__build_connections'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.0.1/lib/opensearch/transport/transport/base.rb:69:in `initialize'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:650:in `new'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:650:in `client'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:62:in `host_unreachable_exceptions'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:71:in `rescue in retry_operate'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:69:in `retry_operate'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:521:in `handle_last_seen_os_major_version'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:400:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/plugin.rb:187:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/plugin/multi_output.rb:99:in `each'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/plugin.rb:187:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/agent.rb:132:in `add_match'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/agent.rb:74:in `block in configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/agent.rb:64:in `each'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/agent.rb:64:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/root_agent.rb:149:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/engine.rb:105:in `configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/engine.rb:80:in `run_configure'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/supervisor.rb:759:in `block in run_worker'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/supervisor.rb:1032:in `main_process'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/supervisor.rb:751:in `run_worker'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/lib/fluent/command/fluentd.rb:386:in `<top (required)>'
  2022-07-28 20:14:21 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2022-07-28 20:14:21 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.15.0/bin/fluentd:15:in `<top (required)>'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:25:in `load'
  2022-07-28 20:14:21 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:25:in `<main>'

Additional context

Not reproducible with FluentD v1.14.6, fluent-plugin-opensearch v1.0.7
@ashie ashie transferred this issue from fluent/fluentd Aug 1, 2022
@ashie
Copy link
Member

ashie commented Aug 1, 2022

How about adding excon gem?

@brian-finisher
Copy link
Author

I tried doing a "gem install excon" in my custom docker image, and unfortunately still get the same error.
Also tried the new v1.15.1 docker image and have the same issue.
Unfortunately I have no experience in Ruby to really understand the root of these dependencies/errors.
Just to give it a try, I also installed the faraday gem with no luck. Also tried dowgrading faraday to v2.3.0 and 2.2.0 with the same errors.
Are others able to reproduce this error for debugging?

@ashie
Copy link
Member

ashie commented Aug 1, 2022

Hmm, it seems an issue of fluent-plugin-opensearch.
I got same error on running unit test of it.

$ git clone https://github.com/fluent/fluent-plugin-opensearch.git
$ cd fluent-plugin-opensearch
$ bundle install
$ bundle exec rake test
/usr/bin/ruby2.7 -I"lib:test" /home/aho/Projects/Fluentd/fluent-plugin-opensearch/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb "test/plugin/test_filter_opensearch_genid.rb" "test/plugin/test_in_opensearch.rb" "test/plugin/test_oj_serializer.rb" "test/plugin/test_opensearch_error_handler.rb" "test/plugin/test_opensearch_fallback_selector.rb" "test/plugin/test_opensearch_tls.rb" "test/plugin/test_out_opensearch.rb" "test/plugin/test_out_opensearch_data_stream.rb" "test/test_log-ext.rb" 
Loaded suite /home/aho/Projects/Fluentd/fluent-plugin-opensearch/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader
Started
.F
=============================================================================================================================================
Failure: test_fallback_on_info(OpenSearchFallbackSelectorTest)
/home/aho/Projects/Fluentd/fluent-plugin-opensearch/test/plugin/test_opensearch_fallback_selector.rb:89:in `test_fallback_on_info'
     86:       reload_after 10
     87:       catch_transport_exception_on_retry false # For fallback testing
     88:     ]
  => 89:     assert_raise(OpenSearch::Transport::Transport::Errors::NotFound) do
     90:       driver(config)
     91:     end
     92:     driver.run(default_tag: 'test') do

<OpenSearch::Transport::Transport::Errors::NotFound> expected but was
<#<Faraday::Error #<Faraday::Error: :excon is not registered on Faraday::Adapter>>>

diff:
? OpenSearch::Transport::Transport::Errors         ::N                 ot               Found             
? #<F            d          y             #<Faraday  Error: :excon is n   registered on  ara ay::Adapter>>
=============================================================================================================================================
.```

Probably something is changed recent days in dependent gems.
When I install faraday-excon and add `require 'faraday-excon'` to out_opensearch.rb, the issue is resolved.

ref: https://github.com/opensearch-project/opensearch-ruby/blob/main/opensearch-transport/README.md

@ashie
Copy link
Member

ashie commented Aug 1, 2022 •
edited
Loading

Note: opensearch-transport 2.0.1 & 1.0.1 released at July 26 unlocks faraday 2.x:

As described in https://github.com/opensearch-project/opensearch-ruby/blob/main/opensearch-transport/README.md faraday-excon is additionally required since faraday 2.0.

@ashie ashie mentioned this issue Aug 1, 2022
Closed
2 tasks
@ashie
Copy link
Member

ashie commented Aug 1, 2022 •
edited
Loading

I close this issue since it's not this docker image's issue.
Let's continue at fluent/fluent-plugin-opensearch#70

@ashie ashie closed this as completed Aug 1, 2022
@yelouarti
Copy link

Just wanted to add, that the uken/fluent-plugin-elasticsearch has/had a similar issue (scroll down in issue 1014). In all versions "elasticsearch" is added as dependency, that loads internally the latest faraday version. Starting with version 5.3.0, the plugin becomes usuable again with the latest faraday version, as it relies on version 2.x (before it was 1.x).

In summary, not an issue of this image, but one of the plugins installed on top.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ashie @yelouarti @brian-finisher