Skip to content

Daemonset restarting several times #495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wlopes39 opened this issue Sep 30, 2020 · 7 comments
Closed

Daemonset restarting several times #495

wlopes39 opened this issue Sep 30, 2020 · 7 comments

Comments

@wlopes39
Copy link

wlopes39 commented Sep 30, 2020 •
edited by cosmo0920
Loading

Hi,

I'm using the EFK stack on my k8s cluster. Everything was normal, until the last few days Fluentd is restarting several times.

Pod logs:

2020-09-30 19:19:17 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-7c6744f885-rv55p_app-98fd4195a2ae0f7a7042ed1a55d887e6c78c5b2479cd36852a5ba835da896b8c.log
2020-09-30 19:19:17 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-77b579456c-jgknz_tsetup-946b7b413d893b575366d433a23ad43a4a65e7bfacf61c17aa26a872fa1f9285.log
2020-09-30 19:19:17 +0000 [info]: #0 fluentd worker is now running worker=0
2020-09-30 19:19:45 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 15, pod_cache_size: 16, namespace_cache_api_updates: 16, pod_cache_api_updates: 16, id_cache_miss: 16, pod_cache_watch_misses: 11, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:20:15 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 18, pod_cache_size: 19, namespace_cache_api_updates: 19, pod_cache_api_updates: 19, id_cache_miss: 19, pod_cache_watch_misses: 11, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:20:45 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 18, pod_cache_size: 19, namespace_cache_api_updates: 19, pod_cache_api_updates: 19, id_cache_miss: 19, pod_cache_watch_misses: 11, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:21:15 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 18, pod_cache_size: 19, namespace_cache_api_updates: 19, pod_cache_api_updates: 19, id_cache_miss: 19, pod_cache_watch_misses: 11, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:21:45 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 18, pod_cache_size: 19, namespace_cache_api_updates: 19, pod_cache_api_updates: 19, id_cache_miss: 19, pod_cache_watch_misses: 11, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:22:15 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 19, pod_cache_size: 20, namespace_cache_api_updates: 20, pod_cache_api_updates: 20, id_cache_miss: 20, pod_cache_watch_misses: 11, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:22:45 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 19, pod_cache_size: 20, pod_cache_watch_misses: 13, namespace_cache_api_updates: 20, pod_cache_api_updates: 20, id_cache_miss: 20, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
2020-09-30 19:23:15 +0000 [info]: #0 [filter_kube_metadata] stats - namespace_cache_size: 19, pod_cache_size: 20, pod_cache_watch_misses: 17, namespace_cache_api_updates: 20, pod_cache_api_updates: 20, id_cache_miss: 20, pod_cache_watch_ignored: 2, pod_cache_watch_delete_ignored: 2
#<Thread:0x00007f5c05b6bf08@/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265 run> terminated with exception (report_on_exception is true):
/usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in sysread: error reading from socket: Connection reset by peer (HTTP::ConnectionError)
        from /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in read_more
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.0/lib/kubeclient/watch_stream.rb:25:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in start_namespace_watch
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in block in configure
/usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in sysread: Connection reset by peer (Errno::ECONNRESET)
        from /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in read_more
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.0/lib/kubeclient/watch_stream.rb:25:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in start_namespace_watch
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in block in configure
#<Thread:0x00007f3e55b34490@/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265 run> terminated with exception (report_on_exception is true):
/usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in sysread: error reading from socket: Connection reset by peer (HTTP::ConnectionError)
        from /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in read_more
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.0/lib/kubeclient/watch_stream.rb:25:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in start_namespace_watch
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in block in configure
/usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in sysread: Connection reset by peer (Errno::ECONNRESET)
        from /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in read_more
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in readpartial
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.0/lib/kubeclient/watch_stream.rb:25:in each
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in start_namespace_watch
        from /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in block in configure
Unexpected error error reading from socket: Connection reset by peer
  /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in sysread
  /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in readpartial
  /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in readpartial
  /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in read_more
  /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in readpartial
  /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in readpartial
  /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in each
  /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.0/lib/kubeclient/watch_stream.rb:25:in each
  /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in start_namespace_watch
  /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in block in configure
2020-09-30 19:23:17 +0000 [error]: #0 unexpected error error_class=HTTP::ConnectionError error="error reading from socket: Connection reset by peer"
  2020-09-30 19:23:17 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in sysread
  2020-09-30 19:23:17 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/openssl/buffering.rb:125:in readpartial
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/timeout/null.rb:45:in readpartial
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:212:in read_more
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/connection.rb:92:in readpartial
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:30:in readpartial
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:36:in each
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/kubeclient-4.9.0/lib/kubeclient/watch_stream.rb:25:in each
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in start_namespace_watch
  2020-09-30 19:23:17 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-kubernetes_metadata_filter-2.3.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:265:in block in configure
2020-09-30 19:23:17 +0000 [error]: #0 unexpected error error_class=HTTP::ConnectionError error="error reading from socket: Connection reset by peer"
  2020-09-30 19:23:17 +0000 [error]: #0 suppressed same stacktrace

Image:
fluent / fluentd-kubernetes-daemonset:v1.11.2-debian-elasticsearch7-1.0

Elastic version: elasticsearch: 7.2.0

Fluentd Componentes:

fluent-plugin-concat version 2.4.0
fluent-plugin-dedot_filter version 1.0.0
fluent-plugin-detect-exceptions version 0.0.13
fluent-plugin-elasticsearch version 4.1.1
fluent-plugin-grok-parser version 2.6.1
fluent-plugin-json-in-json-2 version 1.0.2
fluent-plugin-kubernetes_metadata_filter version 2.3.0
fluent-plugin-multi-format-parser version 1.0.0
fluent-plugin-prometheus version 1.6.1
fluent-plugin-record-modifier version 2.0.1
fluent-plugin-rewrite-tag-filter version 2.2.0
fluent-plugin-systemd version 1.0.2
fluentd version 1.11.2

Communication with elasticsearch is ok.

With other images the error persists, example:
fluent / fluentd-kubernetes-daemonset: v1.4.2-debian-elasticsearch-1.1

Has anyone had this problem and have an idea of ​​what it will be?

Thanks
@chdeepak
Copy link

I am having the same issue. Is there any solution?

@sridharratnareddy
Copy link

I too have the same issue

@wlopes39
Copy link
Author

Alternatively, I'm using fluentbit.

@nunoguerreirorosa
Copy link

Same issue. Any particular solution?

@robbiezhang
Copy link

I think the issue is from the filter_kubernetes_metadata plugin.
Can we take the new version? It's fixed in 2.4.2.
fabric8io/fluent-plugin-kubernetes_metadata_filter#194

@cosmo0920
Copy link
Contributor

This issue is already fixed in v1.11 images:
#512

@rmweir rmweir mentioned this issue Dec 21, 2020
Merged
@dmitrytokarev
Copy link

Experiencing this issue with fluentd kubernetes daemon set v1.11.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@cosmo0920 @dmitrytokarev @robbiezhang @nunoguerreirorosa @chdeepak @wlopes39 @sridharratnareddy