Fluentd pods are restarted when connection to API server is lost

[andrzej-stencel]

Here's the stack trace:

#<Thread:0x00007f1ee7cc5918@/usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/filter_kubernetes_metadata.rb:269 run> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/http-4.4.1/lib/http/response/parser.rb:31:in `add': error reading from socket: Could not parse data (HTTP::ConnectionError)
	from /usr/local/bundle/gems/http-4.4.1/lib/http/response/parser.rb:14:in `add'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/connection.rb:214:in `read_more'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/response/body.rb:36:in `each'
	from /usr/local/bundle/gems/kubeclient-4.5.0/lib/kubeclient/watch_stream.rb:25:in `each'
	from /usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in `start_namespace_watch'
	from /usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/filter_kubernetes_metadata.rb:269:in `block in configure'
/usr/local/bundle/gems/http-4.4.1/lib/http/response/parser.rb:31:in `add': Could not parse data (IOError)
	from /usr/local/bundle/gems/http-4.4.1/lib/http/connection.rb:214:in `read_more'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'
	from /usr/local/bundle/gems/http-4.4.1/lib/http/response/body.rb:36:in `each'
	from /usr/local/bundle/gems/kubeclient-4.5.0/lib/kubeclient/watch_stream.rb:25:in `each'
	from /usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in `start_namespace_watch'
	from /usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/filter_kubernetes_metadata.rb:269:in `block in configure'
Unexpected error error reading from socket: Could not parse data
  /usr/local/bundle/gems/http-4.4.1/lib/http/response/parser.rb:31:in `add'
  /usr/local/bundle/gems/http-4.4.1/lib/http/connection.rb:214:in `read_more'
  /usr/local/bundle/gems/http-4.4.1/lib/http/connection.rb:92:in `readpartial'
  /usr/local/bundle/gems/http-4.4.1/lib/http/response/body.rb:30:in `readpartial'
  /usr/local/bundle/gems/http-4.4.1/lib/http/response/body.rb:36:in `each'
  /usr/local/bundle/gems/kubeclient-4.5.0/lib/kubeclient/watch_stream.rb:25:in `each'
  /usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/kubernetes_metadata_watch_namespaces.rb:36:in `start_namespace_watch'
  /usr/local/bundle/gems/fluent-plugin-kubernetes_metadata_filter-2.4.1/lib/fluent/plugin/filter_kubernetes_metadata.rb:269:in `block in configure'

The stack trace looks very similar to the one in this issue on fluentd-plugin-kubernetes_metadata_filter: fabric8io/fluent-plugin-kubernetes_metadata_filter#194.
This issue was fixed in versions later than 2.4.1 of the plugin, so I suggest to bump the version of the plugin used here:

sumologic-kubernetes-collection/deploy/docker/Dockerfile

Line 38 in e9912c5

fluent-plugin-kubernetes_metadata_filter:2.4.1 \