Create connection failed when queue is empty

[naughtyGitCat]

            await Task.Delay(new TimeSpan(0, 0, 1), stoppingToken);
            try
            {
                _logger.LogInformation($"now reading {_rabbitMQQueueName} rabbit msg");
                var factory = new ConnectionFactory()
                {
                    HostName = _rabbitConfig.Host,
                    Port = _rabbitConfig.Port,
                    UserName = _rabbitConfig.User,
                    Password = _rabbitConfig.Password,
                    VirtualHost = _rabbitConfig.VirtualHostName,
                    AutomaticRecoveryEnabled = true,
                    TopologyRecoveryEnabled = true
                };
                factory.DispatchConsumersAsync = true;
                using var connection = factory.CreateConnection();
                using var channel = connection.CreateModel();
                channel.ExchangeDeclare(exchange: _rabbitConfig.ExchangeName, type: ExchangeType.Direct, durable: true);
                channel.QueueDeclare(
                    _rabbitMQQueueName,
                    durable: true, exclusive: false,
                    autoDelete: false, arguments: new Dictionary<string, object> { { "x-message-ttl", 3600000 } });
                var consumer = new AsyncEventingBasicConsumer(channel);

                consumer.Received += async (model, ea) =>
                {
                    var msg = Encoding.UTF8.GetString(ea.Body.Span);
                    try
                    {
                        await HandleMessageAsync(msg);
                    }
                    catch (Exception ex)
                    {
                        await _alertService.SendExceptionAsync(ex, "HandleMessageAsync failed");
                        _errorBudget -= 1;
                        if (_errorBudget < 0) throw new Exception($"Consume {_rabbitMQQueueName} failed for {_errorBudget} times, terminate the task");
                    }
                };

                var consumeResult = channel.BasicConsume(_rabbitMQQueueName, autoAck: true, consumer: consumer);

                while (!stoppingToken.IsCancellationRequested)
                {
                    _logger.LogInformation("Blocking for waiting rabbit message...");
                    //https://stackoverflow.com/questions/44880870/rabbitmq-eventbasicconsumer-not-working
                    await Task.Delay(new TimeSpan(0, 0, 56), stoppingToken);
                }

            }
            catch (Exception ex)
            {
                await _alertService.SendExceptionAsync(ex, $"{this.GetType().Name}");
            }

version

RabbitMQ 3.8
RabbitMQ.Client 6.2.2

output

2021-11-15 20:18:51.346 +08:00 [WRN] =>[DBAInternalCommon.Services.DBAAlertService]=> DBAMessageDTO { Project = DBAMetaConsumerHelper, Exception = RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> RabbitMQ.Client.Exceptions.PossibleAuthenticationFailureException: Possibly caused by authentication failure
 ---> RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text='Unexpected Exception', classId=0, methodId=0, cause=System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BufferedStream.ReadByteSlow()
   at System.IO.BufferedStream.ReadByte()
   at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)
   at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
   at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
   at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
   at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)
   at RabbitMQ.Client.Impl.ModelBase.ConnectionStartOk(IDictionary`2 clientProperties, String mechanism, Byte[] response, String locale)
   at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
   at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist)
   at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName)
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IFrameHandler fh)
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)
   at DBAMetaConsumer.Workers.RabbitConsumerBase.ExecuteAsync(CancellationToken stoppingToken), Comment = HostCrontabConsumer , Receivers = System.String[], EmailSubject = XXXXYYYY, EmailSender = [DBAMessage], SendChannels = 20 }

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[naughtyGitCat]

on same time, same rabbitmq server, same rabbitmq consumer client and logic, the queue which have msg pump in, does not raise a connection failed exception

[michaelklishin]

I fail to see any connection between those facts. Queue operations are not possible before a connection is fully accepted. So this very well may be a coincidence or just an indication of a difference in configuration.

Instead of guessing as to what it may be, inspect server logs for clues.

[michaelklishin]

There is no evidence of a bug. There is evidence of a connection begin refused by the target RabbitMQ node:

2021-11-15 20:18:51.346 +08:00 [WRN] =>[DBAInternalCommon.Services.DBAAlertService]=> DBAMessageDTO { Project = DBAMetaConsumerHelper, Exception = RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> RabbitMQ.Client.Exceptions.PossibleAuthenticationFailureException: Possibly caused by authentication failure

Queue contents (or presence) do not matter at this point because the client failed to connect and "attach" to a virtual host.

There is a dedicated doc guide for troubleshooting connections.
All connection lifecycle events are logged, in particular authentication failures
and default user connectivity restrictions.

[michaelklishin]

Unable to read data from the transport connection: Connection reset by peer.

with a modern client that supports Authentication failure notifications can mean a genuine TCP connection failure.

I now see there is a doc section that provides examples of server log messages in case of authentication failures. If you don't see any of those, follow the methodology described in Troubleshooting Networking.

[naughtyGitCat]

here is rabbit mq server log after consumer start and some queue connection raise exception

2021-11-16 09:43:37.795 [info] <0.26181.1> accepting AMQP connection <0.26181.1> (192.168.1.10:42529 -> 192.168.1.9:5672)

2021-11-16 09:43:42.275 [error] <0.26181.1> closing AMQP connection <0.26181.1> (192.168.1.10:42529 -> 192.168.1.9:5672):
{handshake_timeout,frame_header}

[naughtyGitCat]

I have 10 background consumer services, when I start up the application process, some of consumers happened to fail connection

[naughtyGitCat]

RabbitMQ 3.8.25
Erlang 24.0.6
dotnet 5
both consumer and rabbitmq-server runs on centos7.4 os

[naughtyGitCat]

info.txt

[lukebakken]

@naughtyGitCat I edited or deleted your comments where you pasted large amounts of text. I first saved them in a file and attached the file to this comment. Please do not paste large amounts of text into comments on GitHub! Instead, save the text into files and attach the files.

Here is the error that matters:

2021-11-16 09:43:37.795 [info] <0.26181.1> accepting AMQP connection <0.26181.1> (192.168.1.10:42529 -> 192.168.1.9:5672)

2021-11-16 09:43:42.275 [error] <0.26181.1> closing AMQP connection <0.26181.1> (192.168.1.10:42529 -> 192.168.1.9:5672):
{handshake_timeout,frame_header

Notice that 5 seconds elapses between when that connection is established and when it times out. This means that your client application did not respond in time during connection establishment. Perhaps you are trying to start too many connections at once and your servers can't keep up.

[naughtyGitCat]

sorry about the comment, I have the same consumer logic written by python, there is no same problem, Should I share the same connection across the different consumers?

[naughtyGitCat]

add

await Task.Delay(new TimeSpan(0,0,random.Next(1,30)), stoppingToken);

on ConsumerBaseClass seems to works