What should I do with ack/nack when connection lost

[Kation]

When use AsyncEventingBasicConsumer with manual acknowledgements, we call BasicAck or BasicNack end of event.
But if connection lost when handling event, call BasicAck or BasicNack will throw exception.
After connection recovery, rabbitmq server show message still in unacked mode.
Specially if I use prefetch count with channel, it will not release count.

[lukebakken]

When an AMQP client connection is lost with pending acks, RabbitMQ will re-enqueue the messages, so something else must be going on here. I am assuming you are using the latest version of Erlang (25.1), RabbitMQ (3.10.8), and .NET client (6.4.0)

But if connection lost when handling event, call BasicAck or BasicNack will throw exception

• What is the full exception message and stack trace here?
• At this point in time, what is logged by RabbitMQ?
• How are you recovering connections?
• Can you share a complete, runnable code sample and steps to reproduce the issue? I am assuming that the following will work:
  • Run RabbitMQ, a producer and .NET consumer via docker compose
  • Configure the .NET consumer to use auto connection and topology recovery
  • In the .NET consumer, when a message is consumed but not acked, print a message indicating such
  • Run docker compose disconnect on the .NET consumer to abruptly kill the connection
  • Run docker compose connect to re-connect, which will re-connect the consumer
  • See what happens to the messsages

[lukebakken]

@Kation - https://github.com/lukebakken/rabbitmq-dotnet-client-1259

[lukebakken]

@Kation please see the above project as a starting point. I have not been able to reproduce what you describe.

git clone https://github.com/lukebakken/rabbitmq-dotnet-client-1259.git
cd rabbitmq-dotnet-client-1259
docker compose up --build

You will see the PRODUCER sending messages to the CONSUMER. The following command will instruct toxiproxy to sever the connection between the CONSUMER and RabbitMQ:

./add-consumer-toxic.sh

You will see the consumer exceptions being logged in the other terminal window. To allow the CONSUMER to re-connect, run this command:

./delete-consumer-toxic.sh

You will see the consumer re-connect, and correctly ack messages.

I have yet to see messages remain in the un-acked state. @Kation if you can provide an exact way to reproduce what you report, we would appreciate it. Otherwise, there is little we can do at this point.

[michaelklishin]

When connection (actually, consumer channel) is closed for any reason, RabbitMQ automatically requeues all deliveries pending acknowledgement.

At least one client IIRC, Bunny, uses a versioned approach where every recovery bumps client "epoch" tracks max delivery tag provided to the basic_ack and similar methods. Then, for stale deliveries (from earlier epochs) the actual delivery tag is compared to the max seen and so is the epoch, with some adjustment.

For example,

• If max seen delivery tag is 60
• and the client tries to ack 61 but the epoch has changed
• We reset max seen delivery tag and recompute the tag we need to sent to RabbitMQ as (61 - 60) = 1

I am not suggesting that this client should do that but Bunny has been doing that for some 9 years, with few bugs found and fixed. So the approach is not crazy and not too difficult to reason about.

Or you can simply handle those ack exceptions. You can also observe connection recovery events in your application code for extra logging or decision making information.