Message properties size is not accounted for when calculating message size for queue size limit in bytes

[swagataroy]

The current implementation for max-length-bytes doesn't account for the bytes in the message headers and only takes the bytes in the message body for consideration. This leaves it vulnerable to denial of service (Rabbit crash or publishers blocked) when a queue is loaded with messages containing dis-proportionally more data in headers than in the body. The queue can grow without limits -- message body size is zero, i.e. all data is in headers.

The Event Exchange plugin also follows the similar logic (putting all data into headers).

[michaelklishin]

The queue can still be limited to a fixed number of messages and TTL can still be applied to them, so this is not nearly as some may think it is.

Calculating message properties size can have a non-trivial throughput effect because it will affect every single message.

[uvzubovs]

When I loaded the queue with messages having empty body but tons of data in the headers, management UI showed queue taking 750MB+ of RAM even though total message body size remained zero. Doesn't it mean that Rabbit already has some idea of how much space is consumed by the headers?

[michaelklishin]

It does, we cannot store things to disk without knowing the total size. Specifically in your case the runtime knows how much RAM a queue process uses, which is a different piece of information that cannot not be used to calculate queue length in bytes.

[uvzubovs]

I understand. Please consider the following -- If the policy has max-length-bytes but queue memory usage starts to exceed this, isn't it already an indication that the queue is over the limit without calculating anything additional?

[uvzubovs]

Second point -- let's assume calculating message header size has non-trivial throughput effect, even though I do not understand why it would (we are adding up a dozen of integers). What is wrong with that? Maybe I would like to trade performance for reliability.

[michaelklishin]

Some will be willing to take this trade-off, others won't.

[uvzubovs]

Correct. So, is max-length-bytes-include-headers an option?

Although I would think that you should be able to take accurate measurement of the message size (body and headers included) without overhead/performance hit before the message is parsed per the AMQP spec (into the body and headers). No? I am certainly not informed of Rabbit internals, but from a general perspective.

[michaelklishin]

We need to first see what kind of overhead it has.

Message bodies are framed. Finding a way to pre-calculate message size very early is a good idea
but chances are it will break inter-node communication protocol and thus can only go into 3.7.0. 

On 23 May 2016 at 19:04:09, uvzubovs (notifications@github.com) wrote:

Correct. So, is max-length-bytes-include-headers an option?

---

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#785 (comment)

MK

Staff Software Engineer, Pivotal/RabbitMQ

[uvzubovs]

We can wait and use max-length in tandem with max-length bytes in the meantime,

[michaelklishin]

Then let's see if we can make the overhead from including full message size insignificant for most users.

[uvzubovs]

Sorry, what does this mean?