Skip to content

Use rabbit_queue_type API in shovels #14098

Open
@ansd

Description

@ansd
  • Use rabbit_queue_type API directly if shovel source or destination is in the local cluster.
  • Use credit API v2 of the rabbit_queue_type API. This ensures that the shovel shovels as many messages as can be handled by the destination. For example, if the destination grants lots of link credit to the shovel, the shovel can also pass a high number of link credit to the source, therefore enabling high throughput shovelling. If the destination grants few link credit to the shovel, the shovel passes only few credits to the source. This ensures that the shovel itself isn’t flooded with messages.
  • If the source queue is in the local cluster, the shovel should run co-located on a node with a queue member to reduce intra cluster network traffic. (This is currently being worked on separately by @michaelklishin.)
  • Likewise, if the destination is a queue in the local cluster, the shovel should ideally run the queue leader node.
  • If both source and destination are in the local cluster, no message cross protocol translation should take place. This ensures no message metadata gets lost and guarantees the immutability of the AMQP 1.0 bare message.
  • If a message needs to be re-published with a protocol that’s different in how the message was stored in the local source queue, a translation takes place using the mc API according to our well documented conversion rules. This solves AMQP 1.0 Application Properties lost when shovelling to an AMQP 0.9.1 broker #13957
  • If a message gets shovelled only within the local cluster, the message should be neither encoded/serialised nor decoded/deserialised.
  • If a message gets shovelled from a local queue to a remote broker, the message should get encoded/serialised once (as opposed to multiple times as for example done in Shovel: use message containers #14096).
  • If a message gets shovelled from a remote broker to a local destination, the message should be decoded/deserialised once.
  • The shovel should use AMQP address format v2.
  • The shovel should guarantee the immutability of the AMQP bare message. For example if a message is shovelled from a 3rd party AMQP 1.0 broker to RabbitMQ, the integrity of the bare message must not be violated. This solves ActiveMQ to RabbitMQ shovel results in amqp header data being inserted in message data #12455

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions