Queues are a common computer-science concept: a system that stores an ordered, linear sequence of Items. Usually the Items are Messages which is just fancy-pants container your data + other special metadata.

Messages have a size limit, though. For example, Azure Storage Queues have a size limit of 64KB for Plain Text messages or 48KB for Base64 Encoded.

So if you try and place your content into a queue and the content is too big, then you will get an error.

A Hybrid Queue is the concept of throwing anything onto a normal Queue and if the size of the Message (which contains your content) is too big, it then automatically puts your content into a Blob (which can contain any size**) and then stores the reference to the blob item in the queue!

Both directions (sending a message to the queue and popping a message off the queue) handle the smarts if the message is too big and needs to retrieve the contents from the blob.

Under the hood, if the content is not a Primitive Type like a string or int, etc (more or less), then we convert the content to a Json representation of the source item. So if you have a custom POCO, it's serialized to Json, then stored in the queue or blob, based on the final size.

📌 Attribution: This original idea was engineered by Jason Ashman who created the original Hybrid Queue (Private Homely code). This is an updated version based entirely on his original concept and effort.

Note about message size limits: the Azure Queue SDK doesn't allow you to ask what Encoding is used for messages for the queue. Base64? Plain Text? As such, we need to assume the worst and set the max size limit to the lower-sized Base64 limit of 48KB. Anything larger will then be stored in a blob.

Package Name: WorldDomination.SimpleAzure.Storage.HybridQueues
CLI: install-package WorldDomination.SimpleAzure.Storage.HybridQueues

// _queueClient, _blobContainerClient and _logger would be injected via your IoC/DI
// These are normally setup in elsewhere, like in your program.cs, etc.
// e.g.
//    _queueClient = new QueueClient(connectionStringText, "test-queue");
//    _blobContainerClient = new BlobContainerClient(connectionStringText, "test-container");

// Create the Hybrid Queue.
var hybridQueue = new HybridQueue(_queueClient, _blobContainerClient, logger);

// Content to store on a queue.
var message = "hello";

// Adding the content to the queue.
await hybridQueue.AddMessageAsync(message, cancellationToken);

// The queue message will contain the value 'hello'. Nothing will be placed into the blob container.

// Create our POCO.
public record User(string Name, int Age);
var user = new User("Pure Krome", 100);

// Create the Hybrid Queue.
var hybridQueue = new HybridQueue(_queueClient, _blobContainerClient, logger);

// Adding the content to the queue.
await hybridQueue.AddMessageAsync(user, cancellationToken);

// The queue message will contain the value '{"name":"Pure Krome", "age": 100}'

// Generate some really long content larger than the queue size.
const string chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";

var length = queueClient.MessageMaxBytes + 1; // Just larger than the queue message max size.
var longName = new string("a", length);

// Create our POCO
public record User(string Name, int Age);
var user = new User(longName, 100);

// Create the Hybrid Queue.
var hybridQueue = new HybridQueue(_queueClient, _blobContainerClient, logger);

// Adding the content to the queue.
await hybridQueue.AddMessageAsync(user, cancellationToken);


// The queue message will contain the value <Some Guid>'
// The blob container will contain a blob with the Json representation of that POCO.
// The Blob will have the name <Some Guid>, so the Queue Message "links" to this Blob.

• AddMessageAsync : Single item.
• AddMessagesAsync : multiple items added at once. Batching if the collection of items is large.
• GetMessageAsync : Single item.
• GetMessagesAsync : Multiple messages.
• DeleteMessageAsync : Single message. Knows if it needs to remove it from queue and blob (if required).

Yep - contributions are always welcome. Please read the contribution guidelines first.

If you wish to participate in this repository then you need to abide by the code of conduct.

Yes! Please use the Issues section to provide feedback - either good or needs improvement 🆒