Message Brokers: Translators for the Digital Age

How many messages does the technology in your organization deal with on any given day? How many different applications, software programs and hardware devices do you utilize in processing those messages?

If the answer is ‘a lot’, it may be time to ensure all those variables can talk to one another as efficiently as possible. An open source Message Broker could be the answer.

How Message Queuing Works

The first message brokers solved a big problem for stock traders, whose workstations in the 1980s were cluttered with multiple terminals connected by mountains of cable to each type of information the trader needed to do their job. Teknekron Software Systems (which eventually became TIBCO) saw an opportunity to replace the various terminals and their siloed applications with a software bus. 

The concept utilized a ‘publish–subscribe’ messaging pattern - publishers send messages to categories rather than to specific recipients, and subscribers receive messages only from categories that are of interest. The result was that a single workstation could display programs by plugging into the software bus.

Origins of Message Brokers

TIBCO MQ software dominated the financial services industry initially. Then IBM developed competing message queuing software, with its WebSphere MQ commercial platform.  These proprietary systems have pricing structures which are barriers for many organizations, resulting in open source software products becoming attractive options in developer circles. 

Open Source Message Brokers: RabbitMQ

RabbitMQ was one of the first and is the mostly widely deployed of the open source options.

• RabbitMQ is extremely low cost to set up and to operate. 
• It can be set up on virtually any kind of server, making it attractive for small businesses, start-ups, or large companies.
• It supports multiple messaging protocols, message queuing, delivery acknowledgement, flexible routing to queues, and multiple exchange types.
• Because of the way RabbitMQ is set up, the producer is not aware of the consumer. And the consumer is not aware of the producer.
  • They are only aware of the middleware to which they are connected
  • Delivery acknowledgement is important. 
  • RabbitMQ needs to know that the right message has been delivered to the right consumer.
• RabbitMQ is also extremely fast, capable of handling more than one million messages per second.
• It is not designed to store messages long-term. 

Open Source Message Brokers: Apache Kafka

Apache Kafka is another open source option. This software started out at LinkedIn to connect different internal systems. 

• Apache Kafka is designed for high volume publish-subscribe messages and streams.
• It is meant to be durable, fast, and scalable.
• Apache Kafka provides message storage.
• It is expensive, in part because it requires multiple servers.
• Apache Kafka has message persistence built into it, which can be a key consideration.
• Apache Kafka is especially attractive for large scale enterprise applications.

To Recap

Message brokers are popular in an increasing number of applications, especially where several different servers are interconnected with tightly coupled consumers. This solution consolidates information from all kinds of sources down to one and provides a way to communicate with one another without consumers being tightly coupled.