Messaging System Architecture – Low Bandwidth, High Data Volume, Disconnected Systems

Problem Statement

1. You need to ingest tons of data from your on premises devices.
2. You have a low bandwidth connection to the internet / to your datacenter.
3. You have tons and tons of data that need to be transmitted in near real-time.
4. You have to transmit this data securely and near real time using your low bandwidth compression.

Solution (Use Compression for data, Use Kafka or RabbitMQ On Edge, for ingestion)

1. Step 1 – Compression – Since you have large data volumes to ingest and transfer, how about compressing the data? The compression can happen natively at the OS level – using Unix Compress or Windows GZIP.
2. Step 2 – Edge Devices with RabbitMQ  – These can serve dual purposes – ingesting and processing data as it comes in.  The processing can include the COMPRESSION.
3. Step 3  – Transmitting the data  – This needs to be a reliable messaging protocol

IoT and MQTT based communication – RabbitMQ

Microservices and RabbitMQ (or any messaging system)

Anuj holds professional certifications in Google Cloud, AWS as well as certifications in Docker and App Performance Tools such as New Relic. He specializes in Cloud Security, Data Encryption and Container Technologies.

Initial Consultation

Anuj Varma – who has written 1216 posts on Anuj Varma, Hands-On Technology Architect, Clean Air Activist.

---