About data serialization

# About data serialization

Transporter needs a Serializer Module which serializes & deserializes the transferable data. The default Serializer is the JsonSerializer but there are several built-in Serializers can convert messages into MessagePack, BSON, CBOR, SMILE, Amazon ION or other binary formats.

Cross-language clusters: only JSON and MessagePack interoperate

A Java node and a Node.js node can understand each other only with the JsonSerializer (the default) or the MsgPackSerializer — these are the only two formats the Node.js implementation also speaks. Every binary serializer below (BSON, CBOR, Amazon ION, SMILE and the Java-object serializer) is Java-only. Both nodes in a cluster must use the same serializer, so if you pick one of the Java-only formats for performance, a Node.js node will silently fail to decode every packet — there is no error, the two nodes simply never see each other. The cross-language common denominator is JSON (or MessagePack).

# JSON Serializer

This is the built-in default Serializer. It serializes the packets to JSON string and deserializes the received JSON bytes to Tree (opens new window) objects. The performance of JSON Serializers in Java and JavaScript is very good, JSON serialization is usually faster than most binary Serializers. This Serializer is compatible with the JavaScript/Node.js version of Moleculer.

NatsTransporter transporter = new NatsTransporter("nats://nats.server:4222");

// Do not have to set it because this is the default
transporter.setSerializer(new JsonSerializer());

ServiceBroker broker = ServiceBroker.builder()
                                    .nodeID("server-1")
                                    .transporter(transporter)
                                    .readers("jackson,gson")
                                    .writers("jackson,gson")
                                    .build();

The "readers" and "writers" parameters are used to specify the JSON API to be used by Moleculer for deserialization / serialization. This is important if you have multiple JSON implementations on the classpath. Several implementations can be specified in order of importance, separated by commas. If not specified, ServiceBroker will automatically try to choose the fastest JSON API. The values of the "readers" and "writers" parameters are listed below:

Reader/writer ID	JSON API and Dependency
"bson"	BSON (MongoDB) (opens new window)
"dsl"	DSLJson (opens new window)
"genson"	Genson (opens new window)
"gson"	Google Gson (opens new window)
"jackson"	Jackson JSON (opens new window)
"jodd"	Jodd Json (opens new window)
"johnzon"	Apache Johnzon (opens new window)
"jsonio"	JsonIO (opens new window)
"nano"	NanoJson (opens new window)
"smart"	Json-smart (opens new window)
"ion"	Amazon Ion (opens new window)
"builtin"	Built-in JSON parser (no dependencies)

So, for example, if you want to use a "Gson" implementation, put the implementation reference in the "dependencies" block of the (build.gradle or pom.xml) build script, then set "readers" and "writers" to "gson". To verify, type "info" command into the REPL console. The "info" command will display the current Moleculer configuration, including the the JSON API which is in use.

# MessagePack Serializer

Built-in MsgPack (opens new window) Serializer. MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves. This Serializer is compatible with the JavaScript version of Moleculer.

transporter.setSerializer(new MsgPackSerializer());

MessagePack dependencies

To use MessagePack Serializer, add the following dependency to the build script:
group: 'org.msgpack', name: 'jackson-dataformat-msgpack', version: '0.9.9' (opens new window)

# BSON Serializer

Java-only — not compatible with the Node.js implementation; use it only in a Java-to-Java cluster.

Built-in BSON (opens new window) Serializer. BSON, short for Binary JSON, is a binary-encoded serialization of JSON-like documents. Like JSON, BSON supports the embedding of documents and arrays within other documents and arrays.

transporter.setSerializer(new BsonSerializer());

BSON dependencies

To use BSON Serializer, add the following dependency to the build script:
group: 'de.undercouch', name: 'bson4jackson', version: '2.18.0' (opens new window)

# CBOR Serializer

Java-only — not compatible with the Node.js implementation; use it only in a Java-to-Java cluster.

Built-in CBOR (opens new window) Serializer. CBOR is based on the wildly successful JSON data model: numbers, strings, arrays, maps (called objects in JSON), and a few values such as false, true, and null.

transporter.setSerializer(new CborSerializer());

CBOR dependencies

To use CBOR Serializer, add the following dependency to the build script:
group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-cbor', version: '2.19.0' (opens new window)

# Amazon ION Serializer

Java-only — not compatible with the Node.js implementation; use it only in a Java-to-Java cluster.

Built-in ION (opens new window) Serializer. Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The binary representation is efficient to store, transmit, and skip-scan parse.

transporter.setSerializer(new IonSerializer());

Amazon ION dependencies

To use ION Serializer, add the following dependency to the build script:
group: 'com.amazon.ion', name: 'ion-java', version: '1.11.10' (opens new window)

# SMILE Serializer

Java-only — not compatible with the Node.js implementation; use it only in a Java-to-Java cluster.

Built-in SMILE (opens new window) Serializer. SMILE is a computer data interchange format based on JSON. It can also be considered as a binary serialization of generic JSON data model, which means that tools that operate on JSON may be used with SMILE as well, as long as proper encoder/decoder exists for tool to use. Compared to JSON, SMILE is both more compact and more efficient to process.

transporter.setSerializer(new SmileSerializer());

SMILE dependencies

To use SMILE Serializer, add the following dependency to the build script:
group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-smile', version: '2.19.0' (opens new window)

# Custom Serializer

Custom Serializer module can be created. To make your own Serializer, you need to derive it from the services.moleculer.serializer.Serializer superclass, and implement the "write" and "read" methods.

Create custom Serializer

public class CustomSerializer extends Serializer {

    // --- SERIALIZE TREE TO BYTE ARRAY ---

    public byte[] write(Tree value) throws Exception {
        Object content = value.asObject();
        // Write Java Object into byte array...
        // The "content" is "java.util.Map" or "java.util.List".
    }

    // --- DESERIALIZE BYTE ARRAY TO TREE ---

    public Tree read(byte[] source) throws Exception {
        Object content = // Read Java Object from "source"...
        return new CheckedTree(content);
    }

}

Use custom Serializer

transporter.setSerializer(new CustomSerializer());

# Message-level encryption

The BlockCipherSerializer is capable of encrypting data packets using the specified algorithm. You can give it a "parent" Serializer, if not specified it uses JSON serialization before encryption.

BlockCipherSerializer serializer = new BlockCipherSerializer();

// Same as "aes-256-cbc" in Node.js
serializer.setAlgorithm("AES/CBC/PKCS5Padding"); // Algorithm
serializer.setPassword("12345678901234567890123456789012"); // 32 bytes of password
serializer.setIv("1234567890123456"); // 16 bytes of IV block - required for AES/CBC

// Create Transporter and Service Broker
Transporter transporter = new NatsTransporter("localhost");
transporter.setSerializer(serializer);
ServiceBroker broker = ServiceBroker.builder()
                                    .nodeID("node1")
                                    .transporter(transporter)
                                    .build();

In Node.js-based Moleculer, code with similar functionality looks like this:

// JavaScript code
const broker = new ServiceBroker({
    transporter: "NATS",
    middlewares: [
        Middlewares.Transmit.Encryption("12345678901234567890123456789012", // Password
                                        "aes-256-cbc", // Algorithm
                                        "1234567890123456") // 16 bytes of IV block
    ]
});

256-bit keys work out of the box on Java 17

Unlimited-strength cryptography has been enabled by default since Java 8u161 / Java 9, so on the Java 17 baseline no JCE policy files are needed for AES-256. (On long-obsolete JREs you had to install the Unlimited Strength Jurisdiction Policy Files by hand — that step is no longer required.)

# Compressing messages

The DeflaterSerializer can compress messages larger than the specified size. It can also have a "parent" Serializer, if not specified it uses JSON serialization before compression.

Transporter transporter = new NatsTransporter("localhost");
transporter.setSerializer(new DeflaterSerializer(512));

// Create Service Broker
ServiceBroker broker = ServiceBroker.builder()
                                    .nodeID("node1")
                                    .transporter(transporter)
                                    .build();

In Node.js-based Moleculer, code with similar functionality looks like this:

const broker = new ServiceBroker({
    transporter: "NATS",
    middlewares: [
        Middlewares.Transmit.Compression({ method: "deflateRaw",
                                           threshold: "512b" }),
    ]
});

WARNING

Using compression reduces performance, so use it only on slow networks.

# Chaining Serializers

By chaining Serializers, you can combine multiple Serializers. In the example below, the data is first serialized using the MessagePack algorithm. Deflater then compresses large data packets (larger than 1024 bytes) and then encrypts the compressed packet using the AES algorithm:

Transporter natsTransporter = new NatsTransporter("localhost");

// Create Serializer-chain
MsgPackSerializer msgPack = new MsgPackSerializer();
DeflaterSerializer deflater = new DeflaterSerializer(msgPack);
BlockCipherSerializer cipher = new BlockCipherSerializer(deflater, "1234567890123456");

// Set Serializer-chain to Transporter
natsTransporter.setSerializer(cipher);

// Create Service Broker
ServiceBroker broker = ServiceBroker.builder()
                                    .nodeID("node1")
                                    .transporter(natsTransporter)
                                    .build();

Double encryption

Chaining allows multiple encryption methods to be applied to a data packet. In the example below, the data is first encrypted using the "ARCFOUR" and then the "Blowfish" algorithm:

// Create JSON Serializer
JsonSerializer jsonSerializer = new JsonSerializer();
        
// First cipher
BlockCipherSerializer first = new BlockCipherSerializer(jsonSerializer,
                                                        "password1",
                                                        "ARCFOUR");
// Second cipher
BlockCipherSerializer second = new BlockCipherSerializer(first,
                                                         "password2",
                                                         "Blowfish");

// Set Serializer-chain to Transporter
transporter.setSerializer(second);

// Create Service Broker
ServiceBroker broker = ServiceBroker.builder()
                                    .nodeID("node1")
                                    .transporter(transporter)
                                    .build();

← Transporters Load balancing →