Transformers

Transformers allow logical operations to be performed on messages as they pass down the pipeline. There are currently two classes of Transformer, native and JavaScript. Native Transformers are written in Go and are hard-coded to perform a particular function on a message as quickly as possible. JavaScript transformers take user's own JavaScript functions and then apply them to the message.

Given a pipeline such as:

t.Source(source).Save(sink);

Adding a Transformer would look like this:

t.Source(source).Transform(transformer({"param":"value"})).Save(sink);

For example, to call the omit transformer to drop an "internalid" field use:

t.Source(source).Transform(omit({"fields":["internalid"]})).Save(sink);

Native Transformers

omit() - Removes top-level fields from a message. This is ideal for thinning down a document by simply removing some elements. readme

pick() - Selects only particular top-level fields from a message and discards all other top-level fields. This is ideal for aggressively thinning a document by selecting only the fields you need. readme

rename() - Renames top-level fields. This is useful for the simple restructuring of a document. readme

skip() - Skips messages based on a test of a top-level field against a value. Often, the test for whether a message should move on to the next stage is a simple test so this native transformer makes it simple. readme

pretty() - Pretty prints the message to the log for debugging. When creating a pipeline is is useful to be able to see the contents of messages as they pass through it. The pretty() native transformer does nothing to the message itself but does print out a formatted version to the log. readme

JavaScript Transformers

JavaScript Transformers send messages into a JavaScript script for processing. To call a JavaScript Transformer, use js() as the transformer function. This function takes a "filename" parameter, the value of which should be the name of a file with a JavaScript function named transform(msg).

There are two JavaScript engines currently in Transformer; the older otto and newer goja. The js() transformer function is actually an alias to the Goja JavaScript engine. The only external difference between the two is how functions should be created.

Goja style

function transform(msg) {
    return msg
}

Otto style

module.exports = function(msg) {
    return msg;
};

In both cases, the msg is returned unmodified for continued processing. Messages have a number of fields which represent the data being carried in the message, the operation the data is associated with, the namespace which the data belongs to and a timestamp for then it was ingested into the Transporter.