diff --git a/pipeline/outputs/output_formats.md b/pipeline/outputs/output_formats.md new file mode 100644 index 000000000..803d76224 --- /dev/null +++ b/pipeline/outputs/output_formats.md @@ -0,0 +1,137 @@ +# Output Formats + +Some of the output plugin's has a config field called `formats`, with a limited +amount of options are available (`json`, `json_stream`, `json_lines`, `msgpack`, `gelf`). + +The specifications, or reference to external documentation of the formats, is +available in this page. + +## JSON Formats + +The most custom format options are the JSON types, these are three quite +common variants of sending JSON encoded data over various API's. + +The example payloads below, are all "prettified" for readability. + +_Records shown in example payloads, are the three first commits of the fluent-bit repository_ + +### json + +Queued records encoded as a single JSON array of JSON encoded fluent-bit records. + +#### Example payload + +```json +[ + { + "timestamp": 1398282091.000, + "log": "Core: initialization and 32 bit fixes.", + "commit": "b327996bfcbc0f1f47e62c862c6f303592dfacea", + "author": "Eduardo Silva" + }, + { + "timestamp": 1422350608.000, + "log": "Initial import.", + "commit": "49269c5ec3c74411943e362cfef85052665ae97f", + "author": "Eduardo Silva" + }, + { + "timestamp": 1422418113.000, + "log": "Doc: add reference to Apache License v2.0", + "commit": "6e09478733e8273ec216328dfc921e622740695d", + "author": "Eduardo Silva" + } +] +``` + +### json_stream + +_a.k.a. Concatenated JSON_ + +Queued JSON-encoded fluent-bit records, with no separator between records. + +#### Example payload + +```json +{ + "timestamp": 1398282091.000, + "log": "Core: initialization and 32 bit fixes.", + "commit": "b327996bfcbc0f1f47e62c862c6f303592dfacea", + "author": "Eduardo Silva" +}{ + "timestamp": 1422350608.000, + "log": "Initial import.", + "commit": "49269c5ec3c74411943e362cfef85052665ae97f", + "author": "Eduardo Silva" +}{ + "timestamp": 1422418113.000, + "log": "Doc: add reference to Apache License v2.0", + "commit": "6e09478733e8273ec216328dfc921e622740695d", + "author": "Eduardo Silva" +} +``` + +### json_lines + +_a.k.a. Newline-Delimited JSON_ + +Queued JSON-encoded fluent-bit records, separated by line-breaks (`\n`). + +#### Example payload + +```json +{ + "timestamp": 1398282091.000, + "log": "Core: initialization and 32 bit fixes.", + "commit": "b327996bfcbc0f1f47e62c862c6f303592dfacea", + "author": "Eduardo Silva" +} +{ + "timestamp": 1422350608.000, + "log": "Initial import.", + "commit": "49269c5ec3c74411943e362cfef85052665ae97f", + "author": "Eduardo Silva" +} +{ + "timestamp": 1422418113.000, + "log": "Doc: add reference to Apache License v2.0", + "commit": "6e09478733e8273ec216328dfc921e622740695d", + "author": "Eduardo Silva" +} +``` + +## msgpack + +MessagePack is a binary format for serializing objects. + +Please refer to the official MessagePack format specification, found here: +https://github.com/msgpack/msgpack/blob/master/spec.md + +## GELF + +**G**raylog **E**xtended **L**og **F**ormat is a JSON based log format which is +designed to avoid the shortcomings of classic plain syslog. + +The official documentation for the GELF format can be found here: +https://archivedocs.graylog.org/en/latest/pages/gelf.html + +It uses a fixed set of keys, with the addition of adding custom keys prefixed +by one single underscore character. + +### Example payload + +Here is an example payload, copied directly from the official docs (linked above) + +```json +{ + "version": "1.1", + "host": "example.org", + "short_message": "A short message that helps you identify what is going on", + "full_message": "Backtrace here\n\nmore stuff", + "timestamp": 1385053862.3072, + "level": 1, + "_user_id": 9001, + "_some_info": "foo", + "_some_env_var": "bar" +} +```