|
1 | 1 | # Forward
|
2 | 2 |
|
3 |
| -_Forward_ is the protocol used by [Fluent Bit](http://fluentbit.io) and [Fluentd](http://www.fluentd.org) to route messages between peers. This plugin implements the input service to listen for Forward messages. |
| 3 | +_Forward_ is the protocol used by [Fluentd](http://www.fluentd.org) to route messages between peers. The **forward** output plugin allows to provide interoperability between [Fluent Bit](http://fluentbit.io) and [Fluentd](http://fluentd.org). There are not configuration steps required besides to specify where [Fluentd](http://fluentd.org) is located, it can be in the local host or a in a remote machine. |
| 4 | + |
| 5 | +This plugin offers two different transports and modes: |
| 6 | + |
| 7 | +* Forward \(TCP\): It uses a plain TCP connection. |
| 8 | +* Secure Forward \(TLS\): when TLS is enabled, the plugin switch to Secure Forward mode. |
4 | 9 |
|
5 | 10 | ## Configuration Parameters
|
6 | 11 |
|
7 |
| -The plugin supports the following configuration parameters: |
| 12 | +The following parameters are mandatory for either Forward for Secure Forward modes: |
| 13 | + |
| 14 | +| Key | Description | Default | |
| 15 | +| :--- | :--- | :--- | |
| 16 | +| Host | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | 127.0.0.1 | |
| 17 | +| Port | TCP Port of the target service. | 24224 | |
| 18 | +| Time\_as\_Integer | Set timestamps in integer format, it enable compatibility mode for Fluentd v0.12 series. | False | |
| 19 | +| Upstream | If Forward will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/configuration/upstream_servers.md) documentation section. | | |
| 20 | +| Tag | Overwrite the tag as we transmit. This allows the receiving pipeline start fresh, or to attribute source. | | |
| 21 | +| Send\_options | Always send options \(with "size"=count of messages\) | False | |
| 22 | +| Require\_ack\_response | Send "chunk"-option and wait for "ack" response from server. Enables at-least-once and receiving server can control rate of traffic. \(Requires Fluentd v0.14.0+ server\) | False | |
| 23 | + |
| 24 | +## Secure Forward Mode Configuration Parameters |
| 25 | + |
| 26 | +When using Secure Forward mode, the [TLS](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/configuration/tls_ssl.md) mode requires to be enabled. The following additional configuration parameters are available: |
8 | 27 |
|
9 | 28 | | Key | Description | Default |
|
10 | 29 | | :--- | :--- | :--- |
|
11 |
| -| Listen | Listener network interface. | 0.0.0.0 | |
12 |
| -| Port | TCP port to listen for incoming connections. | 24224 | |
13 |
| -| Buffer\_Max\_Size | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](../administration/configuring-fluent-bit/unit-sizes.md) specification. | _Buffer\_Chunk\_Size_ | |
14 |
| -| Buffer\_Chunk\_Size | By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_. The value must be according to the [Unit Size](../administration/configuring-fluent-bit/unit-sizes.md) specification. | 32KB | |
| 30 | +| Shared\_Key | A key string known by the remote Fluentd used for authorization. | | |
| 31 | +| Empty\_Shared\_Key | Use this option to connect to Fluentd with a zero-length secret. | False | |
| 32 | +| Username | Specify the username to present to a Fluentd server that enables `user_auth`. | | |
| 33 | +| Password | Specify the password corresponding to the username. | | |
| 34 | +| Self\_Hostname | Default value of the auto-generated certificate common name \(CN\). | localhost | |
| 35 | +| tls | Enable or disable TLS support | Off | |
| 36 | +| tls.verify | Force certificate validation | On | |
| 37 | +| tls.debug | Set TLS debug verbosity level. It accept the following values: 0 \(No debug\), 1 \(Error\), 2 \(State change\), 3 \(Informational\) and 4 Verbose | 1 | |
| 38 | +| tls.ca\_file | Absolute path to CA certificate file | | |
| 39 | +| tls.crt\_file | Absolute path to Certificate file. | | |
| 40 | +| tls.key\_file | Absolute path to private Key file. | | |
| 41 | +| tls.key\_passwd | Optional password for tls.key\_file file. | | |
| 42 | + |
| 43 | +## Forward Setup |
| 44 | + |
| 45 | +Before proceeding, make sure that [Fluentd](http://fluentd.org) is installed in your system, if it's not the case please refer to the following [Fluentd Installation](http://docs.fluentd.org/v0.12/categories/installation) document and go ahead with that. |
| 46 | + |
| 47 | +Once [Fluentd](http://fluentd.org) is installed, create the following configuration file example that will allow us to stream data into it: |
| 48 | + |
| 49 | +```text |
| 50 | +<source> |
| 51 | + type forward |
| 52 | + bind 0.0.0.0 |
| 53 | + port 24224 |
| 54 | +</source> |
| 55 | +
|
| 56 | +<match fluent_bit> |
| 57 | + type stdout |
| 58 | +</match> |
| 59 | +``` |
15 | 60 |
|
16 |
| -## Getting Started |
| 61 | +That configuration file specifies that it will listen for _TCP_ connections on the port _24224_ through the **forward** input type. Then for every message with a _fluent\_bit_ **TAG**, will print the message to the standard output. |
17 | 62 |
|
18 |
| -In order to receive Forward messages, you can run the plugin from the command line or through the configuration file: |
| 63 | +In one terminal launch [Fluentd](http://fluentd.org) specifying the new configuration file created \(in\_fluent-bit.conf\): |
| 64 | + |
| 65 | +```bash |
| 66 | +$ fluentd -c test.conf |
| 67 | +2017-03-23 11:50:43 -0600 [info]: reading config file path="test.conf" |
| 68 | +2017-03-23 11:50:43 -0600 [info]: starting fluentd-0.12.33 |
| 69 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1' |
| 70 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-docker' version '0.1.0' |
| 71 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-elasticsearch' version '1.4.0' |
| 72 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flatten-hash' version '0.2.0' |
| 73 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.0.4' |
| 74 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-influxdb' version '0.2.8' |
| 75 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4' |
| 76 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-mongo' version '0.7.10' |
| 77 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-out-http' version '0.1.3' |
| 78 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-parser' version '0.6.0' |
| 79 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-record-reformer' version '0.7.0' |
| 80 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.1' |
| 81 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-stdin' version '0.1.1' |
| 82 | +2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-td' version '0.10.27' |
| 83 | +2017-03-23 11:50:43 -0600 [info]: adding match pattern="fluent_bit" type="stdout" |
| 84 | +2017-03-23 11:50:43 -0600 [info]: adding source type="forward" |
| 85 | +2017-03-23 11:50:43 -0600 [info]: using configuration file: <ROOT> |
| 86 | + <source> |
| 87 | + type forward |
| 88 | + bind 0.0.0.0 |
| 89 | + port 24224 |
| 90 | + </source> |
| 91 | + <match fluent_bit> |
| 92 | + type stdout |
| 93 | + </match> |
| 94 | +</ROOT> |
| 95 | +2017-03-23 11:50:43 -0600 [info]: listening fluent socket on 0.0.0.0:24224 |
| 96 | +``` |
| 97 | + |
| 98 | +## Fluent Bit + Forward Setup <a id="forward_setup"></a> |
| 99 | + |
| 100 | +Now that [Fluentd](http://fluentd.org) is ready to receive messages, we need to specify where the **forward** output plugin will flush the information using the following format: |
| 101 | + |
| 102 | +```text |
| 103 | +bin/fluent-bit -i INPUT -o forward://HOST:PORT |
| 104 | +``` |
19 | 105 |
|
20 |
| -### Command Line |
| 106 | +If the **TAG** parameter is not set, the plugin will set the tag as _fluent\_bit_. Keep in mind that **TAG** is important for routing rules inside [Fluentd](http://fluentd.org). |
21 | 107 |
|
22 |
| -From the command line you can let Fluent Bit listen for _Forward_ messages with the following options: |
| 108 | +Using the [CPU](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/input/cpu.md) input plugin as an example we will flush CPU metrics to [Fluentd](http://fluentd.org): |
23 | 109 |
|
24 | 110 | ```bash
|
25 |
| -$ fluent-bit -i forward -o stdout |
| 111 | +$ bin/fluent-bit -i cpu -t fluent_bit -o forward://127.0.0.1:24224 |
26 | 112 | ```
|
27 | 113 |
|
28 |
| -By default the service will listen an all interfaces \(0.0.0.0\) through TCP port 24224, optionally you can change this directly, e.g: |
| 114 | +Now on the [Fluentd](http://fluentd.org) side, you will see the CPU metrics gathered in the last seconds: |
29 | 115 |
|
30 | 116 | ```bash
|
31 |
| -$ fluent-bit -i forward://192.168.3.2:9090 -o stdout |
| 117 | +2017-03-23 11:53:06 -0600 fluent_bit: {"cpu_p":0.0,"user_p":0.0,"system_p":0.0,"cpu0.p_cpu":0.0,"cpu0.p_user":0.0,"cpu0.p_system":0.0,"cpu1.p_cpu":0.0,"cpu1.p_user":0.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0} |
| 118 | +2017-03-23 11:53:07 -0600 fluent_bit: {"cpu_p":2.25,"user_p":2.0,"system_p":0.25,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":1.0,"cpu1.p_user":1.0,"cpu1.p_system":0.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":3.0,"cpu3.p_user":2.0,"cpu3.p_system":1.0} |
| 119 | +2017-03-23 11:53:08 -0600 fluent_bit: {"cpu_p":1.75,"user_p":1.0,"system_p":0.75,"cpu0.p_cpu":2.0,"cpu0.p_user":1.0,"cpu0.p_system":1.0,"cpu1.p_cpu":3.0,"cpu1.p_user":1.0,"cpu1.p_system":2.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0} |
| 120 | +2017-03-23 11:53:09 -0600 fluent_bit: {"cpu_p":4.75,"user_p":3.5,"system_p":1.25,"cpu0.p_cpu":4.0,"cpu0.p_user":3.0,"cpu0.p_system":1.0,"cpu1.p_cpu":5.0,"cpu1.p_user":4.0,"cpu1.p_system":1.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":5.0,"cpu3.p_user":4.0,"cpu3.p_system":1.0} |
32 | 121 | ```
|
33 | 122 |
|
34 |
| -In the example the Forward messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090. |
| 123 | +So we gathered [CPU](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/input/cpu.md) metrics and flushed them out to [Fluentd](http://fluentd.org) properly. |
35 | 124 |
|
36 |
| -### Configuration File |
| 125 | +## Fluent Bit + Secure Forward Setup <a id="secure_forward_setup"></a> |
37 | 126 |
|
38 |
| -In your main configuration file append the following _Input_ & _Output_ sections: |
| 127 | +> DISCLAIMER: the following example do not consider the generation of certificates for a proper usage of production environments. |
| 128 | +
|
| 129 | +Secure Forward aims to provide a secure channel of communication with the remote Fluentd service using [TLS](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/configuration/tls_ssl.md). Above there is a minimalist configuration for testing purposes. |
| 130 | + |
| 131 | +### Fluent Bit |
| 132 | + |
| 133 | +Paste this content in a file called _flb.conf_: |
| 134 | + |
| 135 | +```text |
| 136 | +[SERVICE] |
| 137 | + Flush 5 |
| 138 | + Daemon off |
| 139 | + Log_Level info |
39 | 140 |
|
40 |
| -```python |
41 | 141 | [INPUT]
|
42 |
| - Name forward |
43 |
| - Listen 0.0.0.0 |
44 |
| - Port 24224 |
45 |
| - Buffer_Chunk_Size 32KB |
46 |
| - Buffer_Max_Size 64KB |
| 142 | + Name cpu |
| 143 | + Tag cpu_usage |
47 | 144 |
|
48 | 145 | [OUTPUT]
|
49 |
| - Name stdout |
50 |
| - Match * |
| 146 | + Name forward |
| 147 | + Match * |
| 148 | + Host 127.0.0.1 |
| 149 | + Port 24284 |
| 150 | + Shared_Key secret |
| 151 | + Self_Hostname flb.local |
| 152 | + tls on |
| 153 | + tls.verify off |
51 | 154 | ```
|
52 | 155 |
|
53 |
| -## Testing |
| 156 | +### Fluentd |
54 | 157 |
|
55 |
| -Once Fluent Bit is running, you can send some messages using the _fluent-cat_ tool \(this tool is provided by [Fluentd](http://www.fluentd.org): |
| 158 | +Paste this content in a file called _fld.conf_: |
56 | 159 |
|
57 |
| -```bash |
58 |
| -$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | fluent-cat my_tag |
| 160 | +```text |
| 161 | +<source> |
| 162 | + @type secure_forward |
| 163 | + self_hostname myserver.local |
| 164 | + shared_key secret |
| 165 | + secure no |
| 166 | +</source> |
| 167 | +
|
| 168 | +<match **> |
| 169 | + @type stdout |
| 170 | +</match> |
59 | 171 | ```
|
60 | 172 |
|
61 |
| -In [Fluent Bit](http://fluentbit.io) we should see the following output: |
| 173 | +If you're using Fluentd v1, set up it as below: |
| 174 | + |
| 175 | +```text |
| 176 | +<source> |
| 177 | + @type forward |
| 178 | + <transport tls> |
| 179 | + cert_path /etc/td-agent/certs/fluentd.crt |
| 180 | + private_key_path /etc/td-agent/certs/fluentd.key |
| 181 | + private_key_passphrase password |
| 182 | + </transport> |
| 183 | + <security> |
| 184 | + self_hostname myserver.local |
| 185 | + shared_key secret |
| 186 | + </security> |
| 187 | +</source> |
| 188 | +
|
| 189 | +<match **> |
| 190 | + @type stdout |
| 191 | +</match> |
| 192 | +``` |
62 | 193 |
|
63 |
| -```bash |
64 |
| -$ bin/fluent-bit -i forward -o stdout |
65 |
| -Fluent-Bit v0.9.0 |
66 |
| -Copyright (C) Treasure Data |
| 194 | +### Test Communication |
67 | 195 |
|
68 |
| -[2016/10/07 21:49:40] [ info] [engine] started |
69 |
| -[2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224 |
70 |
| -[0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}] |
| 196 | +Start Fluentd: |
| 197 | + |
| 198 | +```text |
| 199 | +$ fluentd -c fld.conf |
| 200 | +``` |
| 201 | + |
| 202 | +Start Fluent Bit: |
| 203 | + |
| 204 | +```text |
| 205 | +$ fluent-bit -c flb.conf |
71 | 206 | ```
|
| 207 | + |
| 208 | +After five seconds, Fluent Bit will write the records to Fluentd. In Fluentd output you will see a message like this: |
| 209 | + |
| 210 | +```text |
| 211 | +2017-03-23 13:34:40 -0600 [info]: using configuration file: <ROOT> |
| 212 | + <source> |
| 213 | + @type secure_forward |
| 214 | + self_hostname myserver.local |
| 215 | + shared_key xxxxxx |
| 216 | + secure no |
| 217 | + </source> |
| 218 | + <match **> |
| 219 | + @type stdout |
| 220 | + </match> |
| 221 | +</ROOT> |
| 222 | +2017-03-23 13:34:41 -0600 cpu_usage: {"cpu_p":1.0,"user_p":0.75,"system_p":0.25,"cpu0.p_cpu":1.0,"cpu0.p_user":1.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":1.0,"cpu1.p_system":1.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0} |
| 223 | +2017-03-23 13:34:42 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.75,"system_p":0.0,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0} |
| 224 | +2017-03-23 13:34:43 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.25,"system_p":0.5,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0} |
| 225 | +2017-03-23 13:34:44 -0600 cpu_usage: {"cpu_p":5.0,"user_p":3.25,"system_p":1.75,"cpu0.p_cpu":4.0,"cpu0.p_user":2.0,"cpu0.p_system":2.0,"cpu1.p_cpu":8.0,"cpu1.p_user":5.0,"cpu1.p_system":3.0,"cpu2.p_cpu":4.0,"cpu2.p_user":3.0,"cpu2.p_system":1.0,"cpu3.p_cpu":4.0,"cpu3.p_user":2.0,"cpu3.p_system":2.0} |
| 226 | +``` |
| 227 | + |
0 commit comments