diff --git a/docs/en/observability/synthetics-lightweight.asciidoc b/docs/en/observability/synthetics-lightweight.asciidoc index 6dee857f8f..e4dbd9bf8e 100644 --- a/docs/en/observability/synthetics-lightweight.asciidoc +++ b/docs/en/observability/synthetics-lightweight.asciidoc @@ -3,33 +3,44 @@ beta[] -To configure lightweight monitors define a set of `monitors` to check your remote hosts. -Each `monitor` item is an entry in a YAML list and begins with a dash (`-`). -You can define the type of monitor to use, the hosts to check, and other -optional settings. +In the {synthetics-app} you can monitor the status of network endpoints using the following lightweight checks: + +* *HTTP*: Monitor your website. The HTTP monitor checks to make sure specific endpoints return the correct + status code and display the correct text. +* *ICMP*: Check the availability of your hosts. The ICMP monitor uses ICMP (v4 and v6) Echo + Requests to check the network reachability of the hosts you are pinging. This will tell you whether the + host is available and connected to the network, but doesn't tell you if a service on the host is running or + not. +* *TCP*: Monitor the services running on your hosts. The TCP monitor checks individual ports + to make sure the service is accessible and running. -Lightweight monitors can be configured using any of the following: +Lightweight monitors can be configured using either the <> +or <>. -* *YAML files in {project-monitors}*. - <> and configure monitors in YAML - files in the `lightweight` directory. -* *{synthetics-app}*. - <> in {kib} and configure monitors. +[discrete] +[[synthetics-lightweight-ui]] += {synthetics-app} + +To use the UI in the {synthetics-app}, go to the {synthetics-app} in {kib} to create and configure monitors. +For step-by-step instructions, refer to <>. + +[role="screenshot"] +image::images/uptime-set-up-ui.asciidoc.png[Synthetics Create monitor UI] [discrete] -[[synthetics-monitor-types]] -= Monitor types +[[synthetics-lightweight-projects]] += {project-monitors-cap} + +To use YAML files in to create {project-monitors}, <> +and configure monitors in YAML files in the `lightweight` directory. -`http`:: Connects via HTTP and optionally verifies that the host returns the -expected response. Will use `Elastic-Heartbeat` as -the user agent product. -`icmp`:: Uses an ICMP (v4 and v6) Echo Request to ping the configured hosts. -Requires special permissions or root access. -`tcp`:: Connects via TCP and optionally verifies the endpoint by sending and/or -receiving a custom payload. +In each YAML file, define a set of `monitors` to check your remote hosts. +Each `monitor` item is an entry in a YAML list and begins with a dash (`-`). +You can define the type of monitor to use, the hosts to check, and other +optional settings. The following example configures three monitors checking via the `http`, `icmp`, and `tcp` -protocols and demonstrates how to use TCP Echo and HTTP response verification: +protocols and demonstrates how to use TCP Echo response verification: [source,yaml] ---- @@ -51,32 +62,160 @@ heartbeat.monitors: check.send: "Check" check.receive: "Check" schedule: '@every 5s' -heartbeat.scheduler: - limit: 10 ---- -[discrete] [[synthetics-monitor-options]] -= Configuration options +There are some common monitor configuration options that are the same for all lightweight monitor types. +For a complete list, refer to <>. -You can find a complete reference of available configuration options in -the {heartbeat} documentation. +Each monitor type also has additional configuration options that are specific to that type. +Refer to: -[NOTE] -==== -While the most commonly used {heartbeat} configuration options are supported, -some configuration options are not yet supported in {project-monitors} and the {synthetics-app}. -==== +* <> +* <> +* <> -There are some common monitor configuration options that are the same for all monitors. -For a complete list, refer to {heartbeat-ref}/monitor-options.html[Common monitor options]. +The `tcp` and `http` monitor types both support SSL/TLS and some proxy settings. -Each monitor type has additional configuration options that are specific to that monitor type. -Refer to: +[discrete] +[[synthetics-lightweight-common-options]] +== Common options + +You can specify the following options when defining a synthetic monitor in any location. +These options are the same for all monitors. Each monitor type has additional configuration +options that are specific to that monitor type. + +// Reference table +include::synthetics-reference/lightweight-config/common.asciidoc[] + +[discrete] +[[synthetics-lightweight-http]] +== HTTP options + +The options described here configure Synthetics to connect via HTTP and +optionally verify that the host returns the expected response. + +Valid options for HTTP monitors include <> +and the following HTTP-specific options: + +// Reference table +include::synthetics-reference/lightweight-config/http.asciidoc[] + +[discrete] +[[synthetics-lightweight-icmp]] +== ICMP options + +The options described here configure Synthetics to use ICMP (v4 and v6) Echo +Requests to check the configured hosts. On most platforms you must execute +Synthetics with elevated permissions to perform ICMP pings. + +On Linux, regular users may perform pings if the right file capabilities are set. Run +`sudo setcap cap_net_raw+eip /path/to/heartbeat` to grant Synthetics ping capabilities on Linux. +Alternatively, you can grant ping permissions to the user being used to run Synthetics. +To grant ping permissions in this way, run `sudo sysctl -w net.ipv4.ping_group_range='myuserid myuserid'`. + +Other platforms may require Synthetics to run as root or administrator to execute pings. + +Valid options for ICMP monitors include <> +and the following ICMP-specific options: + +// Reference table +include::synthetics-reference/lightweight-config/icmp.asciidoc[] + +[discrete] +[[synthetics-lightweight-tcp]] +== TCP options + +The options described here configure Synthetics to connect via TCP and +optionally verify the endpoint by sending and/or receiving a custom payload. + +Valid options for TCP monitors include <> +and the following TCP-specific options: + +// Reference table +include::synthetics-reference/lightweight-config/tcp.asciidoc[] + +[discrete] +[[synthetics-lightweight-data-types]] +== Data types reference + +Values of configuration settings are interpreted as required by Synthetics. +If a value can't be correctly interpreted as the required type - for example a +string is given when a number is required - Synthetics will fail to start up. + +[discrete] +[[synthetics-lightweight-data-bool]] +=== Boolean + +Boolean values can be either `true` or `false`. Alternative names for `true` are +`yes` and `on`. Instead of `false` the values `no` and `off` can be used. + +[source,yaml] +----- +enabled: true +disabled: false +----- + +[discrete] +[[synthetics-lightweight-data-numbers]] +=== Number + +Number values require you to enter the number _without_ single or +double quotes. + +[source,yaml] +----- +integer: 123 +negative: -1 +float: 5.4 +----- + +NOTE: Some settings only support a restricted number range. + +[discrete] +[[synthetics-lightweight-data-string]] +=== String + +In http://www.yaml.org[YAML], multiple styles of string definitions are supported: +double-quoted, single-quoted, unquoted. + +The double-quoted style is specified by surrounding the string with `"`. This +style provides support for escaping unprintable characters using `\`, but comes +at the cost of having to escape `\` and `"` characters. + +The single-quoted style is specified by surrounding the string with `'`. This +style supports no escaping (use `''` to quote a single quote). Only printable +characters can be used when using this form. + +Unquoted style requires no quotes, but does not support any escaping and can't +include any symbol that has a special meaning in YAML. + +NOTE: Single-quoted style is recommended when defining regular expressions, +event format strings, windows file paths, or non-alphabetical symbolic characters. + +[discrete] +[[synthetics-lightweight-data-duration]] +=== Duration + +Durations require a numeric value with optional fraction and required unit. +Valid time units are `ns`, `us`, `ms`, `s`, `m`, `h`. Sometimes features based +on durations can be disabled by using zero or negative durations. + +[source,yaml] +----- +duration1: 2.5s +duration2: 6h +duration_disabled: -1s +----- + +[discrete] +[[synthetics-lightweight-data-regex]] +=== Regular expression -* {heartbeat-ref}/monitor-http-options.html[HTTP monitors] -* {heartbeat-ref}/monitor-tcp-options.html[ICMP monitors] -* {heartbeat-ref}/monitor-tcp-options.html[TCP monitors] +Regular expressions are special strings that are compiled into regular +expressions at load time. -The `tcp` and `http` monitor types both support SSL/TLS and some proxy -settings. +As regular expressions and YAML use `\` for escaping +characters in strings, it's highly recommended to use single quoted strings when +defining regular expressions. When single quoted strings are used, the `\` character +is not interpreted by YAML parser as an escape symbol. diff --git a/docs/en/observability/synthetics-reference/lightweight-config/common.asciidoc b/docs/en/observability/synthetics-reference/lightweight-config/common.asciidoc new file mode 100644 index 0000000000..65a7a30ddc --- /dev/null +++ b/docs/en/observability/synthetics-reference/lightweight-config/common.asciidoc @@ -0,0 +1,140 @@ +:hardbreaks-option: + +[%header] +|=== +| Option (type) | Description + +// type +| [[monitor-type]] *`type`* +(`"http"` \| `"icmp"` \| `"tcp"`) +a| *Required*. The type of monitor to run. One of: + +* `http`: Connects via HTTP and optionally verifies that the host returns the expected response. +* `icmp`: Uses an ICMP (v4 and v6) Echo Request to ping the configured hosts. Requires special permissions or root access. +* `tcp`: Connects via TCP and optionally verifies the endpoint by sending and/or receiving a custom payload. + +//////////////////////// +id +//////////////////////// +| [[monitor-id]] *`id`* +(<>) +a| *Required*. A unique identifier for this configuration. This should not change with edits to the monitor configuration regardless of changes to any config fields. + +*Examples*: + +[source,yaml] +---- +id: uploader-service +---- + +[source,yaml] +---- +id: http://example.net +---- + +[NOTE] +==== +When querying against indexed monitor data this is the field you will be aggregating with. It appears in the exported fields as `monitor.id`. + +If you do not set an `id` explicitly, the monitor's config will be hashed and a generated value will be used. This value will change with any options change to this monitor making aggregations over time between changes impossible. For this reason, it's recommended that you set this manually. +==== + +//////////////////////// +name +//////////////////////// +| [[monitor-name]] *`name`* +(<>) +a| Human readable name for this monitor. + +*Examples*: + +[source,yaml] +---- +name: Uploader service +---- + +[source,yaml] +---- +name: Example website +---- + +//////////////////////// +service.name +//////////////////////// +| [[monitor-service_name]] *`service.name`* +(<>) +a| APM service name for this monitor. Corresponds to the `service.name` ECS field. Set this when monitoring an app that is also using APM to enable integrations between Synthetics and APM data in Kibana. + +//////////////////////// +enabled +//////////////////////// +| [[monitor-enabled]] *`enabled`* +(<>) +a| Whether the module is enabled. + +*Default*: `true` + +*Example*: + +[source,yaml] +---- +enabled: false +---- + +//////////////////////// +schedule +//////////////////////// +| [[monitor-schedule]] *`schedule`* +(<>) +a| *Required*. The task schedule. + +NOTE: Schedules with less than 1 minute resolution will be saved to the nearest minute. For example, `@every 5s` will be changed to `@every 60s` when the monitor is pushed using the CLI. + +*Example*: +Run the task every 5 minutes from the time the monitor was started. + +[source,yaml] +---- +schedule: @every 5m +---- + +//////////////////////// +timeout +//////////////////////// +| [[monitor-timeout]] *`timeout`* +(<>) +a| The total running time for each ping test. This is the total time allowed for testing the connection and exchanging data. + +*Default*: `16s` + +*Example*: + +[source,yaml] +---- +timeout: 2m +---- + +//////////////////////// +tags +//////////////////////// +| [[monitor-tags]] *`tags`* +(list of <>s) +a| A list of tags that will be sent with the monitor event. + +*Examples*: + +[source,yaml] +---- +tags: + - tag one + - tag two +---- + +[source,yaml] +---- +tags: ["tag one", "tag two"] +---- + +|=== + +:!hardbreaks-option: diff --git a/docs/en/observability/synthetics-reference/lightweight-config/http.asciidoc b/docs/en/observability/synthetics-reference/lightweight-config/http.asciidoc new file mode 100644 index 0000000000..d4c12f1b1a --- /dev/null +++ b/docs/en/observability/synthetics-reference/lightweight-config/http.asciidoc @@ -0,0 +1,227 @@ +:hardbreaks-option: + +[%header] +|=== +| Option (type) | Description + +// hosts +| [[monitor-http-hosts]] *`hosts`* +(list of <>s) +a| *Required*. A list of URLs to ping. + +//////////////////////// +max_redirects +//////////////////////// +| [[monitor-http-max_redirects]] *`max_redirects`* +(<>) +a| The total number of redirections Synthetics will follow. + +By default, Synthetics will not follow redirects, but will report the status of the redirect. If set to a number greater than `0`, Synthetics will follow that number of redirects. + +When this option is set to a value greater than `0`, the `monitor.ip` field will no longer be reported, as multiple DNS requests across multiple IPs may return multiple IPs. Fine-grained network timing data will also not be recorded, as with redirects that data will span multiple requests. Specifically the fields `http.rtt.content.us`, `http.rtt.response_header.us`, `http.rtt.total.us`, `http.rtt.validate.us`, `http.rtt.write_request.us` and `dns.rtt.us` will be omitted. + +*Default*: `0` + +//////////////////////// +proxy_url +//////////////////////// +| [[monitor-http-proxy_url]] *`proxy_url`* +(<>) +a| The HTTP proxy URL. This setting is optional. + +*Example*: + +[source,yaml] +http://proxy.mydomain.com:3128 + +//////////////////////// +username +//////////////////////// +| [[monitor-http-username]] *`username`* +(<>) +a| The username for authenticating with the server. The credentials are passed with the request. This setting is optional. + +You need to specify credentials when your `check.response` settings require it. For example, you can check for a 403 response (`check.response.status: [403]`) without setting credentials. + +//////////////////////// +password +//////////////////////// +| [[monitor-http-password]] *`password`* +(<>) +a| The password for authenticating with the server. This setting is optional. + +//////////////////////// +ssl +//////////////////////// +| [[monitor-http-ssl]] *`ssl`* +({heartbeat-ref}/configuration-ssl.html[SSL]) +a| The TLS/SSL connection settings for use with the HTTPS endpoint. If you don't specify settings, the system defaults are used. + +*Example*: + +[source,yaml] +---- +- type: http + id: my-http-service + name: My HTTP Service + hosts: ["https://myhost:443"] + schedule: '@every 5s' + ssl: + certificate_authorities: ['/etc/ca.crt'] + supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"] +---- + +//////////////////////// +headers +//////////////////////// +| [[monitor-http-headers]] *`headers`* +(<>) +a| Controls the indexing of the HTTP response headers `http.response.body.headers` field. Set `response.include_headers` to `false` to disable. + +*Default*: `true` + +//////////////////////// +response +//////////////////////// +| [[monitor-http-response]] *`response`* +a| Controls the indexing of the HTTP response body contents to the `http.response.body.contents` field. + +//////////////////////// +response.include_body +//////////////////////// +*`include_body`* (`"on_error"` \| `"never"` \| `"always"`):: +Set `response.include_body` to one of the options listed below. ++ +* `on_error`: Include the body if an error is encountered during the check. This is the default. +* `never`: Never include the body. +* `always`: Always include the body with checks. + +//////////////////////// +check +//////////////////////// +| [[monitor-http-check]] *`check`* +a| +//////////////////////// +// check.request +//////////////////////// +*`request`*:: An optional `request` to send to the remote host. Under `check.request`, specify these options: ++ +-- +//////////////////////// +// check.request.method +//////////////////////// +*`method`* (`"HEAD"` \| `"GET"` \| `"POST"` \| `"OPTIONS"`)::: The HTTP method to use. + +//////////////////////// +// check.request.headers +//////////////////////// +*`headers`* (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers[HTTP headers])::: + A dictionary of additional HTTP headers to send. By default Synthetics will set the 'User-Agent' header to identify itself. + +//////////////////////// +// check.request.body +//////////////////////// +*`body`* (<>)::: + Optional request body content. +-- ++ +*Example*: This monitor POSTs an `x-www-form-urlencoded` string to the endpoint `/demo/add`. ++ +[source,yaml] +---- +check.request: + method: POST + headers: + 'Content-Type': 'application/x-www-form-urlencoded' + # urlencode the body: + body: "name=first&email=someemail%40someemailprovider.com" +---- + +//////////////////////// +// check.response +//////////////////////// +*`response`*:: The expected `response`. ++ +-- +Under `check.response`, specify these options: + +//////////////////////// +// check.response.status +//////////////////////// +*`status`* (list of <>s)::: +A list of expected status codes. 4xx and 5xx codes are considered `down` by default. Other codes are considered `up`. ++ +*Example*: ++ +[source,yaml] +---- +check.response: + status: [200, 201] +---- + +//////////////////////// +// check.response.headers +//////////////////////// +*`headers`* (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers[HTTP headers])::: +The required response headers. + +//////////////////////// +// check.response.body.positive +//////////////////////// +*`body.positive`* (list of <>s)::: +A list of regular expressions to match the body output. Only a single expression needs to match. ++ +*Example*: ++ +This monitor examines the response body for the strings 'foo' or 'Foo': ++ +[source,yaml] +---- +check.response: + status: [200, 201] + body: + positive: + - foo + - Foo +---- + +//////////////////////// +// check.response.body.negative +//////////////////////// +*`body.negative`* (list of <>s)::: +A list of regular expressions to match the body output negatively. Return match failed if single expression matches. HTTP response bodies of up to 100MiB are supported. ++ +This monitor examines match successfully if there is no 'bar' or 'Bar' at all, examines match failed if there is 'bar' or 'Bar' in the response body: ++ +*Example*: ++ +[source,yaml] +---- +check.response: + status: [200, 201] + body: + negative: + - bar + - Bar +---- ++ +*Example*: ++ +This monitor examines match successfully only when 'foo' or 'Foo' in body AND no 'bar' or 'Bar' in body: ++ +[source,yaml] +---- +check.response: + status: [200, 201] + body: + positive: + - foo + - Foo + negative: + - bar + - Bar +---- +-- +|=== + +:!hardbreaks-option: diff --git a/docs/en/observability/synthetics-reference/lightweight-config/icmp.asciidoc b/docs/en/observability/synthetics-reference/lightweight-config/icmp.asciidoc new file mode 100644 index 0000000000..7d360753d4 --- /dev/null +++ b/docs/en/observability/synthetics-reference/lightweight-config/icmp.asciidoc @@ -0,0 +1,37 @@ +:hardbreaks-option: + +[%header] +|=== +| Option (type) | Description + +// hosts +| [[monitor-icmp-hosts]] *`hosts`* +(list of <>s) +a| *Required*. A list of hosts to ping. + +*Example*: + +[source,yaml] +---- +hosts: ["myhost"] +---- + +//////////////////////// +wait +//////////////////////// +| [[monitor-icmp-wait]] *`wait`* +(<>) +a| The duration to wait before emitting another ICMP Echo Request if no response is received. + +*Default*: `1s` + +*Example*: + +[source,yaml] +---- +wait: 1m +---- + +|=== + +:!hardbreaks-option: diff --git a/docs/en/observability/synthetics-reference/lightweight-config/tcp.asciidoc b/docs/en/observability/synthetics-reference/lightweight-config/tcp.asciidoc new file mode 100644 index 0000000000..582583c700 --- /dev/null +++ b/docs/en/observability/synthetics-reference/lightweight-config/tcp.asciidoc @@ -0,0 +1,126 @@ +:hardbreaks-option: + +[%header] +|=== +| Option (type) | Description + +// hosts +| [[monitor-tcp-hosts]] *`hosts`* +(list of <>s) +a| *Required*. A list of hosts to ping. The entries in the list can be: + +* *A plain host name, such as `localhost`, or an IP address.* + If you specify this option, you must also specify a value for <>. If the monitor is {heartbeat-ref}/configuration-ssl.html[configured to use SSL], Synthetics establishes an SSL/TLS-based connection. Otherwise, it establishes a plain TCP connection. + +* *A hostname and port, such as `localhost:12345`.* + Synthetics connects to the port on the specified host. If the monitor is {heartbeat-ref}/configuration-ssl.html[configured to use SSL], Synthetics establishes an SSL/TLS-based connection. Otherwise, it establishes a TCP connection. + +* *A full URL using the syntax `scheme://:[port]`*, where: +** `scheme` is one of `tcp`, `plain`, `ssl` or `tls`. If `tcp` or `plain` is specified, Synthetics establishes a TCP connection even if the monitor is configured to use SSL. If `tls` or `ssl` is specified, Synthetics establishes an SSL connection. However, if the monitor is not configured to use SSL, the system defaults are used (currently not supported on Windows). +** `host` is the hostname. +** `port` is the port number. If `port` is missing in the URL, the <> setting is required. + +*Examples*: + +[source,yaml] +---- +hosts: ["localhost"] +---- + +[source,yaml] +---- +hosts: ["localhost:8000"] +---- + +[source,yaml] +---- +hosts: ["tcp://localhost:8000"] +---- + +//////////////// +ports +//////////////// +| [[monitor-tcp-ports]] *`ports`* +(list of <>s) +a| A list of ports to ping if the host specified in <> does not contain a port number. It is generally preferable to use a single value here, since each port will be monitored using a separate `id`, with the given `id` value, used as a prefix in the Synthetics data, and the configured `name` shared across events sent via this check. + +*Example*: + +[source,yaml] +---- +hosts: ["localhost"] +ports: [80, 9200, 5044] +---- + +//////////////// +check +//////////////// +| [[monitor-tcp-check]] *`check`* +a| An optional payload string to send to the remote host and the expected answer. If no payload is specified, the endpoint is assumed to be available if the connection attempt was successful. If `send` is specified without `receive`, any response is accepted as OK. If `receive` is specified without `send`, no payload is sent, but the client expects to receive a payload in the form of a "hello message" or "banner" on connect. + +*Example*: + +[source,yaml] +---- +check.send: 'Hello World' +check.receive: 'Hello World' +---- + +[source,yaml] +---- +check: + send: 'Hello World' + receive: 'Hello World' +---- + + +//////////////// +proxy_url +//////////////// +| [[monitor-tcp-proxy_url]] *`proxy_url`* +a| The URL of the SOCKS5 proxy to use when connecting to the server. The value must be a URL with a scheme of socks5://. + +If the SOCKS5 proxy server requires client authentication, then a username and password can be embedded in the URL. + +When using a proxy, hostnames are resolved on the proxy server instead of on the client. You can change this behavior by setting the `proxy_use_local_resolver` option. + +*Examples*: + +A proxy URL that requires client authentication: + +[source,yaml] +---- +proxy_url: socks5://user:password@socks5-proxy:2233 +---- + + +//////////////// +proxy_use_local_resolver +//////////////// +| [[monitor-tcp-proxy_use_local_resolver]] *`proxy_use_local_resolver`* +(<>) +a| A Boolean value that determines whether hostnames are resolved locally instead of being resolved on the proxy server. The default value is `false`, which means that name resolution occurs on the proxy server. + +*Default*: `false` + +//////////////// +ssl +//////////////// +| [[monitor-tcp-ssl]] *`ssl`* +({heartbeat-ref}/configuration-ssl.html[SSL]) +a| The TLS/SSL connection settings. If the monitor is {heartbeat-ref}/configuration-ssl.html[configured to use SSL], it will attempt an SSL handshake. If `check` is not configured, the monitor will only check to see if it can establish an SSL/TLS connection. This check can fail either at TCP level or during certificate validation. + +*Example*: + +[source,yaml] +---- +ssl: + certificate_authorities: ['/etc/ca.crt'] + supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"] +---- + +Also see {heartbeat-ref}/configuration-ssl.html[Configure SSL] for a full description of the `ssl` options. + +|=== + +:!hardbreaks-option: