Merge branch 'master' into doc-unix-dgram

Author: zsoltgyulai94

.github/workflows/jekyll-gh-pages.yml

@@ -29,7 +29,7 @@ concurrency:
 jobs:
   # Build job
   build:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -105,7 +105,7 @@ jobs:
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: build
     steps:
       - name: Deploy to GitHub Pages

_data/external_links.yml

@@ -608,6 +608,11 @@ gh-es:
   url: https://github.com/syslog-ng/syslog-ng/tree/master/scl/elasticsearch
   title: [ "Elasticsearch configuration snippet on GitHub" ]
 
+gh-es-ds:
+  id: gh-es-ds
+  url: https://github.com/syslog-ng/syslog-ng/blob/master/scl/elasticsearch/elastic-datastream.conf
+  title: [ "Elasticsearch datastream configuration snippet on GitHub" ]
+  
 gh-fortigate:
   id: gh-fortigate
   url: https://github.com/syslog-ng/syslog-ng/blob/master/scl/fortigate/fortigate.conf
@@ -796,6 +801,11 @@ ebpf:
   url: https://ebpf.io/
   title: [ "Extended Berkeley Packet Filter" ]
 
+elastic-ds:
+  id: elastic-ds
+  url: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
+  title: [ "Elasticsearch data streams" ]
+
 gcp-ser:
   id: gcp-ser
   url: https://cloud.google.com/iam/docs/service-account-overview

_includes/doc/admin-guide/options/headers-gRPC.md

@@ -0,0 +1,17 @@
+## headers()
+
+|  Type: |      arrow list|
+|Default:|           empty|
+
+Available in syslog-ng OSE 4.8 and later versions.
+
+*Description:* Adds custom gRPC headers to each RPC call. Currently only static header names and values are supported.
+
+```config
+headers(
+    "organization" => "org-name"
+    "stream-name" => "org-stream"
+  )
+```
+
+> *Copyright © 2024 Axoflow*

doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md

@@ -71,6 +71,10 @@ destination {
 
 *Description:* This option configures the upper limit of in-flight gRPC requests per worker. It is advisd to set this value in the range of 10s or 100s when there are a high number of clients sending simultaneously. In an optimzed solution, the number of `workers()` and `concurrent-requests()` is greater than or equal to the number of clients. However, this can cause an increase in memory usage.
 
+## keep-hostname()
+
+The `syslog-ng-otlp()` and `opentelemetry()` sources ignore this option and use the hostname from the message as the `${HOST}`.
+
 ## log-fetch-limit()
 
 |   Type:|       number|

doc/_admin-guide/060_Sources/175_syslog-otlp/000_otlp_source_options.md

@@ -129,7 +129,9 @@ source {
 
 {% include doc/admin-guide/options/host-override.md %}
 
-{% include doc/admin-guide/options/keep-hostname.md %}
+## keep-hostname()
+
+The `syslog-ng-otlp()` and `opentelemetry()` sources ignore this option and use the hostname from the message as the `${HOST}`.
 
 {% include doc/admin-guide/options/keep-timestamp.md %}
 

doc/_admin-guide/060_Sources/180_System/README.md

@@ -50,7 +50,10 @@ such messages without losing any information to CIM-aware applications
 |              | Note that on Linux, the so-rcvbuf() option of the system() source is automatically set to 8192.        |
 |              | If the host is running under systemd, {{ site.product.short_name }} reads directly from the systemd journal file using the systemd-journal() source.                        |
 |              | If the kernel of the host is version 3.5 or newer, and /dev/kmsg is seekable, {{ site.product.short_name }} will use that instead of /proc/kmsg, using the multi-line-mode(indented), keep-timestamp(no), and the format(linux-kmsg)options.                      |
-|              | If {{ site.product.short_name }} is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsg or /proc/kmsg files.                                 |
+|              |If {{ site.product.short_name }} is running in a jail or a Linux Container (LXC), it will not read from the `/dev/kmsg` or `/proc/kmsg` files.
+|              |With systemd: `systemd-journal();`
+|              |Without systemd, on kernel 3.5 or newer: `unix-dgram("/dev/log"); file("/dev/kmsg" program-override("kernel") flags(kernel) format("linux-kmsg") keep-timestamp(no));`
+|              |Without systemd, on kernels older than 3.5: `unix-dgram("/dev/log"); file("/proc/kmsg" program-override("kernel") flags(kernel) keep-timestamp(no));`
 | macOS        |     file("/var/log/system.log" follow-freq(1));      |
 |              | **NOTE:** Starting with version 3.7, the {{ site.product.short_name }} system() driver automatically extracts the msgid  from the message (if available), and stores it in the .solaris.msgid macro. To extract the msgid from the message without using the system()driver, use the **extract-solaris-msgid()** parser. You can find the exact source of the Solaris parser on GitHub.|
 | NetBSD       |     unix-dgram("/var/run/log");                      |

doc/_admin-guide/070_Destinations/035_elasticsearch-datastream/README.md

@@ -0,0 +1,31 @@
+---
+title: 'elasticsearch-datastream: Elasticsearch data streams'
+short_title: elasticsearch-datastream
+id: adm-dest-es-datastream
+description: >-
+    From {{ site.product.short_name }} 4.8 and later versions, you  can send messages and metrics to Elasticsearch data streams to store your log and metrics data as time series data.
+---
+
+**Declaration**
+
+```config
+destination d_elastic_data_stream {
+  elasticsearch-datastream(
+    url("https://elastic-endpoint:9200/my-data-stream/_bulk")
+    user("elastic")
+    password("ba253DOn434Tc0pY22OI")
+  );
+};
+```
+
+This driver is a reusable configuration snippet configured to send log messages using the http() driver using a template. You can find the Elasticsearch datastream configuration snippet on GitHub.
+
+## Prerequisites
+
+* An account for Elasticsearch datastreams with a username and a password.
+
+## Options
+
+Elasticsearch datastream is an HTTP based driver, hence it utilizes the HTTP destination options.
+
+> *Copyright © 2024 Axoflow*

doc/_admin-guide/070_Destinations/045_Google_bigQuery/000_bigquery_dest_options.md

@@ -44,6 +44,8 @@ Available in {{ site.product.short_name }} 4.5 and later versions.
 
 {% include doc/admin-guide/options/frac-digits.md %}
 
+{% include doc/admin-guide/options/headers-gRPC.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-alive.md %}

doc/_admin-guide/070_Destinations/125_Loki/001_Loki_options.md

@@ -68,6 +68,8 @@ destination {
 
 {% include doc/admin-guide/options/channel-args.md %}
 
+{% include doc/admin-guide/options/headers-gRPC.md %}
+
 {% include doc/admin-guide/options/gRPC-keep-alive.md %}
 
 ## labels()

doc/_admin-guide/070_Destinations/157_OpenTelemetry/000_opentelemetry-destination-options.md

@@ -75,4 +75,6 @@ Available in syslog-ng OSE 4.5 and later versions.
 
 *Description:* This option enables compression in gRPC requests. Currently, only the deflate compression method is supported.
 
+{% include doc/admin-guide/options/headers-gRPC.md %}
+
 {% include doc/admin-guide/options/workers.md %}

doc/_admin-guide/090_Global_options/000_Global_options.md

@@ -338,7 +338,7 @@ driver.
 
 ## stats()
 
-Available in {{ site.product.short_name }} 4.1 and later versions.
+Available in {{ site.product.short_name }} 4.1 and later versions, introduced by Axoflow.
 
 *Description:* The stats() option is an aggregated collection of statistic-related sub-options.
 
@@ -358,12 +358,16 @@ options {
 
 The following sub-options are available within the stats() option:
 
-- freq()
+### freq()
 
-|  Accepted values:|   number|
+|  Accepted values:|   number (seconds)|
 |Default:|           600|
 
-- level()
+*Description:* The period between two STATS messages in seconds. STATS
+are log messages sent by syslog-ng, containing statistics about dropped
+log messages. Set to **0** to disable the STATS messages.
+
+### level()
 
 |  Accepted values:|   0, 1, 2, 3|
 |Default:|           0|
@@ -379,8 +383,16 @@ The following sub-options are available within the stats() option:
 - Level 3 contains detailed statistics based on various message parameters like facility, severity, or tags.
 
 **NOTE:** Level 2 and 3 increase the memory requirements and CPU load. For details on message statistics, see Statistics of syslog-ng.
+{: .notice--info}
+
+### lifetime()
 
-- max-dynamics()
+|  Accepted values:|   number (seconds)|
+|Default:|           N/A|
+
+*Description:* Dynamic counters in metrics are pruned after lifetime expires. Note that orphaned counters are not pruned (you can prune them by running `syslog-ng-ctl stats --remove-orphans`)
+
+### max-dynamics()
 
 |  Accepted values:|   number|
 |Default:|           N/A|
@@ -392,27 +404,28 @@ The following sub-options are available within the stats() option:
     If this option is not used, dynamic counters are not limited. This can be useful in cases where you are extremely interested in dynamic counters, and use these statistics extensively.
 
     ![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** In some cases, there might be even millions of dynamic counters.
+    {: .notice--warning}
 
 - **Limited dynamic counter clusters:**
 
-    To limit dynamic counters, enter a number, and only a maximum of <number> counters will be registered in the statistics.
+    To limit dynamic counters, enter a number, and only a maximum of \<number\> counters will be registered in the statistics.
 
     In practice, this means dynamic counter clusters. A program name produces one dynamic counter cluster, that can include several counters, such as processed, stamp, and so on.
 
     **Example: Limiting dynamic counter clusters 1:**
 
-    If stats-max-dynamics() is set to 1, and 2 programs send messages, only one of these programs will be tracked in the dynamic counters, but it will have more than one counters.
-
+    If max-dynamics() is set to 1, and 2 programs send messages, only one of these programs will be tracked in the dynamic counters, but it will have more than one counters.
 
     **Example: Limiting dynamic counter clusters 2:**
 
-    If you have 500 clients, and set stats-max-dynamics() to 1000, you will have enough number of counters reserved for these clients, but at the same time, you limit the use of your resources and therefore protect your system from being overloaded.
+    If you have 500 clients, and set max-dynamics() to 1000, you will have enough number of counters reserved for these clients, but at the same time, you limit the use of your resources and therefore protect your system from being overloaded.
 
 - **No dynamic counters:**
 
     To disable dynamic counters completely, set the value of this option to 0. This is the recommended value if  statistics are not used, or if dynamic counters are irrelevant (for example, the number of logs arriving from programs).
 
-**NOTE:** If a lower value is set to stats-max-dynamics() (or, any limiting value, if this option has not been configured before) and {{ site.product.short_name }} is restarted, the changes are only applied after stats-freq() time has passed. That is, the previously allocated dynamic clusters are only removed after this time.
+**NOTE:** If a lower value is set to max-dynamics() (or, any limiting value, if this option has not been configured before) and {{ site.product.short_name }} is restarted, the changes are only applied after freq() time has passed. That is, the previously allocated dynamic clusters are only removed after this time.
+{: .notice--info}
 
 ## so-passcred()
 
@@ -442,19 +455,19 @@ Possible values:
 
 ## stats-freq() (DEPRECATED)
 
-This is a deprecated legacy option. Use the stats() option.
+This is a deprecated legacy option. Use stats(freq()) instead.
 
 ## stats-level() (DEPRECATED)
 
-This is a deprecated legacy option. Use the stats() option.
+This is a deprecated legacy option. Use stats(level()) instead.
 
 ## stats-max-dynamics() (DEPRECATED)
 
-This is a deprecated legacy option. Use the stats() option.
+This is a deprecated legacy option. Use stats(max-dynamics()) instead.
 
 ## sync() or sync-freq() (DEPRECATED)
 
-This is a deprecated legacy option. Use the stats() option.
+Obsolete aliases for flush-lines().
 
 ## threaded()
 

doc/_admin-guide/110_Template_and_rewrite/000_Customize_message_format/004_Macros_of_syslog-ng.md

@@ -503,6 +503,7 @@ following macros are available in {{ site.product.short_name }} version 3.9 and
 - .tls.x509_o: The value of the Organization field.
 
 - .tls.x509_ou: The value of the Organization Unit field.
+- .tls.x509_fp: The key fingerprint of the peer, when the trusted-key() option is used. Available in {{ site.product.short_name }} 4.8 and later versions.
 
 ## ${UNIQID}
 

doc/_admin-guide/120_Parser/002_Parsing_key-value_pairs/000_kv_parser_options.md

@@ -1,6 +1,6 @@
 ---
 title: Options of key=value parsers
-parser: kv-parser
+parser: kv
 prefix: kv
 id: adm-parser-kv-opt
 description: >-

doc/_admin-guide/120_Parser/023_db_parser/001_Using_pattern_databases/002_Correlating_log_messages.md

@@ -69,7 +69,7 @@ Note the following points about timeout values:
 
     Consider the following two messages:
 
-    ><38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage
+    ><38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage\
     ><38>1990-01-01T14:46:25 customhostname program6[1234]: program6 testmessage
 
     If the context-timeout is 10 seconds and {{ site.product.short_name }} receives the

doc/_admin-guide/130_Correlating_log_messages/002_grouping-by_options.md

@@ -162,6 +162,15 @@ sends the triggered message (specified in the
 aggregate(), and clears the
 context.
 
+### Example: Configuring trigger() with a 100 message context length
+
+```config
+parser p_grouping-by {grouping-by(
+    ...
+    trigger("$(context-length)" > "100")
+);};
+```
+
 ## where()
 
 *Description:* Specifies a filter condition. Messages not matching the

doc/_admin-guide/190_The_syslog-ng_manual_pages/007_syslog-ng_control_tool.md

@@ -223,8 +223,8 @@ The stats command has the following options:
     >dst.file;#anon-destination0#0;/tmp/2021-08-17.log;o;processed;156
     >dst.file;#anon-destination0#0;/tmp/2021-08-18.log;a;processed;961
 
-    **NOTE:** The stats-lifetime() can be used to do the same
-    automatically and periodically, but currently stats-lifetime()
+    **NOTE:** The stats(lifetime()) can be used to do the same
+    automatically and periodically, but currently stats(lifetime())
     removes only dynamic counters that have a timestamp field set.
     {: .notice--info}