Skip to content

Commit 0af6cea

Browse files
committed
Added doc of new follow-method() option and related changes
Signed-off-by: Hofi <[email protected]>
1 parent bdb3b55 commit 0af6cea

File tree

9 files changed

+63
-21
lines changed

9 files changed

+63
-21
lines changed

_data/navigation.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -793,7 +793,7 @@ admin-guide-nav:
793793
subnav:
794794
- title: "Options of CSV parsers"
795795
url: /admin-guide/120_Parser/004_CSV_parser/000_CSV_parser_options
796-
- title: "Parsing dates and timestamps"
796+
- title: "Dates and timestamps parsing"
797797
url: /admin-guide/120_Parser/005_dates_parser/README
798798
subnav:
799799
- title: "Options of date-parser() parsers"
@@ -908,16 +908,16 @@ admin-guide-nav:
908908
url: /admin-guide/120_Parser/012_Linux_audit_parser/000_Linux_audit_parser_options
909909
- title: "MariaDB parser"
910910
url: /admin-guide/120_Parser/013_mariadb_parser
911-
- title: "metrics-probe"
911+
- title: "metrics probe"
912912
url: /admin-guide/120_Parser/014_metrics_probe_parser/README
913913
subnav:
914914
- title: "metrics-probe options"
915915
url: /admin-guide/120_Parser/014_metrics_probe_parser/000_metrics_probe_options
916-
- title: "Parsing tags"
916+
- title: "Tags parser"
917917
url: /admin-guide/120_Parser/014_tags_parser
918918
- title: "Netskope parser"
919919
url: /admin-guide/120_Parser/015_netskope_parser
920-
- title: "panos-parser"
920+
- title: "panos parser"
921921
url: /admin-guide/120_Parser/015_panos_parser/README
922922
subnav:
923923
- title: "Message format parsed by panos-parser()"

_includes/doc/admin-guide/options/follow-freq.md

+10-6
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,16 @@
33
| Type:| number|
44
|Default: | 1|
55

6-
*Description:* Indicates that the source should be checked periodically.
7-
This is useful for files which always indicate readability, even though
8-
no new content were appended (e.g. regular file system files). If this value
9-
is higher than zero, syslog-ng will not attempt to use kqueue or ivykis file change
10-
notification methods on the file (poll(), epoll(), etc.), but checks whether
11-
the file changed every time the follow-freq() interval (in seconds) has elapsed.\
6+
*Description:* This is useful for files that always indicate readability, even though
7+
no new content has been appended (e.g., regular file system files).\
8+
In pre-4.9 versions of {{ site.product.short_name }}, where follow-method() was not yet available,
9+
this indicated that the source should be checked periodically. If this value is higher than zero,
10+
{{ site.product.short_name }} will not attempt to use kqueue or ivykis file change notification
11+
methods on the file (poll(), epoll(), etc.), but instead checks whether the file has changed every time the follow-freq()
12+
interval (in seconds) has elapsed.\
13+
In version 4.9 and later, this option only has an effect if follow-method() is set to `legacy` or `poll`,
14+
in which case it is used as in pre-4.9 versions. For any other follow-method() values, the value of this option is ignored.
15+
1216
Floating-point numbers (for example, **1.5**) can be used as well.
1317

1418
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
## follow-method()
2+
3+
|Accepted values:| legacy \| inotify \| poll \| system |
4+
|Default: | legacy |
5+
6+
*Description:* This option is available in version 4.9 and higher and its value controls how {{ site.product.short_name }} will follow file changes.\
7+
The default `legacy` mode preserves the pre-4.9 version file follow-mode behavior of {{ site.product.short_name }}, which is based on the value of follow-freq().\
8+
The `poll` value forces {{ site.product.short_name }} to poll for file changes at the interval specified by the monitor-freq() option, even if a more efficient method (such as `inotify` or `kqueue`) is available.\
9+
If `inotify` is selected and supported by the platform, {{ site.product.short_name }} uses it to detect changes in source files. This is the most efficient and least resource-consuming option available on Linux for regular files.\
10+
The `system` value will use system poll methods (via ivykis) like `port-timer` `port` `dev_poll` `epoll-timerfd` `epoll` `kqueue` `ppoll` `poll` and `uring`. For more information about how to control the system polling methods used, see How content changes are followed in file() and wildcard-file() sources.
11+
12+
**NOTE:** Using `inotify` also requires setting monitor-method() to `inotify`.
13+
{: .notice--info}
14+
15+
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), the monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
1+
![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), follow-method(), monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
22
{: .notice--warning}

doc/_admin-guide/060_Sources/020_File/000_File_source_options.md

+2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ The file() driver has the following options:
1717

1818
{% include doc/admin-guide/options/follow-freq.md %}
1919

20+
{% include doc/admin-guide/options/follow-method.md %}
21+
2022
{% include doc/admin-guide/options/hook.md %}
2123

2224
{% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/020_File/001_File_following.md

+25-10
Original file line numberDiff line numberDiff line change
@@ -19,16 +19,22 @@ Setting the `auto` method will select the best available method on the given OS.
1919

2020
Detecting file content changes involves more factors that can affect resource usage and overall performance. {{ site.product.short_name }} uses two main methods to monitor file content changes.
2121

22-
The first method is automatically selected if the follow-freq() option has a value greater than 0. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires. Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
22+
The first method is automatically selected in pre-4.9 version of {{ site.product.short_name }} if the follow-freq() option has a value greater than 0, or in version 4.9 or higher if follow-method() `legacy` or `poll` selected. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires.\
23+
Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
2324

24-
The second method is activated if the follow-freq() option is set to 0. It uses ivykis poll methods, sometimes resembling the poll method for directory change watching described above (with its performance penalties), but often working similarly to the `inotify` or `kqueue` version (with seamless performance).
25+
The second method is activated in pre-4.9 versions of {{ site.product.short_name }} if the follow-freq() option is set to 0, or in version 4.9 and higher if follow-method() is set to inotify or system.\
26+
If inotify is available on the system and selected in both follow-method() and monitor-method(), it will be used, resulting in significantly lower resource usage (especially on Linux). This option is accessible only on version 4.9 or later.\
27+
Otherwise, {{ site.product.short_name }} uses ivykis polling methods, which sometimes resemble the polling method used for directory change watching described above (with its associated performance penalties), but can also operate similarly to the kqueue version (offering seamless performance).
2528

2629
The following table shows which method is selected in different cases.
2730

2831
<table border="0" cellspacing="0" cellpadding="0" width="1397">
2932
<tr>
30-
<td width="99" class="right-edged-col">
31-
<p align="center">follow-freq()</p>
33+
<td width="174" class="right-edged-col">
34+
<p align="center">pre-4.9 version<br>or<br>follow-method(legacy)<br>or<br>follow-method(poll)</p>
35+
</td>
36+
<td width="151" class="right-edged-col">
37+
<p align="center">version 4.9 or higher<br>follow-method()</p>
3238
</td>
3339
<td width="200" colspan="2" class="right-edged-col">
3440
<p align="center">file follow method</p>
@@ -44,8 +50,11 @@ The following table shows which method is selected in different cases.
4450
</td>
4551
</tr>
4652
<tr>
47-
<td width="99" rowspan="8" class="right-edged-col">
48-
<p align="center">0</p>
53+
<td width="174" rowspan="8" class="right-edged-col">
54+
<p align="center">follow-freq(0)</p>
55+
</td>
56+
<td width="150" rowspan="7" class="right-edged-col">
57+
<p align="center">system</p>
4958
</td>
5059
<td width="150" rowspan="7" class="right-edged-col">
5160
<p align="center">ivykis poll</p>
@@ -148,7 +157,10 @@ The following table shows which method is selected in different cases.
148157
</td>
149158
</tr>
150159
<tr>
151-
<td width="150" colspan="2" class="right-edged-col">
160+
<td width="150" class="right-edged-col">
161+
<p align="center">inotify</p>
162+
</td>
163+
<td width="150" colspan="2" class="right-edged-col">
152164
<p align="center">inotify from ivykis directory monitor</p>
153165
</td>
154166
<td width="151" class="right-edged-col">
@@ -162,8 +174,11 @@ The following table shows which method is selected in different cases.
162174
</td>
163175
</tr>
164176
<tr>
165-
<td width="99" rowspan="3" class="right-edged-col">
166-
<p align="center">&gt; 0</p>
177+
<td width="174" rowspan="3" class="right-edged-col">
178+
<p align="center">follow-freq(n)<br> n &gt; 0</p>
179+
</td>
180+
<td width="150" rowspan="3" class="right-edged-col">
181+
<p align="center">poll</p>
167182
</td>
168183
<td width="150" colspan="2" rowspan="3" class="right-edged-col">
169184
<p align="center">syslog-ng poll<br>using an ivykis timer with timer freq set to follow-freq() value</p>
@@ -180,7 +195,7 @@ The following table shows which method is selected in different cases.
180195
</tr>
181196
</table>
182197

183-
**NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-freq() set to 0.
198+
**NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-method() set to `inotify`.
184199
{: .notice--info}
185200

186201
A bit more detail about the notation in the platform columns and what they really mean:

doc/_admin-guide/060_Sources/030_Wildcard-file/000_Wildcard-file_options.md

+2
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,8 @@ source s_files {
7676

7777
{% include doc/admin-guide/options/follow-freq.md %}
7878

79+
{% include doc/admin-guide/options/follow-method.md %}
80+
7981
{% include doc/admin-guide/options/hook.md %}
8082

8183
{% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/110_Pipe/000_Pipe_source_options.md

+2
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ The pipe driver has the following options:
1313

1414
{% include doc/admin-guide/options/follow-freq.md %}
1515

16+
{% include doc/admin-guide/options/follow-method.md %}
17+
1618
{% include doc/admin-guide/options/hook.md %}
1719

1820
{% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/160_sun-streams/000_sun-streams_options.md

+2
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ above 2.5.1.
1919

2020
{% include doc/admin-guide/options/follow-freq.md %}
2121

22+
{% include doc/admin-guide/options/follow-method.md %}
23+
2224
{% include doc/admin-guide/options/hook.md %}
2325

2426
{% include doc/admin-guide/options/keep-timestamp.md %}

0 commit comments

Comments
 (0)