Skip to content

Commit 4080d3f

Browse files
authored
wildcard-file monitoring changes (#185)
Changes introduced in, and depends on syslog-ng/syslog-ng#5315, syslog-ng/syslog-ng#5312 Signed-off-by: Hofi <[email protected]>
2 parents b6ffa3c + f76e3f1 commit 4080d3f

File tree

2 files changed

+44
-11
lines changed

2 files changed

+44
-11
lines changed

_includes/doc/admin-guide/options/follow-freq.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55

66
*Description:* Indicates that the source should be checked periodically.
77
This is useful for files which always indicate readability, even though
8-
no new lines were appended (e.g. regular file system files). If this value
9-
is higher than zero, syslog-ng will not attempt to use ivykis file change
8+
no new content were appended (e.g. regular file system files). If this value
9+
is higher than zero, syslog-ng will not attempt to use kqueue or ivykis file change
1010
notification methods on the file (poll(), epoll(), etc.), but checks whether
1111
the file changed every time the follow-freq() interval (in seconds) has elapsed.\
1212
Floating-point numbers (for example, **1.5**) can be used as well.

doc/_admin-guide/060_Sources/020_File/001_File_following.md

+42-9
Original file line numberDiff line numberDiff line change
@@ -44,10 +44,10 @@ The following table shows which method is selected in different cases.
4444
</td>
4545
</tr>
4646
<tr>
47-
<td width="99" rowspan="6" class="right-edged-col">
47+
<td width="99" rowspan="8" class="right-edged-col">
4848
<p align="center">0</p>
4949
</td>
50-
<td width="150" rowspan="6" class="right-edged-col">
50+
<td width="150" rowspan="7" class="right-edged-col">
5151
<p align="center">ivykis poll</p>
5252
</td>
5353
<td width="152" class="right-edged-col">
@@ -99,7 +99,7 @@ The following table shows which method is selected in different cases.
9999
<p align="center">n.a.</p>
100100
</td>
101101
<td width="152" class="right-edged-col">
102-
<p align="center">works</p>
102+
<p align="center">works best</p>
103103
</td>
104104
<td width="153">
105105
<p align="center">n.a.</p>
@@ -133,12 +133,40 @@ The following table shows which method is selected in different cases.
133133
<p align="center">works, but always signals readability</p>
134134
</td>
135135
</tr>
136+
<tr>
137+
<td width="152" class="right-edged-col">
138+
<p align="center">uring</p>
139+
</td>
140+
<td width="151" class="right-edged-col">
141+
<p align="center">works, but always signals readability</p>
142+
</td>
143+
<td width="152" class="right-edged-col">
144+
<p align="center">n.a.</p>
145+
</td>
146+
<td width="153">
147+
<p align="center">n.a.</p>
148+
</td>
149+
</tr>
150+
<tr>
151+
<td width="150" colspan="2" class="right-edged-col">
152+
<p align="center">inotify from ivykis directory monitor</p>
153+
</td>
154+
<td width="151" class="right-edged-col">
155+
<p align="center">works best</p>
156+
</td>
157+
<td width="152" class="right-edged-col">
158+
<p align="center">n.a.</p>
159+
</td>
160+
<td width="153">
161+
<p align="center">n.a.</p>
162+
</td>
163+
</tr>
136164
<tr>
137165
<td width="99" rowspan="3" class="right-edged-col">
138166
<p align="center">&gt; 0</p>
139167
</td>
140168
<td width="150" colspan="2" rowspan="3" class="right-edged-col">
141-
<p align="center">syslog-ng poll<br>using ivykis timer with timer freq set to follow-freq() value</p>
169+
<p align="center">syslog-ng poll<br>using an ivykis timer with timer freq set to follow-freq() value</p>
142170
</td>
143171
<td width="151" rowspan="3" class="right-edged-col">
144172
<p align="center">works</p>
@@ -152,15 +180,20 @@ The following table shows which method is selected in different cases.
152180
</tr>
153181
</table>
154182

183+
**NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-freq() set to 0.
184+
{: .notice--info}
185+
155186
A bit more detail about the notation in the platform columns and what they really mean:
156187

157-
`n.a.` - Means that the feature is not supported on the given platform by default, which has a significant impact on how the final ivykis poll method is selected. Ivykis tries to set up (at initialization time) the method to be used in the order enumerated in the table above. If an option is `n.a.` (determined at build time), then the next option will be used automatically. The first available option will be used, and if it does not work on the given platform (see `does not work` below), then {{ site.product.short_name }} will stop that file source with an error. This can be controlled using the `IV_EXCLUDE_POLL_METHOD` environment variable. Methods enumerated in it will be excluded from the ivykis initialization flow, and the next available (and not excluded) one will be used. The strings that can be used in `IV_EXCLUDE_POLL_METHOD` are `port-timer port dev_poll epoll-timerfd epoll kqueue ppoll poll` in the same order as in the table.\
158-
e.g., on Linux you should use `IV_EXCLUDE_POLL_METHOD="epoll-timerfd epoll"` to force the usage of the `ppoll` method, as `port-timer port dev_poll` are not available, and `epoll-timerfd epoll` are not working currently. However, note that currently `ppoll` and `poll` are the only working options on Linux, and they are far from optimal, unlike on BSD-based systems like macOS, where the default `kqueue` is a perfect option to use.
188+
`n.a.` - Means that the feature is not supported on the given platform by default, which has a significant impact on how the final ivykis poll method is selected. Ivykis tries to set up (at initialization time) the method to be used in the order enumerated in the table above. If an option is `n.a.` (determined at build time), then the next option will be used automatically. The first available option will be used, and if it does not work on the given platform (see `does not work` below), then {{ site.product.short_name }} will stop that file source with an error.\
189+
This can be controlled using the `IV_SELECT_POLL_METHOD` and the `IV_EXCLUDE_POLL_METHOD` environment variables.\
190+
The method appears in `IV_SELECT_POLL_METHOD` will be forced at startup if available, otherwise {{ site.product.short_name }} will stop and display an error that it `can't find suitable event dispatcher`.\
191+
Methods enumerated in `IV_EXCLUDE_POLL_METHOD` will be excluded from the ivykis initialization flow, and the next available (and not excluded) one will be used. The strings that can be used in `IV_EXCLUDE_POLL_METHOD` are `port-timer port dev_poll epoll-timerfd epoll kqueue ppoll poll uring` in the same order as in the table.\
192+
e.g., on Linux you should use `IV_EXCLUDE_POLL_METHOD="epoll-timerfd epoll"` to force the usage of the `ppoll` method, as `port-timer port dev_poll` are not available, and `epoll-timerfd epoll` are not working currently. However, note that all the options marked as `works, but always signals readability` (like `ppoll`, `poll`, etc.) are far from optimal, unlike on BSD-based systems like macOS, where the default `kqueue` is a perfect option to use.
159193

160-
**NOTE:** We are planning to add an `inotify`, `io_uring` or similar-based solution to ivykis that could perform similarly to `kqueue`. Stay tuned!
161-
{: .notice--info}
194+
`works` - Means it has been tested and works seamlessly (based on our tests).
162195

163-
`works` - Means it is tested and works seamlessly (based on our tests).
196+
`works best` - Means it has been tested and functions seamlessly, delivering the best performance on the given platform (based on our tests).
164197

165198
`works, but always signals readability` - Means that the method is available on the given platform, but it is primarily designed for sockets, pipes, and similar uses, not for regular files. For regular files, it is always triggered (because regular files are always readable), behaving similarly to the `poll` method of directory monitoring with all of its disadvantages. Moreover, it could lead to even higher resource consumption (mainly CPU load) because follow-freq() does not control the frequency of the triggered internal file checks, which could occur hundreds or thousands of times per second.
166199

0 commit comments

Comments
 (0)