enhancement(loki sink, codecs): Integrate encoding::Encoder with loki sink

[pablosichert]

Part of #9459.

[netlify]

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	7ccb741
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62842ce71ab270000924c4a6
😎 Deploy Preview	https://deploy-preview-12558--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

Soak Test Results

Baseline: 6d657ac
Comparison: da58584
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_datadog_filter_blackhole	311.67KiB	2.15	100.00%	14.14MiB	791.64KiB	13.52KiB	0	0.0546637	14.44MiB	768.98KiB	13.15KiB	0	0.0519799	False	True
datadog_agent_remap_blackhole_acks	762.57KiB	1.49	100.00%	50.05MiB	3.18MiB	54.68KiB	0	0.063588	50.79MiB	2.29MiB	39.97KiB	0	0.0451655	False	False
http_pipelines_no_grok_blackhole	184.52KiB	1.46	100.00%	12.33MiB	685.63KiB	11.72KiB	0	0.0542906	12.51MiB	758.54KiB	12.97KiB	0	0.0591989	False	False
syslog_log2metric_humio_metrics	52.79KiB	0.39	100.00%	13.15MiB	475.11KiB	7.95KiB	0	0.0352813	13.2MiB	476.82KiB	7.98KiB	0	0.0352702	False	False
splunk_transforms_splunk3	39.63KiB	0.32	55.31%	12.22MiB	2.14MiB	36.71KiB	0	0.174782	12.26MiB	2.13MiB	36.97KiB	0	0.174038	False	False
splunk_hec_indexer_ack_blackhole	413.34B	0	4.08%	23.84MiB	333.3KiB	5.57KiB	0	0.0136521	23.84MiB	335.03KiB	5.59KiB	0	0.0137229	False	False
splunk_hec_to_splunk_hec_logs_acks	-322.07B	-0	3.23%	23.84MiB	328.21KiB	5.48KiB	0	0.0134441	23.84MiB	329.84KiB	5.51KiB	0	0.0135111	False	False
fluent_elasticsearch	-351.7B	-0	21.60%	79.47MiB	53.57KiB	907.9B	0	0.000658115	79.47MiB	53.27KiB	906.31B	0	0.000654445	False	False
http_to_http_noack	-2.15KiB	-0.01	27.71%	23.85MiB	255.38KiB	4.31KiB	0	0.0104565	23.85MiB	251.1KiB	4.24KiB	0	0.0102821	False	False
splunk_hec_to_splunk_hec_logs_noack	-2.59KiB	-0.01	25.73%	23.85MiB	328.38KiB	5.55KiB	0	0.0134465	23.84MiB	331.55KiB	5.61KiB	0	0.0135779	False	False
http_to_http_json	-4.76KiB	-0.02	43.88%	23.84MiB	341.14KiB	5.73KiB	0	0.0139725	23.83MiB	347.14KiB	5.86KiB	0	0.0142209	False	False
http_pipelines_blackhole_acks	-1.65KiB	-0.12	7.99%	1.36MiB	679.14KiB	11.58KiB	0	0.488012	1.36MiB	687.98KiB	11.74KiB	0	0.494948	True	True
http_to_http_acks	-50.2KiB	-0.27	20.67%	18.45MiB	7.88MiB	135.59KiB	0	0.426722	18.41MiB	7.83MiB	135.33KiB	0	0.42515	True	True
splunk_hec_route_s3	-61.65KiB	-0.32	71.59%	18.99MiB	2.34MiB	40.48KiB	0	0.12325	18.93MiB	2.36MiB	40.91KiB	0	0.124639	False	False
http_pipelines_blackhole	-5.58KiB	-0.4	27.86%	1.35MiB	650.85KiB	10.95KiB	0	0.469088	1.35MiB	655.82KiB	11.18KiB	0	0.474579	True	True
syslog_humio_logs	-129.4KiB	-0.78	100.00%	16.27MiB	392.65KiB	6.68KiB	0	0.0235682	16.14MiB	358.27KiB	5.98KiB	0	0.021673	False	False
syslog_splunk_hec_logs	-161.05KiB	-0.95	100.00%	16.5MiB	179.89KiB	3.03KiB	0	0.0106457	16.34MiB	139.22KiB	2.35KiB	0	0.00831829	False	False
syslog_log2metric_splunk_hec_metrics	-176.14KiB	-1.08	100.00%	15.97MiB	626.55KiB	10.68KiB	0	0.0383107	15.8MiB	602.87KiB	10.31KiB	0	0.0372644	False	False
syslog_regex_logs2metric_ddmetrics	-153.67KiB	-1.24	100.00%	12.06MiB	525.77KiB	8.8KiB	0	0.0425785	11.91MiB	501.32KiB	8.54KiB	0	0.04111	False	False
datadog_agent_remap_datadog_logs	-1.47MiB	-2.19	100.00%	66.86MiB	2.33MiB	40.21KiB	0	0.0348944	65.4MiB	2.03MiB	35.26KiB	0	0.0310603	False	False
datadog_agent_remap_datadog_logs_acks	-1.58MiB	-2.3	100.00%	68.5MiB	2.29MiB	39.82KiB	0	0.0334624	66.93MiB	1.73MiB	29.62KiB	0	0.025832	False	False
datadog_agent_remap_blackhole	-1.2MiB	-2.34	100.00%	51.26MiB	1.62MiB	28.44KiB	0	0.0316739	50.07MiB	2.76MiB	48.24KiB	0	0.0550237	False	False
syslog_loki	-1.07MiB	-7.18	100.00%	14.87MiB	504.65KiB	8.45KiB	0	0.0331287	13.81MiB	592.78KiB	10.04KiB	0	0.0419225	False	False
socket_to_socket_blackhole	-2.19MiB	-8.84	100.00%	24.83MiB	267.25KiB	4.51KiB	0	0.0105099	22.63MiB	119.08KiB	2.03KiB	0	0.0051369	False	False

[github-actions]

Soak Test Results

Baseline: aa540a2
Comparison: 7ccb741
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

Changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

experiment	Δ mean	Δ mean %	confidence
syslog_loki	-2.04MiB	-12.14	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_log2metric_splunk_hec_metrics	416.37KiB	2.21	100.00%	18.38MiB	287.18KiB	5.99KiB	0	0.0152552	18.79MiB	291.87KiB	6.11KiB	0	0.0151691	False	False
datadog_agent_remap_blackhole_acks	1.33MiB	1.99	100.00%	66.91MiB	1.73MiB	36.4KiB	0	0.025862	68.24MiB	969.83KiB	20.2KiB	0	0.0138759	False	False
syslog_splunk_hec_logs	325.63KiB	1.69	100.00%	18.84MiB	170.95KiB	3.53KiB	0	0.00885758	19.16MiB	145.91KiB	3.01KiB	0	0.00743478	False	False
syslog_humio_logs	261.09KiB	1.38	100.00%	18.53MiB	531.29KiB	11.08KiB	0	0.0280001	18.78MiB	563.08KiB	11.51KiB	0	0.0292729	False	False
syslog_regex_logs2metric_ddmetrics	162.24KiB	1.18	100.00%	13.39MiB	512.86KiB	10.51KiB	0	0.0374026	13.55MiB	516.03KiB	10.76KiB	0	0.0371938	False	False
http_pipelines_no_grok_blackhole	68.36KiB	0.34	99.84%	19.84MiB	725.03KiB	15.18KiB	0	0.0356765	19.91MiB	735.52KiB	15.4KiB	0	0.0360712	False	False
socket_to_socket_blackhole	80.36KiB	0.32	100.00%	24.71MiB	417.17KiB	8.62KiB	0	0.0164857	24.79MiB	376.99KiB	7.86KiB	0	0.0148506	False	False
splunk_hec_route_s3	58.97KiB	0.27	67.72%	21.01MiB	2.0MiB	42.39KiB	0	0.0952949	21.07MiB	1.98MiB	41.95KiB	0	0.093794	False	False
http_to_http_acks	6.89KiB	0.04	2.35%	18.12MiB	7.87MiB	166.06KiB	0	0.434343	18.13MiB	7.78MiB	165.0KiB	0	0.429033	True	True
fluent_elasticsearch	-465.88B	-0	23.22%	79.47MiB	54.18KiB	1.1KiB	0	0.000665571	79.47MiB	53.02KiB	1.08KiB	0	0.000651332	False	False
splunk_hec_to_splunk_hec_logs_acks	150.14B	0	1.21%	23.84MiB	330.46KiB	6.76KiB	0	0.0135354	23.84MiB	336.32KiB	6.88KiB	0	0.0137753	False	False
splunk_hec_indexer_ack_blackhole	-751.14B	-0	6.14%	23.84MiB	330.04KiB	6.75KiB	0	0.0135179	23.84MiB	328.11KiB	6.71KiB	0	0.0134394	False	False
http_to_http_json	-2.09KiB	-0.01	16.57%	23.84MiB	343.56KiB	7.08KiB	0	0.0140705	23.84MiB	340.02KiB	7.03KiB	0	0.0139266	False	False
splunk_hec_to_splunk_hec_logs_noack	-2.51KiB	-0.01	19.79%	23.84MiB	339.86KiB	7.03KiB	0	0.0139177	23.84MiB	343.32KiB	7.11KiB	0	0.0140609	False	False
http_to_http_noack	-4.31KiB	-0.02	45.05%	23.85MiB	245.14KiB	5.06KiB	0	0.0100361	23.84MiB	246.68KiB	5.11KiB	0	0.0101009	False	False
http_pipelines_blackhole	-5.5KiB	-0.12	25.76%	4.59MiB	590.81KiB	12.18KiB	0	0.12571	4.58MiB	550.78KiB	11.5KiB	0	0.11733	False	False
http_pipelines_blackhole_acks	-6.25KiB	-0.14	29.82%	4.3MiB	542.05KiB	11.34KiB	0	0.123133	4.29MiB	559.79KiB	11.72KiB	0	0.127346	False	False
syslog_log2metric_humio_metrics	-96.31KiB	-0.69	100.00%	13.71MiB	172.39KiB	3.53KiB	0	0.0122748	13.62MiB	175.74KiB	3.6KiB	0	0.0125995	False	False
datadog_agent_remap_blackhole	-596.71KiB	-0.86	100.00%	68.07MiB	962.35KiB	20.13KiB	0	0.0138033	67.49MiB	953.97KiB	19.98KiB	0	0.0138012	False	False
datadog_agent_remap_datadog_logs_acks	-744.85KiB	-0.94	100.00%	77.73MiB	597.89KiB	12.42KiB	0	0.00750982	77.0MiB	560.13KiB	11.48KiB	0	0.00710201	False	False
datadog_agent_remap_datadog_logs	-927.38KiB	-1.16	100.00%	78.37MiB	461.92KiB	9.51KiB	0	0.00575434	77.47MiB	790.3KiB	16.4KiB	0	0.00996023	False	False
splunk_transforms_splunk3	-215.14KiB	-1.3	99.84%	16.17MiB	2.26MiB	47.49KiB	0	0.139637	15.96MiB	2.3MiB	48.72KiB	0	0.14378	False	False
syslog_loki	-2.04MiB	-12.14	100.00%	16.83MiB	264.23KiB	5.42KiB	0	0.0153245	14.79MiB	192.24KiB	3.99KiB	0	0.0126889	False	False

[pablosichert]

The performance degradation when using json encoding with the new Encoder is a known issue: #11341, #12227.