[Fortinet Fortigate Traffic]: Wrong calculation of network.bytes

[sbehrends83]

Integration Name

Fortinet FortiGate Firewall Logs [fortinet_fortigate]

Dataset Name

fortinet-fortigate.log

Integration Version

1.25.5

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Ubuntu 22.04

Software/API Version

7.x

Error Message

Hi @ALL!

We investigated some issue in counting / sum some traffic data over the fields sentbyte and rcvdbyte. We investigated that the fields are more statistics than metrics. The real metrics are written to the fields sentdelta and rcvddelta, which are more usable than the *bytes fields.

Event Original

timestamp="1724322947" devname="myfgt" devid="FG5H0E5899999999" vd="root" date="2024-08-22" time="12:35:47" eventtime="1724322947595266411" tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" srcip="10.100.208.12" srcport="51396" srcintf="srv" srcintfrole="dmz" dstip="10.200.100.112" dstport="47106" dstintf="lan" dstintfrole="lan" srccountry="Reserved" dstcountry="Reserved" sessionid="1660293071" proto="6" action="success" policyid="15" policytype="policy" poluuid="476097ca-7fe5-51ee-1867-9a120be36b89" policyname="push-to-client" service="tcp/47106" trandisp="noop" duration="5" sentbyte="487" rcvdbyte="310" sentpkt="9" rcvdpkt="1" appcat="unscanned" dsthwvendor="Brocade" masterdstmac="ff:ff:ff:ff:ff:aa" dstmac="ff:ff:ff:ff:ff:ff" dstserver="1"

What did you do?

It a snipped from agent-policy

• id: udp-fortinet_fortigate-86972f9d-2345-4867-9dcd-ae0b8f5eb9ae
  name: test-fgt-port-9004
  revision: 5
  type: udp
  use_output: default
  meta:
  package:
  name: fortinet_fortigate
  version: 1.25.5
  data_stream:
  namespace: test
  package_policy_id: 86972f9d-2345-4867-9dcd-ae0b8f5eb9ae
  streams:
  - id: udp-fortinet_fortigate.log-86972f9d-2345-4867-9dcd-ae0b8f5eb9ae
  data_stream:
  dataset: fortinet_fortigate.log
  type: logs
  host: '0.0.0.0:9004'
  tags:
  - preserve_original_event
  - fortinet-fortigate
  - fortinet-firewall
  - forwarded
  publisher_pipeline.disable_host: true
  processors:
  - add_fields:
  target: _temp
  fields:
  internal_networks:
  - private

What did you see?

Hte prob

What did you expect to see?

The calculated sum in network.bytes (sendbyte (=source.bytes) + rcvdbyte (=(destination.bytes)) which is done over the ingest pipeline logs-fortinet_fortigate.log-1.25.5 and a script processor:

ctx.network.bytes = ctx.source.bytes + ctx.destination.bytes

is using only sendbyte and rcvdbyte, but should use under conditions senddelta and rcvddelta instead of the byte fields.

Maybe something like this:

[
{
"convert": {
"field": "fortinet.firewall.sentdelta",
"type": "long",
"if": "ctx?.fortinet?.firewall?.sentdelta != null",
"ignore_failure": true
}
},
{
"convert": {
"field": "fortinet.firewall.rcvddelta",
"type": "long",
"if": "ctx?.fortinet?.firewall?.rcvddelta != null",
"ignore_failure": true
}
},
{
"script": {
"lang": "painless",
"source": "ctx.network.bytesdelta = ctx.fortinet.firewall.sentdelta + ctx.fortinet.firewall.rcvddelta",
"if": "ctx?.fortinet?.firewall?.sentdelta != null && ctx?.fortinet?.firewall?.rcvddelta != null",
"ignore_failure": true
}
},
]

This is just a workaround over the custom pipeline from the integration. Is it possible to integrate this somehow?

Anything else?

No response

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[lillesvin]

Yes, it appears that sentbyte and rcvdbyte (and consequently ´network.bytes`) are indeed the total amount of bytes exchanged in that session so far. This is hard to do traffic graphs on, since you can't just sum them but instead have to group them by session ID first, take only the last/largest value per session and sum them, which results in some ridiculously heavy queries.

Using sentdelta and rcvddelta, perhaps in a separate field (e.g. network.deltabytes) would make things a lot easier.

[taylor-swanson]

@lillesvin @sbehrends83, sorry for getting to this a bit late.

Using sentdelta and rcvddelta, perhaps in a separate field (e.g. network.deltabytes) would make things a lot easier.

That's the approach I'm taking in #13668. I'm adding a new fortinet.firewall.deltabytes field to hold the sum of delta bytes from the event. ECS doesn't have a field to represent this kind of field, which is why I'm putting it into the fortinet fields. I'll also convert the sentdelta and rcvddelta fields to longs as part of the change (addressing #11433). I'm not going to touch the existing network.bytes field.