Use system time offset for local datetimes

[stefk]

The Time_Offset parameter allows to add a fixed UTC offset to local datetimes. However, that offset is subject to changes in countries that observe a daylight saving time, so hardcoding it in a config file may produce incorrect results at some point. This seems like a common problem but I don't see a way to address it with the current configuration options of Fluent Bit. Am I missing something? Or would you consider adding a parameter, for example, to retrieve that offset directly from the system instead of using a fixed one?

[edsiper]

@stefk there is one feature in Fluent Bit to use variables in the configuration files. Looking carefully in the source code I see that feature is not exposed for the parsers.conf which could address the main issue stated here: using an environment variable would be enough.

This will be addressed during Fluent Bit v0.14 development cycle.

[stefk]

@edsiper thanks for the feedback! Aren't environment variables resolved at startup time only though? If that's the case, scheduling a re-start at offset changes would still be necessary.

[scuzhanglei]

@edsiper My version is v1.0.1, I test it and found out it still not support variables in parsers.conf, is it? thanks a lot for your works.

[scuzhanglei]

@edsiper plz review this PR

[danischroeter]

we got the same issue here...
we got logs that are written with the local timezone (CEST) but without timezone specification. Depending on Summer/Winter time that would mean Time_Offset of 2 or 1 hour...

Instead of the proposed solution I think it would make more sense to define a new property like:
Time_DefaultZone that let's us specifiy the timezone for local timestamps. This way fluentbit could read the time and convert it from Time_DefaultZone to UTC. This would also solve the daylightsavingtime issues...

Maybe this method could even be used as a quickfix/hack: is there a way to modify the time field of a record (manually add CEST) and later parse the time?

[agileknight]

I found a workaround that uses the configured timezone, e.g. "Europe/Berlin" at the time of parsing the log entry.

• Use a lua filter to parse the timestamp
• Use a linux distribution that fully supports timezone information (ulibc does not work)
• Configure the timezone to use to parse at the distro level

Example code (tested with version 1.3.6):

parse_ts.lua (placed next to fluent-bit.conf)

function cb_parse_ts(tag, timestamp, record)
  time=record["time"]
   if (time == nil) then
    return -1, 0, 0 -- drop entry instead of throwing error
  end
  pattern="(%d+)-(%d+)-(%d+) (%d+):(%d+):(%d+),(%d+)" -- example: 2020-01-14 17:23:58,634
  year,month,day,hour,minute,second,millisecond=time:match(pattern)
  new_timestamp = os.time({day=day,month=month,year=year,hour=hour,min=minute,sec=second}) + millisecond/1000
  return 1, new_timestamp, record
end

filter in fluent-bit.conf

[FILTER]
    Name                lua
    Match               *
    script              parse_ts.lua
    call                cb_parse_ts

Dockerfile

FROM debian:stretch

ENV FLUENTBIT_VERSION 1.3.6
ENV DEBIAN_FRONTEND noninteractive

RUN  apt-get update && apt-get install -y \
    wget \
    gnupg2 \
    dirmngr \
    apt-transport-https \
    ca-certificates \
  && rm -rf /var/lib/apt/lists/*

RUN wget -qO - https://packages.fluentbit.io/fluentbit.key | apt-key add -
RUN echo "deb https://packages.fluentbit.io/debian/stretch stretch main" >> /etc/apt/sources.list
RUN  apt-get update && apt-get install -y \
    td-agent-bit=$FLUENTBIT_VERSION \
  && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /fluent-bit/bin && \
  ln -s /opt/td-agent-bit/bin/td-agent-bit /fluent-bit/bin/fluent-bit

USER 1000
CMD ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.conf"]

Run container with env set:

TZ=Europe/Berlin

Also requires mounted files:

• /fluent-bit/etc/fluent-bit.conf
• /fluent-bit/etc/parse_ts.lua

[jtackaberry]

I was also bit by this, so +1 for a native solution.

[shqear93]

@edsiper are you going to support a field like Time_DefaultZone ?

[lacksfish]

Is fluentbit gonna address this issue? Ignoring the timezone when parsing to UTC timestamp is a pretty major bug

Time_Key dateTime
Time_Format %Y-%m-%dT%H:%M:%S,%L%z

Is getting parsed without problems, but the timezone is simply dropped and all our timestamps are 2 hours in the future.

We can not use the Time_Offset key as we have summer/winter-time so time offset fluctuates twice a year between 1 and 2 hours.

Troubling issue......

[LionelCons]

+1