Log rotation results in lost or duplicate events

Stack

Filebeat supports reading from rotating log files. However, some log rotation strategies can result in lost or duplicate events when using Filebeat to forward messages. To avoid this issue:

• Be careful when changing the default file identity of the filestream input

  By default, the file identity of the filestream input is set to fingerprint, which identifies files using file fingerprints produced by the scanner component of the filestream input. Changing the file identity configuration may result in duplicated events in the output.

• Make sure Filebeat is configured to read from all rotated logs

  To avoid missing events from a rotated file, configure the input to read from the log file and all the rotated files.

  When an input log file is moved or renamed during log rotation, Filebeat is able to recognize that the file has already been read. After the file is rotated, a new log file is created, and the application continues logging. If the rotated log are included in configuration paths, Filebeat picks up the new file during the next scan.

  For examples, refer to Example configurations.

• Avoid log rotation strategies that copy and truncate log files

  Strategies that copy and truncate the input log file can result in lost events if lines are written to the log file after it’s copied, but before it’s truncated.

  Note

  In Filebeat versions prior to 9.0.0, using such strategies can also result in the duplication of events. In those versions, Filebeat identifies files by inode and device ID because the file identity configuration of the filestream input is set to native by default. During log rotation, lines that Filebeat has already processed are moved to a new file. If the file identity is set to native, when Filebeat encounters the new file, it starts reading it from the beginning because the previous state information (the offset and read timestamp) is associated with the inode and device ID of the old file.

If you’re using Windows, also check More about log rotation on Windows.

This section shows a typical configuration for logrotate, a popular tool for doing log rotation on Linux, followed by a Filebeat configuration that reads all the rotated logs.

In this example, Filebeat reads web server log. The logs are rotated every day, and the new file is created with the specified permissions.

/var/log/my-server/my-server.log {
    daily
    missingok
    rotate 7
    notifempty
    create 0640 www-data www-data
}
		

In this example, Filebeat is configured to read all log files to make sure it does not miss any events.

filebeat.inputs:
- type: filestream
  id: my-server-filestream-id
  paths:
  - /var/log/my-server/my-server.log*
		

On Windows, log rotation schemes that delete old files and rename newer files to old filenames might get blocked if the old files are being processed by Filebeat. This happens because Windows does not delete files and file metadata until the last process has closed the file. Unlike most *nix filesystems, a Windows filename cannot be reused until all processes accessing the file have closed the deleted file.

To avoid this problem, use dates in rotated filenames. The file will never be renamed to an older filename, and the log writer and log rotator will always be able to open the file. This approach also highly reduces the chance of log writing, rotation, and collection interfering with each other.

Because log rotation is typically handled by the logging application, we are not providing an example configuration for Windows.

Also read Open file handlers cause issues with Windows file rotation.