:type: log

[id="{beatname_lc}-input-{type}"]
=== Log input

++++
<titleabbrev>Log</titleabbrev>
++++

Use the `log` input to read lines from log files.

To configure this input, specify a list of glob-based <<input-paths,`paths`>>
that must be crawled to locate and fetch the log lines.

Example configuration:

["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
----


You can apply additional
<<{beatname_lc}-input-{type}-options,configuration settings>> (such as `fields`,
`include_lines`, `exclude_lines`, `multiline`, and so on) to the lines harvested
from these files. The options that you specify are applied to all the files
harvested by this input.

To apply different configuration settings to different files, you need to define
multiple input sections:

["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: log <1>
  paths:
    - /var/log/system.log
    - /var/log/wifi.log
- type: log <2>
  paths:
    - "/var/log/apache2/*"
  fields:
    apache: true
  fields_under_root: true
----

<1> Harvests lines from two files:  `system.log` and
`wifi.log`.
<2> Harvests lines from every file in the `apache2` directory, and uses the
`fields` configuration option to add a field called `apache` to the output.


IMPORTANT: Make sure a file is not defined more than once across all inputs
because this can lead to unexpected behaviour.

[[file-identity]]
==== Reading files on network shares and cloud providers

WARNING: Filebeat does not support reading from network shares and cloud providers.

However, one of the limitations of these data sources can be mitigated
if you configure Filebeat adequately.

By default, {beatname_uc} identifies files based on their inodes and
device IDs. However, on network shares and cloud providers these
values might change during the lifetime of the file. If this happens
{beatname_uc} thinks that file is new and resends the whole content
of the file. To solve this problem you can configure `file_identity` option. Possible
values besides the default `inode_deviceid` are `path` and `inode_marker`.

Selecting `path` instructs {beatname_uc} to identify files based on their
paths. This is a quick way to avoid rereading files if inode and device ids
might change. However, keep in mind if the files are rotated (renamed), they
will be reread and resubmitted.

The option `inode_marker` can be used if the inodes stay the same even if
the device id is changed. You should choose this method if your files are
rotated instead of `path` if possible. You have to configure a marker file
readable by {beatname_uc} and set the path in the option `path` of `inode_marker`.

The content of this file must be unique to the device. You can put the
UUID of the device or mountpoint where the input is stored. The following
example oneliner generates a hidden marker file for the selected mountpoint `/logs`:
Please note that you should not use this option on Windows as file identifiers might be
more volatile.

["source","sh",subs="attributes"]
----
$ lsblk -o MOUNTPOINT,UUID | grep /logs | awk '{print $2}' >> /logs/.filebeat-marker
----

To set the generated file as a marker for `file_identity` you should configure
the input the following way:

["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: log
  paths:
    - /logs/*.log
  file_identity.inode_marker.path: /logs/.filebeat-marker
----


[[rotating-logs]]
==== Reading from rotating logs

When dealing with file rotation, avoid harvesting symlinks. Instead
use the <<input-paths>> setting to point to the original file, and specify
a pattern that matches the file you want to harvest and all of its rotated
files. Also make sure your log rotation strategy prevents lost or duplicate
messages. For more information, see <<file-log-rotation>>.

Furthermore, to avoid duplicate of rotated log messages, do not use the
`path` method for `file_identity`. Or exclude the rotated files with `exclude_files`
option.

[id="{beatname_lc}-input-{type}-options"]
==== Configuration options

The `log` input supports the following configuration options plus the
<<{beatname_lc}-input-{type}-common-options>> described later.

[float]
[[input-paths]]
===== `paths`

A list of glob-based paths that will be crawled and fetched. All patterns
supported by https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also
supported here. For example, to fetch all files from a predefined level of
subdirectories, the following pattern can be used: `/var/log/*/*.log`. This
fetches all `.log` files from the subfolders of `/var/log`. It does not
fetch log files from the `/var/log` folder itself.
It is possible to recursively fetch all files in all subdirectories of a directory
using the optional <<recursive_glob,`recursive_glob`>> settings.

{beatname_uc} starts a harvester for each file that it finds under the specified
paths. You can specify one path per line. Each line begins with a dash (-).

[float]
[[recursive_glob]]
===== `recursive_glob.enabled`

Enable expanding `**` into recursive glob patterns. With this feature enabled,
the rightmost `**` in each path is expanded into a fixed number of glob
patterns. For example: `/foo/**` expands to `/foo`, `/foo/*`, `/foo/*/*`, and so
on. If enabled it expands a single `**` into a 8-level deep `*` pattern.

This feature is enabled by default. Set `recursive_glob.enabled` to false to
disable it.

include::../inputs/input-common-harvester-options.asciidoc[]

include::../inputs/input-common-file-options.asciidoc[]

[id="{beatname_lc}-input-{type}-common-options"]
include::../inputs/input-common-options.asciidoc[]

:type!:
