// Use these for links to issue and pulls. Note issues and pulls redirect one to
// each other on Github, so don't worry too much on using the right prefix.
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

=== Beats version HEAD
https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD diff]

==== Breaking changes

*Affecting all Beats*

- Update to Golang 1.12.1. {pull}11330[11330]
- Disable Alibaba Cloud and Tencent Cloud metadata providers by default. {pull}13812[12812]
- Libbeat: Do not overwrite agent.*, ecs.version, and host.name. {pull}14407[14407]
- Libbeat: Cleanup the x-pack licenser code to use the new license endpoint and the new format. {pull}15091[15091]
- Refactor metadata generator to support adding metadata across resources {pull}14875[14875]
- Remove `AddDockerMetadata` and `AddKubernetesMetadata` processors from the `script` processor. They can still be used as normal processors in the configuration. {issue}16349[16349] {pull}16514[16514]
- Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. {pull}17938[17938]
- Make error message about locked data path actionable. {pull}18667[18667]

*Auditbeat*

*Filebeat*

- Fix parsing of Elasticsearch node name by `elasticsearch/slowlog` fileset. {pull}14547[14547]
- Improve ECS field mappings in panw module.  event.outcome now only contains success/failure per ECS specification. {issue}16025[16025] {pull}17910[17910]
- Improve ECS categorization field mappings for nginx module. http.request.referrer only populated when nginx sets a value {issue}16174[16174] {pull}17844[17844]
- Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . {issue}16180[16180] {pull}17982[17982]
- Preserve case of http.request.method.  ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. {issue}18154[18154] {pull}18359[18359]
- Adds oauth support for httpjson input. {issue}18415[18415] {pull}18892[18892]
- Adds `split_events_by` option to httpjson input. {pull}19246[19246]
- Adds `date_cursor` option to httpjson input. {pull}19483[19483]
- Adds Gsuite module with SAML support. {pull}19329[19329]
- Adds Gsuite User Accounts support. {pull}19329[19329]
- Adds Gsuite Login audit support. {pull}19702[19702]
- Adds Gsuite Admin support. {pull}19769[19769]
- Adds Gsuite Drive support. {pull}19704[19704]
- Adds Gsuite Groups support. {pull}19725[19725]
- Disable the option of running --machine-learning on its own. {pull}20241[20241]
- Fix PANW field spelling "veredict" to "verdict" on event.action {pull}18808[18808]
- Add support for GMT timezone offsets in `decode_cef`. {pull}20993[20993]
- API address and shard ID are required settings in the Cloud Foundry input. {pull}21759[21759]
- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095]
- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571]
- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]
- Rename `s3` input to `aws-s3` input. {pull}23469[23469]
- Possible values for Netflow's locality fields (source.locality, destination.locality and flow.locality) are now `internal` and `external`, instead of `private` and `public`. {issue}24272[24272] {pull}24295[24295]

*Heartbeat*

*Journalbeat*

- Improve parsing of syslog.pid in journalbeat to strip the username when present {pull}16116[16116]


*Metricbeat*

- Add new dashboard for VSphere host cluster and virtual machine {pull}14135[14135]
- kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. {issue}11975[11975]
- Fix ECS compliance of user.id field in system/users  metricset {pull}19019[19019]
- Remove "invalid zero" metrics on Windows and Darwin, don't report linux-only memory and diskio metrics when running under agent. {pull}21457[21457]

*Packetbeat*

- Added redact_headers configuration option, to allow HTTP request headers to be redacted whilst keeping the header field included in the beat. {pull}15353[15353]
- Add dns.question.subdomain and dns.question.top_level_domain fields. {pull}14578[14578]

*Winlogbeat*

- Add support to Sysmon file delete events (event ID 23). {issue}18094[18094]
- Improve ECS field mappings in Sysmon module. `related.hash`, `related.ip`, and `related.user` are now populated. {issue}18364[18364]
- Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding `process.hash`, `process.pe.imphash`, `file.hash`, or `file.pe.imphash`. {issue}18364[18364]
- Improve ECS field mappings in Sysmon module. `file.name`, `file.directory`, and `file.extension` are now populated. {issue}18364[18364]
- Improve ECS field mappings in Sysmon module. `rule.name` is populated for all events when present. {issue}18364[18364]
- Add Powershell module. Support for event ID's: `400`, `403`, `600`, `800`, `4103`, `4014`, `4105`, `4106`. {issue}16262[16262] {pull}18526[18526]
- Fix Powershell processing of downgraded engine events. {pull}18966[18966]
- Fix unprefixed fields in `fields.yml` for Powershell module {issue}18984[18984]

*Functionbeat*


==== Bugfixes

*Affecting all Beats*

- Fix a race condition with the Kafka pipeline client, it is possible that `Close()` get called before `Connect()` . {issue}11945[11945]
- Allow users to configure only `cluster_uuid` setting under `monitoring` namespace. {pull}14338[14338]
- Update replicaset group to apps/v1 {pull}15854[15802]
- Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data {pull}17223[17223]
- Fix missing output in dockerlogbeat {pull}15719[15719]
- Do not load dashboards where not available. {pull}15802[15802]
- Remove superfluous use of number_of_routing_shards setting from the default template. {pull}16038[16038]
- Fix index names for indexing not always guaranteed to be lower case. {pull}16081[16081]
- Fix loading processors from annotation hints. {pull}16348[16348]
- Add `ssl.ca_sha256` option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. {issue}15717[15717]
- Fix `NewContainerMetadataEnricher` to use default config for kubernetes module. {pull}16857[16857]
- Improve some logging messages for add_kubernetes_metadata processor {pull}16866{16866}
- Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. {pull}17613[17613]
- Fix `setup.dashboards.index` setting not working. {pull}17749[17749]
- Fix goroutine leak and Elasticsearch output file descriptor leak when output reloading is in use. {issue}10491[10491] {pull}17381[17381]
- Fix Elasticsearch license endpoint URL referenced in error message. {issue}17880[17880] {pull}18030[18030]
- Change `decode_json_fields` processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
- Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. {issue}17937[17937] {pull}17991[17991]
- [Autodiscover] Check if runner is already running before starting again. {pull}18564[18564]
- Fix `keystore add` hanging under Windows. {issue}18649[18649] {pull}18654[18654]
- Fix regression in `add_kubernetes_metadata`, so configured `indexers` and `matchers` are used if defaults are not disabled. {issue}18481[18481] {pull}18818[18818]
- Fix the `translate_sid` processor's handling of unconfigured target fields. {issue}18990[18990] {pull}18991[18991]
- Fixed a service restart failure under Windows. {issue}18914[18914] {pull}18916[18916]
- Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed {pull}18979[18979]
- Server-side TLS config now validates certificate and key are both specified {pull}19584[19584]
- Fix terminating pod autodiscover issue. {pull}20084[20084]
- Fix seccomp policy for calls to `chmod` and `chown`. {pull}20054[20054]
- Output errors when Kibana index pattern setup fails. {pull}20121[20121]
- Fix issue in autodiscover that kept inputs stopped after config updates. {pull}20305[20305]
- Add service resource in k8s cluster role. {pull}20546[20546]
- [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. {pull}20571[20571]
- The `o365input` and `o365` module now recover from an authentication problem or other fatal errors, instead of terminating. {pull}21258[21258]
- Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss` as gauges (rather than counters). {pull}22877[22877]
- Fix inode removal tracking code when files are replaced by files with the same name {pull}25002[25002]
- Fix `mage GenerateCustomBeat` instructions for a new beat {pull}17679[17679]
- Fix bug with annotations dedot config on k8s not used {pull}25111[25111]
- Fix negative Kafka partition bug {pull}25048[25048]

*Auditbeat*

- system/socket: Fixed compatibility issue with kernel 5.x. {pull}15771[15771]
- system/package: Fix parsing of Installed-Size field of DEB packages. {issue}16661[16661] {pull}17188[17188]
- system module: Fix panic during initialisation when /proc/stat can't be read. {pull}17569[17569]
- system/package: Fix an error that can occur while trying to persist package metadata. {issue}18536[18536] {pull}18887[18887]
- system/socket: Fix dataset using 100% CPU and becoming unresponsive in some scenarios. {pull}19033[19033] {pull}19764[19764]
- system/socket: Fixed tracking of long-running connections. {pull}19033[19033]

*Filebeat*

- cisco/asa fileset: Fix parsing of 302021 message code. {pull}14519[14519]
- Fix filebeat azure dashboards, event category should be `Alert`. {pull}14668[14668]
- Fixed dashboard for Cisco ASA Firewall. {issue}15420[15420] {pull}15553[15553]
- Add shared_credential_file to cloudtrail config {issue}15652[15652] {pull}15656[15656]
- Fix s3 input with cloudtrail fileset reading json file. {issue}16374[16374] {pull}16441[16441]
- Add queue_url definition in manifest file for aws module. {pull}16640{16640}
- Fixed various Cisco FTD parsing issues. {issue}16863[16863] {pull}16889[16889]
- Fix default index pattern in IBM MQ filebeat dashboard. {pull}17146[17146]
- Fix `elasticsearch.gc` fileset to not collect _all_ logs when Elasticsearch is running in Docker. {issue}13164[13164] {issue}16583[16583] {pull}17164[17164]
- Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. {issue}17216[17216] {pull}17220[17220]
- Fixed a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. {issue}17242[17242] {pull}17243[17243]
- Fixed MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. {issue}17086[17086] {pull}17156[17156]
- CEF: Fixed decoding errors caused by trailing spaces in messages. {pull}17253[17253]
- Fixed activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. {pull}17428[17428]
- Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. {issue}17734[17734] {pull}17735[17735]
- Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. {pull}17425[17425]
- Fixed `cloudfoundry.access` to have the correct `cloudfoundry.app.id` contents. {pull}17847[17847]
- Fixing `ingress_controller.` fields to be of type keyword instead of text. {issue}17834[17834]
- Fixed typo in log message. {pull}17897[17897]
- Unescape file name from SQS message. {pull}18370[18370]
- Improve cisco asa and ftd pipelines' failure handler to avoid mapping temporary fields. {issue}18391[18391] {pull}18392[18392]
- Fix source.address not being set for nginx ingress_controller {pull}18511[18511]
- Fix PANW module wrong mappings for bytes and packets counters. {issue}18522[18522] {pull}18525[18525]
- Fixed ingestion of some Cisco ASA and FTD messages when a hostname was used instead of an IP for NAT fields. {issue}14034[14034] {pull}18376[18376]
- Fix `o365.audit` failing to ingest events when ip address is surrounded by square brackets. {issue}18587[18587] {pull}18591[18591]
- Fix `o365` module ignoring `var.api` settings. {pull}18948[18948]
- Fix improper nesting of session_issuer object in aws cloudtrail fileset. {issue}18894[18894] {pull}18915[18915]
- Fix Cisco ASA ASA 3020** and 106023 messages {pull}17964[17964]
- Add missing `default_field: false` to aws filesets fields.yml. {pull}19568[19568]
- Fix bug with empty filter values in system/service {pull}19812[19812]
- Fix S3 input to trim delimiter /n from each log line. {pull}19972[19972]
- Fix s3 input parsing json file without expand_event_list_from_field. {issue}19902[19902] {pull}19962[19962]
- Fix s3 input parsing json file without expand_event_list_from_field. {issue}19902[19902] {pull}19962[19962] {pull}20370[20370]
- Fix millisecond timestamp normalization issues in CrowdStrike module {issue}20035[20035], {pull}20138[20138]
- Fix support for message code 106100 in Cisco ASA and FTD. {issue}19350[19350] {pull}20245[20245]
- Fix event.outcome logic for azure/siginlogs fileset {pull}20254[20254]
- Fix `fortinet` setting `event.timezone` to the system one when no `tz` field present {pull}20273[20273]
- Fix `okta` geoip lookup in pipeline for `destination.ip` {pull}20454[20454]
- Fix mapping exception in the `googlecloud/audit` dataset pipeline. {issue}18465[18465] {pull}20465[20465]
- Fix `cisco` asa and ftd parsing of messages 106102 and 106103. {pull}20469[20469]
- Fix event.kind for system/syslog pipeline {issue}20365[20365] {pull}20390[20390]
- Fix event.type for zeek/ssl and duplicate event.category for zeek/connection {pull}20696[20696]
- Add json body check for sqs message. {pull}21727[21727]
- Properly update offset in case of unparasable line. {pull}22685[22685]
- Drop aws.vpcflow.pkt_srcaddr and aws.vpcflow.pkt_dstaddr when equal to "-". {pull}22721[22721] {issue}22716[22716]
- Fix cisco umbrella module config by adding input variable. {pull}22892[22892]
- Fix network.direction logic in zeek connection fileset. {pull}22967[22967]
- Fix aws s3 overview dashboard. {pull}23045[23045]
- Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072]
- Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966]
- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126]
- Simplify regex for organization custom prefix in AWS/CloudTrail fileset. {issue}23203[23203] {pull}23204[23204]
- Fix syslog header parsing in infoblox module. {issue}23272[23272] {pull}23273[23273]
- Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277]
- Fix concurrent modification exception in Suricata ingest node pipeline. {pull}23534[23534]
- Change the `event.created` in Netflow events to be the time the event was created by Filebeat
  to be consistent with ECS. {pull}23094[23094]
- Fix Zoom module parameters for basic auth and url path. {pull}23779[23779]
- Fix handling of ModifiedProperties field in Office 365. {pull}23777[23777]
- Use rfc6587 framing for fortinet firewall and clientendpoint filesets when transferring over tcp. {pull}23837[23837]
- Fix httpjson input logging so it doesn't conflict with ECS. {pull}23972[23972]
- Fix Logstash module handling of logstash.log.log_event.action field. {issue}20709[20709]
- aws/s3access dataset was populating event.duration using the wrong unit. {pull}23920[23920]
- Zoom module pipeline failed to ingest some chat_channel events. {pull}23904[23904]
- Fix Netlow module issue with missing `internal_networks` config parameter. {issue}24094[24094] {pull}24110[24110]
- in httpjson input using encode_as "application/x-www-form-urlencoded" now sets Content-Type correctly {issue}24331[24331] {pull}24336[24336]
- Fix default `scope` in `add_nomad_metadata`. {issue}24559[24559]
- Fix Cisco ASA parser for message 722051. {pull}24410[24410]
- Fix `google_workspace` pagination. {pull}24668[24668]
- Fix netflow module ignoring detect_sequence_reset flag. {issue}24268[24268] {pull}24270[24270]
- Fix Cisco ASA parser for message 302022. {issue}24405[24405] {pull}24697[24697]
- Fix Cisco AMP `@metadata._id` calculation {issue}24717[24717] {pull}24718[24718]
- Fix date parsing in GSuite/login and Google Workspace/login filesets. {issue}24694[24694]
- Fix gcp/vpcflow module error where input type was defaulting to file. {pull}24719[24719]
- Improve PanOS parsing and ingest pipeline. {issue}22413[22413] {issue}22748[22748] {pull}24799[24799]
- Fix S3 input validation for non amazonaws.com domains. {issue}24420[24420] {pull}24861[24861]
- Fix google_workspace and okta modules pagination when next page template is empty. {pull}24967[24967]
- Fix gcp module field names to use gcp instead of googlecloud. {pull}25038[25038]

*Heartbeat*

- Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639]
- Fixed scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. {pull}16397[16397]
- Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. {pull}17549[17549]
- Fix panic when initialization of ICMP monitors fail twice. {pull}25073[25073]

*Journalbeat*


*Metricbeat*

- Fix checking tagsFilter using length in cloudwatch metricset. {pull}14525[14525]
- Fixed bug with `elasticsearch/cluster_stats` metricset not recording license expiration date correctly. {issue}14541[14541] {pull}14591[14591]
- Log bulk failures from bulk API requests to monitoring cluster. {issue}14303[14303] {pull}14356[14356]
- Fixed bug with `elasticsearch/cluster_stats` metricset not recording license ID in the correct field. {pull}14592[14592]
- Change lookup_fields from metricset.host to service.address {pull}15883[15883]
- Fix skipping protocol scheme by light modules. {pull}16205[pull]
- Made `logstash-xpack` module once again have parity with internally-collected Logstash monitoring data. {pull}16198[16198]
- Revert changes in `docker` module: add size flag to docker.container. {pull}16600[16600]
- Fix detection and logging of some error cases with light modules. {pull}14706[14706]
- Dynamically choose a method for the system/service metricset to support older linux distros. {pull}16902[16902]
- Reduce memory usage in `elasticsearch/index` metricset. {issue}16503[16503] {pull}16538[16538]
- Fix issue in Jolokia module when mbean contains multiple quoted properties. {issue}17375[17375] {pull}17374[17374]
- Further revise check for bad data in docker/memory. {pull}17400[17400]
- Fix how we filter services by name in system/service {pull}17400[17400]
- Fix cloudwatch metricset missing tags collection. {issue}17419[17419] {pull}17424[17424]
- check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. {pull}17418[17418]
- Fix aws.s3.bucket.name terms_field in s3 overview dashboard. {pull}17542[17542]
- Fix Unix socket path in memcached. {pull}17512[17512]
- Fix azure storage dashboards. {pull}17590[17590]
- Metricbeat no longer needs to be started strictly after Logstash for `logstash-xpack` module to report correct data. {issue}17261[17261] {pull}17497[17497]
- Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. {issue}17154[17154] {pull}17600[17600]
- Add privileged option so as mb to access data dir in Openshift. {pull}17606[17606]
- Fix "ID" event generator of Google Cloud module {issue}17160[17160] {pull}17608[17608]
- Add privileged option for Auditbeat in Openshift {pull}17637[17637]
- Fix storage metricset to allow config without region/zone. {issue}17623[17623] {pull}17624[17624]
- Fix overflow on Prometheus rates when new buckets are added on the go. {pull}17753[17753]
- Add a switch to the driver definition on SQL module to use pretty names {pull}17378[17378]
- Remove specific win32 api errors from events in perfmon. {issue}18292[18292] {pull}18361[18361]
- Remove required for region/zone and make stackdriver a metricset in googlecloud. {issue}16785[16785] {pull}18398[18398]
- Fix application_pool metricset after pdh changes. {pull}18477[18477]
- Fix tags_filter for cloudwatch metricset in aws. {pull}18524[18524]
- Fix panic on `metricbeat test modules` when modules are configured in `metricbeat.modules`. {issue}18789[18789] {pull}18797[18797]
- Fix getting gcp compute instance metadata with partial zone/region in config. {pull}18757[18757]
- Add missing network.sent_packets_count metric into compute metricset in googlecloud module. {pull}18802[18802]
- Fix compute and pubsub dashboard for googlecloud module. {issue}18962[18962] {pull}18980[18980]
- Fix crash on vsphere module when Host information is not available. {issue}18996[18996] {pull}19078[19078]
- Fix incorrect usage of hints builder when exposed port is a substring of the hint {pull}19052[19052]
- Stop counterCache only when already started {pull}19103[19103]
- Remove dedot for tag values in aws module. {issue}19112[19112] {pull}19221[19221]
- Fix mapping of service start type in the service metricset, windows module. {pull}19551[19551]
- Fix SQL module mapping NULL values as string {pull}18955[18955] {issue}18898[18898
- Modify doc for app_insights metricset to contain example of config. {pull}20185[20185]
- Add required option for `metrics` in app_insights. {pull}20406[20406]
- Groups same timestamp metric values to one event in the app_insights metricset. {pull}20403[20403]
- Add support for azure light metricset app_stats. {pull}20639[20639]
- Fix remote_write flaky test. {pull}21173[21173]
- Remove io.time from windows {pull}22237[22237]
- Fix `logstash` module when `xpack.enabled: true` is set from emitting redundant events. {pull}22808[22808]
- Ignore unsupported derive types for filesystem metricset. {issue}22501[22501] {pull}24502[24502]

*Packetbeat*



*Winlogbeat*


*Functionbeat*

*Elastic Logging Plugin*


==== Added

*Affecting all Beats*

- Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors {pull}12451[12451]
- Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. {pull}9260[9260]
- Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. {pull}11394[11394]
- add_host_metadata is no GA. {pull}13148[13148]
- Add `providers` setting to `add_cloud_metadata` processor. {pull}13812[13812]
- Ensure that init containers are no longer tailed after they stop {pull}14394[14394]
- Fingerprint processor adds a new xxhash hashing algorithm {pull}15418[15418]
- Add configuration for APM instrumentation and expose the tracer trough the Beat object. {pull}17938[17938]
- Include network information by default on add_host_metadata and add_observer_metadata. {issue}15347[15347] {pull}16077[16077]
- Add `aws_ec2` provider for autodiscover. {issue}12518[12518] {pull}14823[14823]
- Add support for multiple password in redis output. {issue}16058[16058] {pull}16206[16206]
- Add support for Histogram type in fields.yml {pull}16570[16570]
- Remove experimental flag from `setup.template.append_fields` {pull}16576[16576]
- Add support for kubernetes provider to recognize namespace level defaults {pull}16321[16321]
- Add capability of enrich `container.id` with process id in `add_process_metadata` processor {pull}15947[15947]
- Update documentation for system.process.memory fields to include clarification on Windows os's. {pull}17268[17268]
- Add `urldecode` processor to for decoding URL-encoded fields. {pull}17505[17505]
- Add keystore support for autodiscover static configurations. {pull]16306[16306]
- When using the `decode_json_fields` processor, decoded fields are now deep-merged into existing event. {pull}17958[17958]
- Add keystore support for autodiscover static configurations. {pull]16306[16306]
- Add TLS support to Kerberos authentication in Elasticsearch. {pull}18607[18607]
- Add support for multiple sets of hints on autodiscover {pull}18883[18883]
- Add a configurable delay between retries when an app metadata cannot be retrieved by `add_cloudfoundry_metadata`. {pull}19181[19181]
- Added the `max_cached_sessions` option to the script processor. {pull}19562[19562]
- Set index.max_docvalue_fields_search in index template to increase value to 200 fields. {issue}20215[20215]
- Add capability of enriching process metadata with contianer id also for non-privileged containers in `add_process_metadata` processor. {pull}19767[19767]
- Add replace_fields config option in add_host_metadata for replacing host fields. {pull}20490[20490] {issue}20464[20464]
- Add option to select the type of index template to load: legacy, component, index. {pull}21212[21212]

*Auditbeat*

- Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. {pull}17431[17431]
- Reference kubernetes manifests mount data directory from the host, so data persist between executions in the same node. {pull}17429[17429]
- Log to stderr when running using reference kubernetes manifests. {pull}17443[174443]
- Fix syscall kprobe arguments for 32-bit systems in socket module. {pull}17500[17500]
- Add ECS categorization info for auditd module {pull}18596[18596]

*Filebeat*


- `container` and `docker` inputs now support reading of labels and env vars written by docker JSON file logging driver. {issue}8358[8358]
- Add `index` option to all inputs to directly set a per-input index value. {pull}14010[14010]
- Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb {issue}15757[15757] {pull}15935[15936]
- Add ingress nginx controller fileset {pull}16197[16197]
- move create-[module,fileset,fields] to mage and enable in x-pack/filebeat {pull}15836[15836]
- Work on e2e ACK's for the azure-eventhub input {issue}15671[15671] {pull}16215[16215]
- Add a TLS test and more debug output to httpjson input {pull}16315[16315]
- Add an SSL config example in config.yml for filebeat MISP module. {pull}16320[16320]
- Improve ECS categorization, container & process field mappings in auditd module. {issue}16153[16153] {pull}16280[16280]
- Add cloudwatch fileset and ec2 fileset in aws module. {issue}13716[13716] {pull}16579[16579]
- Improve the decode_cef processor by reducing the number of memory allocations. {pull}16587[16587]
- Add custom string mapping to CEF module to support Forcepoint NGFW {issue}14663[14663] {pull}15910[15910]
- Add ECS related fields to CEF module {issue}16157[16157] {pull}16338[16338]
- Improve ECS categorization, host field mappings in elasticsearch module. {issue}16160[16160] {pull}16469[16469]
- Add pattern for Cisco ASA / FTD Message 734001 {issue}16212[16212] {pull}16612[16612]
- Added new module `o365` for ingesting Office 365 management activity API events. {issue}16196[16196] {pull}16386[16386]
- Add Filebeat Okta module. {pull}16362[16362]
- Add source field in k8s events {pull}17209[17209]
- Improve AWS cloudtrail field mappings {issue}16086[16086] {issue}16110[16110] {pull}17155[17155]
- Added new module `crowdstrike` for ingesting Crowdstrike Falcon streaming API endpoint event data. {pull}16988[16988]
- Move azure-eventhub input to GA. {issue}15671[15671] {pull}17313[17313]
- Added documentation for running Filebeat in Cloud Foundry. {pull}17275[17275]
- Added access_key_id, secret_access_key and session_token into aws module config. {pull}17456[17456]
- Release Google Cloud module as GA. {pull}17511[17511]
- Update filebeat httpjson input to support pagination via Header and Okta module. {pull}16354[16354]
- Added new Checkpoint Syslog filebeat module. {pull}17682[17682]
- Added Unix stream socket support as an input source and a syslog input source. {pull}17492[17492]
- Added new Fortigate Syslog filebeat module. {pull}17890[17890]
- Change the `json.*` input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
- Added http_endpoint input{pull}18298[18298]
- Add support for array parsing in azure-eventhub input. {pull}18585[18585]
- Added `observer.vendor`, `observer.product`, and `observer.type` to PANW module events. {pull}18223[18223]
- Improve ECS categorization field mappings in coredns module. {issue}16159[16159] {pull}18424[18424]
- Improve ECS categorization field mappings in envoyproxy module. {issue}16161[16161] {pull}18395[18395]
- Improve ECS categorization field mappings in cisco module. {issue}16028[16028] {pull}18537[18537]
- Add geoip AS lookup & improve ECS categorization in aws cloudtrail fileset. {issue}18644[18644] {pull}18958[18958]
- Improved performance of PANW sample dashboards. {issue}19031[19031] {pull}19032[19032]
- Add support for v1 consumer API in Cloud Foundry input, use it by default. {pull}19125[19125]
- Add new mode to multiline reader to aggregate constant number of lines {pull}18352[18352]
- Changed the panw module to pass through (rather than drop) message types other than threat and traffic. {issue}16815[16815] {pull}19375[19375]
- Improve ECS categorization field mappings in traefik module. {issue}16183[16183] {pull}19379[19379]
- Improve ECS categorization field mappings in azure module. {issue}16155[16155] {pull}19376[19376]
- Add text & flattened versions of fields with unknown subfields in aws cloudtrail fileset. {issue}18866[18866] {pull}19121[19121]
- Added Microsoft Defender ATP Module. {issue}17997[17997] {pull}19197[19197]
- Add experimental dataset tomcat/log for Apache TomCat logs {pull}19713[19713]
- Add experimental dataset netscout/sightline for Netscout Arbor Sightline logs {pull}19713[19713]
- Add experimental dataset barracuda/waf for Barracuda Web Application Firewall logs {pull}19713[19713]
- Add experimental dataset f5/bigipapm for F5 Big-IP Access Policy Manager logs {pull}19713[19713]
- Add experimental dataset bluecoat/director for Bluecoat Director logs {pull}19713[19713]
- Add experimental dataset cisco/nexus for Cisco Nexus logs {pull}19713[19713]
- Add experimental dataset citrix/virtualapps for Citrix Virtual Apps logs {pull}19713[19713]
- Add experimental dataset cylance/protect for Cylance Protect logs {pull}19713[19713]
- Add experimental dataset fortinet/clientendpoint for Fortinet FortiClient Endpoint Protection logs {pull}19713[19713]
- Add experimental dataset imperva/securesphere for Imperva Secure Sphere logs {pull}19713[19713]
- Add experimental dataset infoblox/nios for Infoblox Network Identity Operating System logs {pull}19713[19713]
- Add experimental dataset juniper/junos for Juniper Junos OS logs {pull}19713[19713]
- Add experimental dataset kaspersky/av for Kaspersky Anti-Virus logs {pull}19713[19713]
- Add experimental dataset microsoft/dhcp for Microsoft DHCP Server logs {pull}19713[19713]
- Add experimental dataset tenable/nessus_security for Tenable Nessus Security Scanner logs {pull}19713[19713]
- Add experimental dataset rapid7/nexpose for Rapid7 Nexpose logs {pull}19713[19713]
- Add experimental dataset radware/defensepro for Radware DefensePro logs {pull}19713[19713]
- Add experimental dataset sonicwall/firewall for Sonicwall Firewalls logs {pull}19713[19713]
- Add experimental dataset squid/log for Squid Proxy Server logs {pull}19713[19713]
- Add experimental dataset zscaler/zia for Zscaler Internet Access logs {pull}19713[19713]
- Add event.ingested for CrowdStrike module {pull}20138[20138]
- Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module {pull}20138[20138]
- Add event.ingested to all Filebeat modules. {pull}20386[20386]
- Add event.ingested for Suricata module {pull}20220[20220]
- Add support for custom header and headersecret for filebeat http_endpoint input {pull}20435[20435]
- Convert httpjson to v2 input {pull}20226[20226]
- Add event.ingested to all Filebeat modules. {pull}20386[20386]
- Added new properties field support for event.outcome in azure module {pull}20998[20998]
- Add type and sub_type to panw panos fileset {pull}20912[20912]
- Add related.hosts ecs field to all modules {pull}21160[21160]
- Keep cursor state between httpjson input restarts {pull}20751[20751]
- New juniper.srx dataset for Juniper SRX logs. {pull}20017[20017]
- Added DNS response IP addresses to `related.ip` in Suricata module. {pull}22291[22291]
- Updating field mappings for Cisco AMP module, fixing certain fields. {pull}24661[24661]
- Add support for upper case field names in Sophos XG module {pull}24693[24693]
- Add `fail_on_template_error` option for httpjson input. {pull}24784[24784]

*Heartbeat*


*Heartbeat*


*Heartbeat*

*Journalbeat*

*Metricbeat*

- Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. {pull}15503[15503]
- Add DynamoDB AWS Metricbeat light module {pull}15097[15097]
- Add IBM MQ light-weight Metricbeat module {pull}15301[15301]
- Add mixer metricset for Istio Metricbeat module {pull}15696[15696]
- Add mesh metricset for Istio Metricbeat module{pull}15535[15535]
- Add pilot metricset for Istio Metricbeat module {pull}15761[15761]
- Add galley metricset for Istio Metricbeat module {pull}15857[15857]
- Add `key/value` mode for SQL module. {issue}15770[15770] {pull]15845[15845]
- Add support for Unix socket in Memcached metricbeat module. {issue}13685[13685] {pull}15822[15822]
- Add `up` metric to prometheus metrics collected from host {pull}15948[15948]
- Add citadel metricset for Istio Metricbeat module {pull}15990[15990]
- Add collecting AuroraDB metrics in rds metricset. {issue}14142[14142] {pull}16004[16004]
- Add database_account azure metricset. {issue}15758[15758]
- Add Load Balancing metricset to GCP {pull}15559[15559]
- Add OpenMetrics Metricbeat module {pull}16596[16596]
- Add system/users metricset as beta {pull}16569[16569]
- Add additional cgroup fields to docker/diskio{pull}16638[16638]
- Add Prometheus remote write endpoint {pull}16609[16609]
- Add support for CouchDB v2 {issue}16352[16352] {pull}16455[16455]
- Release Zookeeper/connection module as GA. {issue}14281[14281] {pull}17043[17043]
- Add dashboard for pubsub metricset in googlecloud module. {pull}17161[17161]
- Replace vpc metricset into vpn, transitgateway and natgateway metricsets. {pull}16892[16892]
- Use Elasticsearch histogram type to store Prometheus histograms {pull}17061[17061]
- Allow to rate Prometheus counters when scraping them {pull}17061[17061]
- Release Oracle module as GA. {issue}14279[14279] {pull}16833[16833]
- Add Storage metricsets to GCP module {pull}15598[15598]
- Release vsphere module as GA. {issue}15798[15798] {pull}17119[17119]
- Add PubSub metricset to Google Cloud Platform module {pull}15536[15536]
- Add final tests and move label to GA for the azure module in metricbeat. {pull}17319[17319]
- Added documentation for running Metricbeat in Cloud Foundry. {pull}17275[17275]
- Reference kubernetes manifests mount data directory from the host when running metricbeat as daemonset, so data persist between executions in the same node. {pull}17429[17429]
- Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[[16471] {pull}17609[17609]
- Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. {issue}17141[[17141] {pull}17719[17719]
- Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[[16471] {pull}17609[17609]
- Collect new `bulk` indexing metrics from Elasticsearch when `xpack.enabled:true` is set. {issue} {pull}17992[17992]
- Remove requirement to connect as sysdba in Oracle module {issue}15846[15846] {pull}18182[18182]
- Update MSSQL module to fix some SSPI authentication and add brackets to USE statements {pull}17862[17862]]
- Add client address to events from http server module {pull}18336[18336]
- Add memory metrics into compute googlecloud. {pull}18802[18802]
- Add Tomcat overview dashboard {pull}14026[14026]
- Add support for v1 consumer API in Cloud Foundry module, use it by default. {pull}19268[19268]
- The `elasticsearch/index` metricset now collects metrics for hidden indices as well. {issue}18639[18639] {pull}18703[18703]
- Adds support for app insights metrics in the azure module. {issue}18570[18570] {pull}18940[18940]
- Added cache and connection_errors metrics to status metricset of MySQL module {issue}16955[16955] {pull}19844[19844]
- Update MySQL dashboard with connection errors and cache metrics {pull}19913[19913] {issue}16955[16955]
- Add cloud.instance.name into aws ec2 metricset. {pull}20077[20077]
- Add `scope` setting for elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. {issue}18539[18539] {pull}18547[18547]
- Add state_daemonset metricset for Kubernetes Metricbeat module {pull}20649[20649]
- Add billing metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738]
- Release lambda metricset in aws module as GA. {issue}21251[21251] {pull}21255[21255]
- Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137]

*Packetbeat*



*Functionbeat*


*Heartbeat*


*Winlogbeat*

- Set process.command_line and process.parent.command_line from Sysmon Event ID 1. {pull}17327[17327]
- Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module {pull}17517[17517]
- Add registry and code signature information and ECS categorization fields for sysmon module {pull}18058[18058]

*Elastic Log Driver*

- Add support for `docker logs` command {pull}19531[19531]

==== Deprecated

*Affecting all Beats*


*Filebeat*


*Heartbeat*

*Journalbeat*

*Metricbeat*


*Packetbeat*

*Winlogbeat*

*Functionbeat*

==== Known Issue

*Journalbeat*
