diff --git a/content/blog/20261023_fluentd-v1.20.0-has-been-released.md b/content/blog/20261023_fluentd-v1.20.0-has-been-released.md new file mode 100644 index 00000000..0330573d --- /dev/null +++ b/content/blog/20261023_fluentd-v1.20.0-has-been-released.md @@ -0,0 +1,273 @@ +# Fluentd v1.20.0 has been released + +Hi users! + +We released v1.20.0 on October 23, 2026. +ChangeLog is [here](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1200---20261023). + +Fluentd v1.20.0 is the first release in the v1.20 series. +It enables Ruby JIT by default and adds options to limit the size of evacuated buffer files and to discover new log files sooner, along with bug fixes and performance improvements. + + + +## Notable changes + +### Ruby 3.2 is no longer supported + +Fluentd now requires Ruby 3.3 or later. +Ruby 3.3 is needed to enable Ruby JIT through `RubyVM::YJIT.enable`. + +### Ruby JIT is enabled by default + +The `enable_jit` option in the `` directive now defaults to `true`. +Fluentd enables YJIT in each worker process, and it can improve the throughput. + +On the other hand, JIT increases the memory usage of each worker process. +If you want to keep the previous behavior, set `enable_jit false` as follows: + +``` + + enable_jit false + +``` + +This option has no effect on Windows, because YJIT is not supported there. +In that case, Fluentd just logs `Ruby JIT is not available on this Ruby` at the info level. + +Ref. [enable\_jit](https://docs.fluentd.org/deployment/system-config#enablejit) + +### Fluentd no longer uses `yajl-ruby` + +Fluentd has used the `json` gem as the default JSON processor since v1.19.0. +This release also uses it to parse JSON streams, for example in `parser_json` with `in_exec`, and in `in_forward` and `in_unix` when they receive JSON. +Fluentd itself no longer uses `yajl-ruby`. + +There is one behavior change. +When an input stream ends in the middle of a JSON document, Fluentd now logs a warning and discards the incomplete tail instead of raising `ParseError`. +Complete JSON documents received before the truncation are processed as before. + +Fluentd now depends on `json` gem v2.21 or later. + +**Note for plugin developers:** Fluentd still keeps `yajl-ruby` as a dependency for a while, because some third-party plugins `require 'yajl'` without declaring it in their gemspec. If your plugin uses `yajl-ruby`, please add it to the dependencies of your gemspec. + +## Major new features and improvements + +### buffer: limit the total size of evacuated chunk files + +The new `evacuate_limit_size` option in the `` section limits the total size of evacuated chunk files. + +Fluentd v1.19.0 added a feature to evacuate buffer chunk files when the retry limit is reached. +During a long outage, this can happen again and again, and the evacuated files can fill the disk. +The new limit also counts the files evacuated before. +Chunks that do not fit are discarded without being evacuated. + +``` + + @type forward + @id forward_app + + host 192.168.1.3 + + + @type file + path /var/log/fluent/buf + evacuate_limit_size 1g + + +``` + +* The default is `nil` (no limit). It keeps the previous v1.19.x behavior. +* Set `0` to disable evacuation. It is the same as the behavior before v1.19.0. +* It is an approximate limit. The total size can exceed it, for example when multiple workers of the same plugin evacuate chunks at the same time. +* We recommend setting `@id` for the output plugin. Without `@id`, the evacuation directory can change when the configuration changes, and the files evacuated before are not counted. + +Ref. [Handling Successive Failures](https://docs.fluentd.org/buffer#handling-successive-failures) + +### in\_tail: discover new files earlier with directory watching + +The new opt-in `enable_dir_watcher` option lets `in_tail` discover new files sooner. + +`in_tail` usually finds new files by periodic checks every `refresh_interval` (60 seconds by default). +With this option, `in_tail` also watches the directories derived from `path`, and refreshes the file list when they change. + +``` + + @type tail + path /var/log/myapp/*.log + pos_file /var/log/fluent/myapp.pos + tag myapp + enable_dir_watcher true + follow_inodes true + + @type none + + +``` + +* `dir_watcher_limit` (default: `100`) sets the maximum number of directories watched at once. If there are more candidate directories, `in_tail` rotates the watched ones at each periodic refresh. +* Periodic refreshes still run as a fallback. +* Detection latency depends on the platform and the file system. On Linux, `inotify` is used where available. +* Rotated files that match `path` may also be found earlier. Use `follow_inodes true` to avoid duplicate collection after rename-based rotation. + +Ref. [enable\_dir\_watcher](https://docs.fluentd.org/input/tail#enabledirwatcher) + +### in\_tail: parse lines on worker threads + +The new opt-in `num_threads` option lets `in_tail` parse lines on worker threads. +It can reduce delays in processing other files while a large or fast-growing file is parsed. + +The default `1` keeps the previous behavior, which parses lines on the event loop. +Set it to `2` or more to enable worker parsing. + +``` + + @type tail + path /var/log/app/*.log + pos_file /var/log/fluentd/app.log.pos + tag app + num_threads 2 + + @type json + + +``` + +* It does not guarantee a higher overall throughput. If all worker threads are busy, other files may still wait. +* The order of the events within a file is preserved. +* Worker parsing is available with the `none`, `regexp`, `json`, `csv`, `tsv`, `ltsv`, `msgpack`, `apache`, `apache2`, `apache_error`, `nginx` and `syslog` parsers. In other cases, such as multiline parsing or `open_on_every_update true`, `in_tail` logs a warning at startup and keeps parsing on the event loop. +* While worker parsing is active, `in_monitor_agent` reports `worker_pending_batch_count`, `worker_pending_bytes` and `worker_parse_error_count`. + +Ref. [num\_threads](https://docs.fluentd.org/input/tail#numthreads) + +### in\_forward: improve interoperability with Fluent Bit v5.0.5 or later + +`in_forward` can now receive events sent by Fluent Bit v5.0.5 or later with its default settings. + +Fluent Bit v5.0.5 changed its `out_forward` plugin to send metadata within each event by default. +The timestamp of such an event is encapsulated with the metadata as `[EventTime, metadata]`, as defined in [Forward Protocol Specification v1.5](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5). +Previous versions of `in_forward` could not handle this format. +With the default `skip_invalid_event true`, these events were dropped with a `skip invalid event` warning. + +Now `in_forward` extracts the timestamp and ingests the record. +The metadata itself is not processed yet, and it is discarded. + +### System configuration: add `umask` option + +You can now set the umask of the Fluentd process with the `umask` option in the `` directive. +Previously, this required the `--umask` command line option, which is inconvenient for services and container images. + +``` + + umask 0022 + +``` + +When both are given, the command line option takes precedence. + +### out\_forward: add `send_keepalive_packet` option + +The new `send_keepalive_packet` option enables TCP keepalive (`SO_KEEPALIVE`) on `out_forward` connections. +The existing `keepalive` option only reuses connections and does not send TCP keepalive packets, so dead connections could remain in `CLOSE-WAIT` state. + +`send_keepalive_packet` is disabled by default, and it requires `keepalive true`. +Otherwise, Fluentd fails to start with a configuration error. +`in_forward` already has a similar option. + +``` + + @type forward + keepalive true + send_keepalive_packet true + + host 192.168.1.3 + + +``` + +### in\_exec: add `command_timeout` option + +The new `command_timeout` option limits how long an `in_exec` command can run. +When the command runs longer than this timeout, Fluentd logs `child process timed out` and kills the process. +The default is `nil` (no timeout). + +Without a timeout, a hung command makes `in_exec` keep logging `previous child process is still running. skipped.`, and it is hard to find the actual cause. +`out_exec` already has the same option. + +``` + + @type exec + command /path/to/command + run_interval 10s + command_timeout 30s + tag example + + @type json + + +``` + +### Performance improvements + +* `parser_csv`, `parser_tsv` and `parser_ltsv` are faster and allocate fewer objects. +* Output plugins cache metadata and timekey calculation to reduce object allocations. +* `formatter_out_file` is faster. +* More JSON handling uses the `json` gem instead of `yajl-ruby`. +* On Windows, `in_tail` reads files in 64KB blocks instead of 8KB, and `EventTime.now` is faster. + +### Other improvements + +* `parser_syslog`: accept one space between the RFC3164 priority and the header. Some syslog senders emit records in this format. +* `fluent-plugin-generate`: add a `.gitignore` file to the generated plugin template. +* The gem for Windows on Arm (`aarch64-mingw-ucrt`) is now available. + +## Major bug fixes + +### in\_tail: improved stability + +* Fixed duplicated logs when a rotated file is appended during `rotate_wait` with `follow_inodes true`. +* Fixed the remaining lines of a file never being collected when the file is rate-limited by `` and then removed from the watch list, for example by `limit_recently_modified`. +* Fixed `NameError` when flushing buffered lines with `emit_unmatched_lines true`. +* Fixed errors and timeouts at shutdown when the shutdown overlaps with file discovery or file rotation. +* `multiline_flush_interval` is no longer affected by system clock adjustments. + +### buffer: fixed a chunk being flushed twice + +Fixed a bug that could cause a file buffer chunk to be flushed twice, resulting in `closed stream` or `ENOENT` errors and incorrect buffer size metrics. +Also fixed the `stage_size` metric that could briefly become negative. + +### http\_server helper: fixed bind failures and IPv6 addresses + +These fixes affect plugins using the http\_server helper, such as `in_monitor_agent` and `fluent-plugin-prometheus`. + +* Fixed HTTP endpoints never recovering after a bind failure. Previously, the plugin started anyway without listening on the port. Now the plugin fails to start, and the worker is restarted until the port is available. You can slow down the restart with `restart_worker_interval` in the `` directive. +* Fixed `URI::InvalidURIError` when binding to IPv6 addresses such as `::` and `::1`. + +### supervisor: fixed graceful reload with YAML configuration + +Fixed graceful reload failing with YAML configuration files, for example when triggered through the RPC endpoint `/api/config.gracefulReload`. +Also fixed `<<: !include` of a file that contains a mapping. +It restores the documented [shared parameter syntax](https://docs.fluentd.org/configuration/config-file-yaml#share-the-same-parameters). + +### Other bug fixes + +* `out_file`: fixed missing symlinks when `symlink_path` is used with chunk keys other than time. Previously, only one of the chunks with the latest timekey got a symlink. +* `parser_msgpack`: fixed leftover data from truncated input mixing into the next request, connection, datagram or record. It affected `in_http`, `in_tcp`, `in_udp` and `filter_parser`. +* Fixed `emit_records` and `emit_size` metrics always being `0` for input plugins with `@label`. +* Fixed the time of an event at the Unix epoch that could be formatted as empty, for example in `out_file`. + +Enjoy logging! + +### Follow us on X + +We have been posting information about Fluentd in Japanese on [@fluentd_jp](https://x.com/fluentd_jp). +We would appreciate it if you followed the X account. + +TAG: Fluentd Announcement +AUTHOR: clearcode diff --git a/content/blog/tag/announcement b/content/blog/tag/announcement index 7c8a34cf..739ea991 100644 --- a/content/blog/tag/announcement +++ b/content/blog/tag/announcement @@ -165,4 +165,5 @@ /blog/20260626_fluent-package-v6.0.4-has-been-released /blog/20260814_fluent-package-v7-scheduled-lifecycle /blog/20260929_fluent-package-v6.0.5-has-been-released -/blog/20260929_fluentd-v1.19.4-has-been-released \ No newline at end of file +/blog/20260929_fluentd-v1.19.4-has-been-released +/blog/20261023_fluentd-v1.20.0-has-been-released \ No newline at end of file diff --git a/content/blog/tag/fluentd b/content/blog/tag/fluentd index 710b714e..eeaa6c3a 100644 --- a/content/blog/tag/fluentd +++ b/content/blog/tag/fluentd @@ -174,4 +174,5 @@ /blog/20260626_fluent-package-v6.0.4-has-been-released /blog/20260814_fluent-package-v7-scheduled-lifecycle /blog/20260929_fluent-package-v6.0.5-has-been-released -/blog/20260929_fluentd-v1.19.4-has-been-released \ No newline at end of file +/blog/20260929_fluentd-v1.19.4-has-been-released +/blog/20261023_fluentd-v1.20.0-has-been-released \ No newline at end of file