in_tail: Repeated Logs Despite pos_file Updating in Fluentd(LTS) on Linux 2023 #4761
Comments
|
@tadkarshirish
So, Please check how the file is updated.
|
daipom
added
waiting-for-user
daipom
changed the title
|
We are testing with Sample log file which we are updating with following steps. 1. Create a Sample Log File Add sample log entries manually (e.g., a, b, c, d) and save the file using :wq. 2. Create Fluentd Configuration
Add the following sample configuration, ensuring correct paths for the log and pos file: Save and exit using 3. Start Fluentd
4. Edit the Sample Log File
Append new log entries (e.g., e, f) Save and exit using the secure save command: 5. Check Fluentd Output Monitor the Fluentd logs or check the stdout output to verify that log entries have been processed. 6. Repeat Log Editing and Save Reopen log1.txt using |
|
@tadkarshirish $ vi fluentd.conf
$ cat fluentd.conf
<source>
@type tail
path /tmp/2/log1.txt
pos_file /tmp/2/pos/pos
tag demosvc
format none
</source>
<match demosvc>
@type stdout
</match>
$ vi log1.txt
$ cat log1.txt
a
b
c
d
(Launch Fluentd with another shell here)
$ vi log1.txt
$ cat log1.txt
a
b
c
d
e
fFluentd $ bundle exec fluentd -c /tmp/2/fluentd.conf
2025-01-15 15:20:15 +0900 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2025-01-15 15:20:15 +0900 [info]: parsing config file is succeeded path="/tmp/2/fluentd.conf"
2025-01-15 15:20:15 +0900 [info]: gem 'fluentd' version '1.18.0'
2025-01-15 15:20:15 +0900 [info]: using configuration file: <ROOT>
<source>
@type tail
path "/tmp/2/log1.txt"
pos_file "/tmp/2/pos/pos"
tag "demosvc"
format none
<parse>
@type none
unmatched_lines
</parse>
</source>
<match demosvc>
@type stdout
</match>
</ROOT>
2025-01-15 15:20:15 +0900 [info]: starting fluentd-1.18.0 pid=732107 ruby="3.2.2"
2025-01-15 15:20:15 +0900 [info]: spawn command to main: cmdline=["/home/daipom/.rbenv/versions/3.2.2/bin/ruby", "-r/home/daipom/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/setup", "-Eascii-8bit:ascii-8bit", "/home/daipom/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/bin/fluentd", "-c", "/tmp/2/fluentd.conf", "--under-supervisor"]
2025-01-15 15:20:15 +0900 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2025-01-15 15:20:15 +0900 [info]: adding match pattern="demosvc" type="stdout"
2025-01-15 15:20:15 +0900 [info]: adding source type="tail"
2025-01-15 15:20:15 +0900 [info]: #0 starting fluentd worker pid=732127 ppid=732107 worker=0
2025-01-15 15:20:15 +0900 [info]: #0 following tail of /tmp/2/log1.txt
2025-01-15 15:20:15 +0900 [info]: #0 fluentd worker is now running worker=0
2025-01-15 15:20:31.528486080 +0900 demosvc: {"message":"e"}
2025-01-15 15:20:31.528499539 +0900 demosvc: {"message":"f"} |
|
as per your steps its working fine and there is no log duplication for first logfile addition. , but after editing log file few times it is showing duplication issue. |
Describe the bug
We are experiencing an issue where fluentd repeatedly processes the same logs despite the pos_file being updated. The problem persists even after verifying the file permissions and Fluenetd's configurations. Logs are repeated in fluentd output and debug logs indicate re-reading of the same log lines.
To Reproduce
Download Link : https://td-agent-package-browser.herokuapp.com/lts/5/amazon/2023/x86_64
Expected behavior
Fluent should process each log line once and continue from the last position after a restart or rotation.
Your Environment
Your Configuration
Your Error Log
Additional context
in logfile we have first added ,
a,b,c,dthene,f,g,htheni,j,k,l,m,n,pbut every time its giving whole log file output. even pos_file is updating the first Unique number updates every time after log change the second unique number remains same whole time./home/ec2-user/log1.txt 000000000000001e 000000000025f4f5The text was updated successfully, but these errors were encountered: