f5 “Log disk usage still higher than 80% after logrotate” – No Files Found to Delete

f5 “Log disk usage still higher than 80% after logrotate” – No Files Found to Delete

F5 error message “Log disk usage still higher than 80% after logrotate” is issued and “df” shows that the var/log partition is nearly full but “du” finds no files to delete. This article describes how to find the “invisible” log files and regain the disk space.

Description of the Issue

The following error can be seen logged to the console:

011d0004:3: Disk partition var_log has only 7% free
Feb 25 09:25:03 f5-01 emerg alertd[7775]: 01100048:0: Log disk usage still higher than 80% after logrotate and 24 times log deletion

“df” shows that the /var/log partition is nearing its limit:

[admin@f5-01:Active:In Sync] ~ # df -h
Filesystem            Size  Used Avail Use% Mounted on
<snip>
/dev/mapper/vg--db--sda-dat.log.1
                      485M  427M   33M  93% /var/log
<snip>

“du” however shows that there are only a few MBs in the /var/log directory:

[admin@f5-01:Active:In Sync] log # du -h ./ –max-depth=1
<snip>
26M     ./

Cause

Monitor logging has been enabled for e.g. a node which is ongoing. The file however has been deleted but as it is still being written to, the disk space is still in use.

Fix

We simply need to identify fo which node the monitoring has been enabled and then disable it. We can see a list of the files which have been deleted from the log directory with the following command:

[admin@f5edin:Active:In Sync] monitors # lsof | grep deleted | grep log
bigd   6775   root  7w   REG   253,2  407867086  109730 /var/log/monitors/Common_icmp-Common_panel1-0.log.1 (deleted)

Now go to Local Traffic -> Nodes -> Select the node and uncheck the monitor logging box:

monitor logging

Running “df” should now show plenty of recovered disk space:

[admin@f5-01:Active:In Sync] log # df -h
Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg--db--sda-dat.log.1
                      485M   36M  424M   8% /var/log

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.