f5 “Log disk usage still higher than 80% after logrotate” – No Files Found to Delete

F5 error message “Log disk usage still higher than 80% after logrotate” is issued and “df” shows that the var/log partition is nearly full but “du” finds no files to delete. This article describes how to find the “invisible” log files and regain the disk space.

Description of the Issue

The following error can be seen logged to the console:

011d0004:3: Disk partition var_log has only 7% free
Feb 25 09:25:03 f5-01 emerg alertd[7775]: 01100048:0: Log disk usage still higher than 80% after logrotate and 24 times log deletion

“df” shows that the /var/log partition is nearing its limit:

[admin@f5-01:Active:In Sync] ~ # df -h
Filesystem            Size  Used Avail Use% Mounted on
<snip>
/dev/mapper/vg--db--sda-dat.log.1
                      485M  427M   33M  93% /var/log
<snip>

“du” however shows that there are only a few MBs in the /var/log directory:

[admin@f5-01:Active:In Sync] log # du -h ./ –max-depth=1
<snip>
26M     ./

Cause

Monitor logging has been enabled for e.g. a node which is ongoing. The file however has been deleted but as it is still being written to, the disk space is still in use.

Fix

We simply need to identify fo which node the monitoring has been enabled and then disable it. We can see a list of the files which have been deleted from the log directory with the following command:

[admin@f5edin:Active:In Sync] monitors # lsof | grep deleted | grep log
bigd   6775   root  7w   REG   253,2  407867086  109730 /var/log/monitors/Common_icmp-Common_panel1-0.log.1 (deleted)

Now go to Local Traffic -> Nodes -> Select the node and uncheck the monitor logging box:

monitor logging

Running “df” should now show plenty of recovered disk space:

[admin@f5-01:Active:In Sync] log # df -h
Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg--db--sda-dat.log.1
                      485M   36M  424M   8% /var/log

CLI: Create a Self IP to Allow BigIP Management

Create a Self IP to Allow BigIP Management via the CLI

This article details how to create a self ip  to allow bigip management on the command line. The need arose when a primary firewall failed and the only entry to the network was an ssh session to the DR BigIP via a jump box.

For the purpose of this article we are using the following IP representations:

x.x.x.100 – eXternal IP

i.i.i.100 – Internal IP

1. Enter the traffic management shell tmsh:

[admin@bigip:Active:Standalone] tmsh
admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)#

2. If there is already a self IP on the external vlan you can modify the port lockdown settings to allow https to the machine but only if you have no virtual servers already using the https port.

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# modify net self x.x.x.100 allow-service default

This will allow all the default ports, shown by “list net self-allow”:

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net self-allow
net self-allow {
defaults {
ospf:any
tcp:domain
tcp:f5-iquery
tcp:https
tcp:snmp
tcp:ssh
udp:520
udp:cap
udp:domain
udp:f5-iquery
udp:snmp
}
}

3. In this case the self IP was already used by a virtual server with a pool of https machines behind it so a new self ip was necessary. Choose another ip on the same external subnet:

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# create net self x.x.x.101/32 vlan external allow-service default

4. We can now list our self ips and the level of access associated with them:

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net self
net self x.x.x.101/32 {
address 90.152.115.244/32
allow-service {
default
}
traffic-group traffic-group-local-only
vlan external
}
net self x.x.x.101 {
address 90.152.115.243/29
allow-service {
none
}
traffic-group traffic-group-local-only
vlan external
}
net self i.i.i.100 {
address i.i.i.100/24
allow-service {
default
}
traffic-group traffic-group-local-only
vlan internal
}

5. We can now access our BigIP on https://x.x.x.101 !

f5 VE on ESXi: “The requested media for interface 1.1 is invalid.”

This article addresses the following error: “The requested media for interface 1.1 is invalid” when using the f5 virtual edition on ESXi.

Once possible reason for this error is mentioned under “Known Issues” in the release notes for 11.x e.g. : http://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ve-11-4-0.print.html

The entry reads as follows:

ID 352856 “If an SCF is migrated between BIG-IP VE running on non-similar hypervisor software, a validation error may prevent configuration loading. Loading the configuration … BIGpipe interface creation error: 01070318:3: “”The requested media for interface 1.1 is invalid.”” When this condition is encountered on BIG-IP Virtual Edition, configuration may be fixed for import by removing the entire line that contains “”media fixed”” statements for each interface.”

If however, like me, you cannot find the “media fixed” anywhere in your bigip_base.conf file then it is most likely to be an issue with the vmxnet3 network adapters that are deployed by default.

My management adapter, also vmxnet3, came up fine but the other 1.1, 1.2 and 1.3 interfaces remained uninitialised and any attempts to edit just threw the error above.

My solution was to change the adapter types in the .vmx file for the virtual machine:

1. Shut down the machine2. SSH / console into your ESXi host and change directory to /vmfs/volumes/<datastore_name>/<vm_directory>f5 VE on ESXi: "The requested media for interface 1.1 is invalid."

 

 

 

 

 

3. Use the “vi” command to edit the <your_vm>.vmx file and change the “vmxnet3” entries to “e1000.” Note: you can generally leave the first interface (management) as vmxnet3.

f5 VE on ESXi: "The requested media for interface 1.1 is invalid."

 

 

 

 

 

 

 

 

 

 

4. Save the file and start up your machine – you should now be able to initialise and edit your interfaces under “Network” -> “Interfaces”

Job done, let me know if this works / doesn’t work for you!