f5 Master Key for RMA and Migration

f5 Master Key Backup and Restore

The f5 master key is used to encrypt and decrypt all that is secure on your f5 appliance including certificate keys, passphrases and UCS configuration files; this is obviously therefore an absolutely vital piece of information in certain situations.

If you have a synchronized cluster then this is not so much of an issue: when you add a new device to the cluster – be that a new member or an RMA replacement for a failed appliance – then the master key will be synchronized as well.

Situations where this can really impact your ability to get back up and running quickly is when you have a standalone appliance or you are transferring config from one machine to another:

Scenario:

  • Standalone appliance fails, monitoring systems go crazy and red lights start blinking everywhere – you call it in and you have a new machine on site in 4 hours (if you bought the right support contract!)
  • You plug it in and give it a management IP and access the GUI
  • Feeling smug because you’ve taken regular backups and stored them offline, you upload your latest UCS archive to restore the configuration
  • Config load fails as it is unable to decrypt the SSL keys passphrases, LDAP profiles passphrases, cookie encryption passphrases etc. etc.
  • You may see the following in the logs: Decryption of the field (field_name) for object (object_name) failed

Resolution:

  • Trawl through the config, edit out the passphrases and re-key most of it from scratch

OR

  • Change the master key to the one from the failed appliance, upload your UCS config and engage ultra-smug mode.

This is very easily done using the “f5 Master Key Utility” and should form part of your backup process for all your f5 appliances:

Backup the Master Key Using f5mku

  • Use the “-K” switch to display the master key and then copy the resulting key securely to an offline vault:
[root@f5] ~ # f5mku -K
7gcOKj3dhFjhr4mFy0AU1p==

Restore the Master Key Uing f5mku

  • Use the “-r” switch to restore the key to your new appliance:
[root@f5] ~ # f5mku -r 7gcOKj3dhFjhr4mFy0AU1p==
Rekeying Master Key...
[root@f5] ~ #
  • Job done! Now you can upload the UCS config archive without needing to worry about decryption failures.

 

Note: f5kmu Options

[root@f5] ~ # f5mku --h
f5mku: invalid option -- -

Usage: f5mku [d:?fhHr:t:uUvYZ]

Commands: (one of these must be specified)
-d bits generate a base64 encoded RSA key and output to stdout
-f fetch unit key
-Z dump debug information
-r key rekey with the specifed master key (b64 encoded)

Options:
-? -h this help
-t # Timeout value in seconds (1-500)
-u Unit test posture (no HAL)
-U Test unit key functionality.
-H Force I/O to HAL storage
-v set verbose mode
-Y Answer Yes to any queries

Log messages (including debug) go to authpriv and local6 facilities.

f5 “Log disk usage still higher than 80% after logrotate” – No Files Found to Delete

F5 error message “Log disk usage still higher than 80% after logrotate” is issued and “df” shows that the var/log partition is nearly full but “du” finds no files to delete. This article describes how to find the “invisible” log files and regain the disk space.

Description of the Issue

The following error can be seen logged to the console:

011d0004:3: Disk partition var_log has only 7% free
Feb 25 09:25:03 f5-01 emerg alertd[7775]: 01100048:0: Log disk usage still higher than 80% after logrotate and 24 times log deletion

“df” shows that the /var/log partition is nearing its limit:

[admin@f5-01:Active:In Sync] ~ # df -h
Filesystem            Size  Used Avail Use% Mounted on
<snip>
/dev/mapper/vg--db--sda-dat.log.1
                      485M  427M   33M  93% /var/log
<snip>

“du” however shows that there are only a few MBs in the /var/log directory:

[admin@f5-01:Active:In Sync] log # du -h ./ –max-depth=1
<snip>
26M     ./

Cause

Monitor logging has been enabled for e.g. a node which is ongoing. The file however has been deleted but as it is still being written to, the disk space is still in use.

Fix

We simply need to identify fo which node the monitoring has been enabled and then disable it. We can see a list of the files which have been deleted from the log directory with the following command:

[admin@f5edin:Active:In Sync] monitors # lsof | grep deleted | grep log
bigd   6775   root  7w   REG   253,2  407867086  109730 /var/log/monitors/Common_icmp-Common_panel1-0.log.1 (deleted)

Now go to Local Traffic -> Nodes -> Select the node and uncheck the monitor logging box:

monitor logging

Running “df” should now show plenty of recovered disk space:

[admin@f5-01:Active:In Sync] log # df -h
Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg--db--sda-dat.log.1
                      485M   36M  424M   8% /var/log

CLI: Create a Self IP to Allow BigIP Management

Create a Self IP to Allow BigIP Management via the CLI

This article details how to create a self ip  to allow bigip management on the command line. The need arose when a primary firewall failed and the only entry to the network was an ssh session to the DR BigIP via a jump box.

For the purpose of this article we are using the following IP representations:

x.x.x.100 – eXternal IP

i.i.i.100 – Internal IP

1. Enter the traffic management shell tmsh:

[admin@bigip:Active:Standalone] tmsh
admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)#

2. If there is already a self IP on the external vlan you can modify the port lockdown settings to allow https to the machine but only if you have no virtual servers already using the https port.

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# modify net self x.x.x.100 allow-service default

This will allow all the default ports, shown by “list net self-allow”:

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net self-allow
net self-allow {
defaults {
ospf:any
tcp:domain
tcp:f5-iquery
tcp:https
tcp:snmp
tcp:ssh
udp:520
udp:cap
udp:domain
udp:f5-iquery
udp:snmp
}
}

3. In this case the self IP was already used by a virtual server with a pool of https machines behind it so a new self ip was necessary. Choose another ip on the same external subnet:

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# create net self x.x.x.101/32 vlan external allow-service default

4. We can now list our self ips and the level of access associated with them:

admin@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# list net self
net self x.x.x.101/32 {
address 90.152.115.244/32
allow-service {
default
}
traffic-group traffic-group-local-only
vlan external
}
net self x.x.x.101 {
address 90.152.115.243/29
allow-service {
none
}
traffic-group traffic-group-local-only
vlan external
}
net self i.i.i.100 {
address i.i.i.100/24
allow-service {
default
}
traffic-group traffic-group-local-only
vlan internal
}

5. We can now access our BigIP on https://x.x.x.101 !