[F5 LTM] Ratio Load Balancing Controlled Server Side

Posted: Feb 20, 2023

The initial idea of this scope come from an internal client who asked “can we have a way to control the ratio of the VIP and healthyness from a json present on the server” ?

The usual methods to allow for this kind of interaction are:

(2) Standard monitor can indeed check a json but cannot control the ratio (example 1)
Dynamic Ratio Load Balancing can interact with the server but with SNMP or WMI and monitors almost all the server components like CPU, Memory and Processor but not a json file. We will not test this methode here.
(3) External monitor allow for any kind of interaction with the system so let’s see how to create one.

Initial setup and testing

For this test, let’s create a simple VIP

F5 Ratio JSON

3root@(ltm)(cfg-sync Standalone)(Active)(/Common)(tmos)# list ltm virtual vip
ltm virtual vip {
    creation-time 2022-02-19:13:49:12
    destination 10.10.10.11:http
    ip-protocol tcp
    last-modified-time 2022-02-19:13:52:19
    mask 255.255.255.255
    pool srv
    profiles {
        tcp { }
    }
    serverssl-use-sni disabled
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    translate-address enabled
    translate-port enabled
    vs-index 2
}
root@(ltm)(cfg-sync Standalone)(Active)(/Common)(tmos)# list ltm pool
ltm pool srv {
    load-balancing-mode ratio-member
    members {
        srv1:http {
            address 172.16.181.136
            session monitor-enabled
            state up
        }
        srv2:http {
            address 172.16.181.137
            session monitor-enabled
            state up
        }
        srv1:http {
            address 172.16.181.138
            session monitor-enabled
            state up
        }
    }
    monitor http
}
ltm monitor http alive_json {
    adaptive disabled
    defaults-from http
    interval 1
    ip-dscp 0
    recv .*alive.*:.*1.*,
    recv-disable .*alive.*:.*0.*,
    send "GET /lbkeepalive.json\r\n"
    time-until-up 0
    timeout 2
}

The monitor is checking the json file and with a regex check the line “alive”. If 1 is present, everything is good. If 0 is present, the member is user deactivated. Anything else the member is errored.

🔛 Let’s see how the ratio balancing is working with 10 concurrent users 100 times (1000 connections):

❯ siege http://10.10.10.11/lbkeepalive.json -c 10 -r 100                                                                                                                                 
** SIEGE 4.1.1
** Preparing 10 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json

[...]

HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.01 secs:      57 bytes ==> GET  /lbkeepalive.json

Transactions:		        1000 hits
Availability:		      100.00 %
Elapsed time:		        5.03 secs
Data transferred:	        0.05 MB
Response time:		        0.05 secs
Transaction rate:	      198.81 trans/sec
Throughput:		        0.01 MB/sec
Concurrency:		        9.90
Successful transactions:        1000
Failed transactions:	           0
Longest transaction:	        0.07
Shortest transaction:	        0.01

And we have a working configuration with the correct ratio ✅:

(tmos)# show ltm pool srv members field-fmt | grep "node-name\|serverside.tot-conns"
            node-name srv1
            serverside.tot-conns 334
            node-name srv2
            serverside.tot-conns 332
            node-name srv1
            serverside.tot-conns 334
    serverside.tot-conns 1.0K

Quick test by modifying the ratio and it looks good:

(tmos)# modify ltm pool srv members modify { srv1:80 { ratio 10 } }
(tmos)# reset-stats ltm pool srv
## TESTING ##
(tmos)# show ltm pool srv members field-fmt | grep "node-name\|serverside.tot-conns"
            node-name srv1
            serverside.tot-conns 832
            node-name srv2
            serverside.tot-conns 84
            node-name srv1
            serverside.tot-conns 84
    serverside.tot-conns 1.0K

Simple HTTP Monitor with JSON basic awareness

We have configured the monitor to take benefit of the json with (configuration extract):

ltm monitor http alive_json {
    defaults-from http                 # It is an http monitor
    recv .*alive.*:.*1.*,              # Check if it is alive
    recv-disable .*alive.*:.*0.*,      # Check if it is must be user deactivated
    send "GET /lbkeepalive.json\r\n"   # Path of the json file with standard get
}

Let’s see how the monitor react with the different values of alive:

Alive

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "alive": 1,
  "ratio": 1
}

User Deactivated

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "alive": 0,
  "ratio": 1
}

Errored

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "ratio": 1
}

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "alive": -1,
  "ratio": 1
}

External Monitor POC

First, before integrating this in an external monitor, we are going to mock it up on the LTM with echo output for easier explaination.

content=$(curl-apd -s -k -1 http://172.16.181.136/lbkeepalive.json)
alive=$( jq -r ".alive" <<< "${content}" )
ratio=$( jq -r ".ratio" <<< "${content}" )
#echo "alive:$alive"
#echo "ratio:$ratio"

# The following command do the following:
# List the configuration of the pool
# For the the server concerned, search the beginning of its config file with its name and the end with the space preceding the closure accolade
# We grep the line with the ratio config (space for exact match to differ from dynamic-ratio)
# We use awk to print the second term which is the ratio value
curratio=$(tmsh list ltm pool srv members  | sed -n '/srv1/,/^        }/p' | grep ' ratio ' | awk '{ print $2 }')

state=$(tmsh list ltm pool srv members  | sed -n '/srv1/,/^        }/p' | grep state | awk {'print $2}')

echo "current ratio is $curratio"
echo "current state is $state"

if [ "$alive" = "0" ]
then
   # Alive is present but with 0 value. Immediate deactivation of the member (if up before). Current sessions will persist to the member.
   echo "alive is 0 -> user disabled"
   tmsh modify ltm pool srv members modify { srv1:80 { session user-disabled } }

elif [ "$alive" = "1" ]
then
   # Alive is present and set. Checking ratio value and change it if different of current one
   echo "alive is 1"
   if [ "$state" != "up" ]; then
     echo "Uping the member"
     tmsh modify ltm pool srv members modify { srv1:80 { session user-enabled  } }
     tmsh modify ltm pool srv members modify { srv1:80 { state user-up  } }
   fi
   if [ "$ratio" != "$curratio" ]; then
   	 echo "Changing ratio to $ratio"
     tmsh  modify ltm pool srv members modify { srv1:80 { ratio $ratio } }
   fi
   
elif [ "$alive" = "-1" ]
then
   echo "alive is -1 -> user down"
   # Alive is present but with -1 value. Immediate shutdown of the member with no keeping of persistent session
   tmsh modify ltm pool srv members modify { srv1:80 { state user-down } }

else
    # Alive not present or with wrong value. Down
   echo "alive is not present -> user down"
   tmsh modify ltm pool srv members modify { srv1:80 { state user-down } }
fi

We are now are going to check the different values:

Initial Configuration

User disabled (keep current connection persistence)

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv1",
  "alive": "0",
  "ratio": "1"
}

[root@ltm] # bash alive_json.sh
current ratio is 1
current state is up
alive is 0 -> user disabled

User down

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv1",
  "alive": "-1",
  "ratio": "1"
}

[root@ltm] # bash alive_json.sh
current ratio is 1
current state is up
alive is -1 -> user down

Changing ratio

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv1",
  "alive": "1",
  "ratio": "20"
}

[root@ltm] # bash alive_json.sh
current ratio is 1
current state is user-down
alive is 1
Uping the member
Changing ratio to 20

Proper external Monitor

Now F5 expect a certain template for external monitor (check examples in System > File Management > External Monitor Program File List)

Here is the draft of the previous script integrated as an external monitor. More testing is needed to cover the corner case (and I will perhaps switch to python for easier troubleshooting).

#!/bin/sh

#
# these arguments supplied automatically for all external pingers:
# $1 = IP (::ffff:nnn.nnn.nnn.nnn notation or hostname)
# $2 = port (decimal, host byte order)
# $3 and higher = additional arguments
# 
# $MONITOR_NAME = name of the monitor
# 
# In this sample script, $3 is the regular expression
#

# Name of the pidfile
pidfile="/var/run/$MONITOR_NAME.$1..$2.pid"

#echo "EAV `basename $0`: \$DEBUG: $DEBUG" | logger -p local0.debug
#echo "EAV IP $1" | logger -p local0.debug
#echo "EAV Port $2" | logger -p local0.debug
#printenv  | logger -p local0.debug

# Send signal to the process group to kill our former self and any children 
# as external monitors are run with SIGHUP blocked
if [ -f $pidfile ]
then
   kill -9 -`cat $pidfile` > /dev/null 2>&1
fi

echo "$$" > $pidfile

# Remove the IPv6/IPv4 compatibility prefix 
node_ip=`echo $1 | sed 's/::ffff://'`
node_name=$( echo $NODE_NAME | cut -d'/' -f3)
node_full="$node_name:$2"

### BEGIN CHECK ####

content=$(curl-apd -s -k -1 http://$node_ip:$2/lbkeepalive.json)
alive=$( jq -r ".alive" <<< "${content}" )
ratio=$( jq -r ".ratio" <<< "${content}" )
#echo "alive:$alive"
#echo "ratio:$ratio"



# The following command do the following:
# List the configuration of the pool
# For the the server concerned, search the beginning of its config file with its name and the end with the space preceding the closure accolade
# We grep the line with the ratio config (space for exact match to differ from dynamic-ratio)
# We use awk to print the second term which is the ratio value
curratio=$(tmsh list ltm pool srv members  | sed -n '/$node_name/,/^        }/p' | grep ' ratio ' | awk '{ print $2 }')

state=$(tmsh list ltm pool srv members  | sed -n '/$node_name/,/^        }/p' | grep "state" | awk {'print $2}')

#echo "$node_name: current ratio is $curratio" | logger -p local0.debug
#echo "$node_name: alive: $alive" | logger -p local0.debug
#echo "$node_full: state is $state" | logger -p local0.debug

if [ "$alive" = "0" ]
then
   # Alive is present but with 0 value. Immediate deactivation of the member (if up before). Current sessions will persist to the member.
#   echo "alive is 0"
   tmsh modify ltm pool srv members modify { $node_full { session user-disabled } }

elif [ "$alive" = "1" ]
then
   # Alive is present and set. Checking ratio value and change it if different of current one
#   echo "alive is 1"

   if [ "$state" != "up" ]; then
     tmsh modify ltm pool srv members modify { $node_full { session user-enabled  } }
     tmsh modify ltm pool srv members modify { $node_full { state user-up  } }
   fi
   if [ "$ratio" != "$curratio" ]; then
     tmsh  modify ltm pool srv members modify { $node_full { ratio $ratio } }
   fi
   # Remove the pidfile before the script echoes anything to stdout and is killed by bigd      
   rm -f $pidfile
   echo "up"

elif [ "$alive" = "-1" ]
then
#   echo "alive is -1"
   # Alive is present but with -1 value. Immediate shutdown of the member with no keeping of persistent session
   tmsh modify ltm pool srv members modify { $node_full { state user-down } }

else
    # Alive not present or with wrong value. Down
#   echo "alive is not present"
   tmsh modify ltm pool srv members modify { $node_full { state user-down } }
fi

# Remove the pidfile before the script ends
rm -f $pidfile

Ressources

The following links contains useful information around external monitors:

https://community.f5.com/t5/technical-articles/ltm-external-monitors-the-basics/ta-p/277128

https://community.f5.com/t5/technical-articles/external-monitor-information-and-templates/ta-p/289008

https://community.f5.com/t5/technical-articles/ltm-external-monitors-troubleshooting/ta-p/277124

https://support.f5.com/csp/article/K71282813

jmanteau

Mon coin de toile - A piece of Web