jmanteau

Mon coin de toile - A piece of Web

[F5 LTM] Ratio Load Balancing Controlled Server Side

Posted: Feb 20, 2022

The initial idea of this scope come from an internal client who asked “can we have a way to control the ratio of the VIP and healthyness from a json present on the server” ?

The usual methods to allow for this kind of interaction are:

Initial setup and testing

For this test, let’s create a simple VIP

F5 Ratio JSON

3root@(ltm)(cfg-sync Standalone)(Active)(/Common)(tmos)# list ltm virtual vip
ltm virtual vip {
    creation-time 2022-02-19:13:49:12
    destination 10.10.10.11:http
    ip-protocol tcp
    last-modified-time 2022-02-19:13:52:19
    mask 255.255.255.255
    pool srv
    profiles {
        tcp { }
    }
    serverssl-use-sni disabled
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    translate-address enabled
    translate-port enabled
    vs-index 2
}
root@(ltm)(cfg-sync Standalone)(Active)(/Common)(tmos)# list ltm pool
ltm pool srv {
    load-balancing-mode ratio-member
    members {
        srv1:http {
            address 172.16.181.136
            session monitor-enabled
            state up
        }
        srv2:http {
            address 172.16.181.137
            session monitor-enabled
            state up
        }
        srv1:http {
            address 172.16.181.138
            session monitor-enabled
            state up
        }
    }
    monitor http
}
ltm monitor http alive_json {
    adaptive disabled
    defaults-from http
    interval 1
    ip-dscp 0
    recv .*alive.*:.*1.*,
    recv-disable .*alive.*:.*0.*,
    send "GET /lbkeepalive.json\r\n"
    time-until-up 0
    timeout 2
}

The monitor is checking the json file and with a regex check the line “alive”. If 1 is present, everything is good. If 0 is present, the member is user deactivated. Anything else the member is errored.

🔛 Let’s see how the ratio balancing is working with 10 concurrent users 100 times (1000 connections):

❯ siege http://10.10.10.11/lbkeepalive.json -c 10 -r 100                                                                                                                                 
** SIEGE 4.1.1
** Preparing 10 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.03 secs:      57 bytes ==> GET  /lbkeepalive.json

[...]

HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.02 secs:      57 bytes ==> GET  /lbkeepalive.json
HTTP/1.1 200     0.01 secs:      57 bytes ==> GET  /lbkeepalive.json

Transactions:		        1000 hits
Availability:		      100.00 %
Elapsed time:		        5.03 secs
Data transferred:	        0.05 MB
Response time:		        0.05 secs
Transaction rate:	      198.81 trans/sec
Throughput:		        0.01 MB/sec
Concurrency:		        9.90
Successful transactions:        1000
Failed transactions:	           0
Longest transaction:	        0.07
Shortest transaction:	        0.01

And we have a working configuration with the correct ratio ✅:

(tmos)# show ltm pool srv members field-fmt | grep "node-name\|serverside.tot-conns"
            node-name srv1
            serverside.tot-conns 334
            node-name srv2
            serverside.tot-conns 332
            node-name srv1
            serverside.tot-conns 334
    serverside.tot-conns 1.0K

Quick test by modifying the ratio and it looks good:

(tmos)# modify ltm pool srv members modify { srv1:80 { ratio 10 } }
(tmos)# reset-stats ltm pool srv
## TESTING ##
(tmos)# show ltm pool srv members field-fmt | grep "node-name\|serverside.tot-conns"
            node-name srv1
            serverside.tot-conns 832
            node-name srv2
            serverside.tot-conns 84
            node-name srv1
            serverside.tot-conns 84
    serverside.tot-conns 1.0K

Simple HTTP Monitor with JSON basic awareness

We have configured the monitor to take benefit of the json with (configuration extract):

ltm monitor http alive_json {
    defaults-from http                 # It is an http monitor
    recv .*alive.*:.*1.*,              # Check if it is alive
    recv-disable .*alive.*:.*0.*,      # Check if it is must be user deactivated
    send "GET /lbkeepalive.json\r\n"   # Path of the json file with standard get
}

Let’s see how the monitor react with the different values of alive:

Alive

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "alive": 1,
  "ratio": 1
}

image-20220220170800853

User Deactivated

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "alive": 0,
  "ratio": 1
}

image-20220220170638612

Errored

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "ratio": 1
}

or

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv3",
  "alive": -1,
  "ratio": 1
}

image-20220220170714740

External Monitor POC

First, before integrating this in an external monitor, we are going to mock it up on the LTM with echo output for easier explaination.

content=$(curl-apd -s -k -1 http://172.16.181.136/lbkeepalive.json)
alive=$( jq -r ".alive" <<< "${content}" )
ratio=$( jq -r ".ratio" <<< "${content}" )
#echo "alive:$alive"
#echo "ratio:$ratio"

# The following command do the following:
# List the configuration of the pool
# For the the server concerned, search the beginning of its config file with its name and the end with the space preceding the closure accolade
# We grep the line with the ratio config (space for exact match to differ from dynamic-ratio)
# We use awk to print the second term which is the ratio value
curratio=$(tmsh list ltm pool srv members  | sed -n '/srv1/,/^        }/p' | grep ' ratio ' | awk '{ print $2 }')

state=$(tmsh list ltm pool srv members  | sed -n '/srv1/,/^        }/p' | grep state | awk {'print $2}')

echo "current ratio is $curratio"
echo "current state is $state"

if [ "$alive" = "0" ]
then
   # Alive is present but with 0 value. Immediate deactivation of the member (if up before). Current sessions will persist to the member.
   echo "alive is 0 -> user disabled"
   tmsh modify ltm pool srv members modify { srv1:80 { session user-disabled } }

elif [ "$alive" = "1" ]
then
   # Alive is present and set. Checking ratio value and change it if different of current one
   echo "alive is 1"
   if [ "$state" != "up" ]; then
     echo "Uping the member"
     tmsh modify ltm pool srv members modify { srv1:80 { session user-enabled  } }
     tmsh modify ltm pool srv members modify { srv1:80 { state user-up  } }
   fi
   if [ "$ratio" != "$curratio" ]; then
   	 echo "Changing ratio to $ratio"
     tmsh  modify ltm pool srv members modify { srv1:80 { ratio $ratio } }
   fi
   
elif [ "$alive" = "-1" ]
then
   echo "alive is -1 -> user down"
   # Alive is present but with -1 value. Immediate shutdown of the member with no keeping of persistent session
   tmsh modify ltm pool srv members modify { srv1:80 { state user-down } }

else
    # Alive not present or with wrong value. Down
   echo "alive is not present -> user down"
   tmsh modify ltm pool srv members modify { srv1:80 { state user-down } }
fi

We are now are going to check the different values:

Initial Configuration

image-20220220171543577

User disabled (keep current connection persistence)

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv1",
  "alive": "0",
  "ratio": "1"
}
[root@ltm] # bash alive_json.sh
current ratio is 1
current state is up
alive is 0 -> user disabled

image-20220220171917437

User down

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv1",
  "alive": "-1",
  "ratio": "1"
}
[root@ltm] # bash alive_json.sh
current ratio is 1
current state is up
alive is -1 -> user down

image-20220220171806219

Changing ratio

root@debian11:/home/debian# cat /var/www/html/lbkeepalive.json | jq
{
  "srv": "srv1",
  "alive": "1",
  "ratio": "20"
}
[root@ltm] # bash alive_json.sh
current ratio is 1
current state is user-down
alive is 1
Uping the member
Changing ratio to 20

image-20220220172501795

Proper external Monitor

Now F5 expect a certain template for external monitor (check examples in System > File Management > External Monitor Program File List)

Here is the draft of the previous script integrated as an external monitor. More testing is needed to cover the corner case (and I will perhaps switch to python for easier troubleshooting).

#!/bin/sh

#
# these arguments supplied automatically for all external pingers:
# $1 = IP (::ffff:nnn.nnn.nnn.nnn notation or hostname)
# $2 = port (decimal, host byte order)
# $3 and higher = additional arguments
# 
# $MONITOR_NAME = name of the monitor
# 
# In this sample script, $3 is the regular expression
#

# Name of the pidfile
pidfile="/var/run/$MONITOR_NAME.$1..$2.pid"

#echo "EAV `basename $0`: \$DEBUG: $DEBUG" | logger -p local0.debug
#echo "EAV IP $1" | logger -p local0.debug
#echo "EAV Port $2" | logger -p local0.debug
#printenv  | logger -p local0.debug

# Send signal to the process group to kill our former self and any children 
# as external monitors are run with SIGHUP blocked
if [ -f $pidfile ]
then
   kill -9 -`cat $pidfile` > /dev/null 2>&1
fi

echo "$$" > $pidfile

# Remove the IPv6/IPv4 compatibility prefix 
node_ip=`echo $1 | sed 's/::ffff://'`
node_name=$( echo $NODE_NAME | cut -d'/' -f3)
node_full="$node_name:$2"

### BEGIN CHECK ####

content=$(curl-apd -s -k -1 http://$node_ip:$2/lbkeepalive.json)
alive=$( jq -r ".alive" <<< "${content}" )
ratio=$( jq -r ".ratio" <<< "${content}" )
#echo "alive:$alive"
#echo "ratio:$ratio"



# The following command do the following:
# List the configuration of the pool
# For the the server concerned, search the beginning of its config file with its name and the end with the space preceding the closure accolade
# We grep the line with the ratio config (space for exact match to differ from dynamic-ratio)
# We use awk to print the second term which is the ratio value
curratio=$(tmsh list ltm pool srv members  | sed -n '/$node_name/,/^        }/p' | grep ' ratio ' | awk '{ print $2 }')

state=$(tmsh list ltm pool srv members  | sed -n '/$node_name/,/^        }/p' | grep "state" | awk {'print $2}')

#echo "$node_name: current ratio is $curratio" | logger -p local0.debug
#echo "$node_name: alive: $alive" | logger -p local0.debug
#echo "$node_full: state is $state" | logger -p local0.debug

if [ "$alive" = "0" ]
then
   # Alive is present but with 0 value. Immediate deactivation of the member (if up before). Current sessions will persist to the member.
#   echo "alive is 0"
   tmsh modify ltm pool srv members modify { $node_full { session user-disabled } }

elif [ "$alive" = "1" ]
then
   # Alive is present and set. Checking ratio value and change it if different of current one
#   echo "alive is 1"

   if [ "$state" != "up" ]; then
     tmsh modify ltm pool srv members modify { $node_full { session user-enabled  } }
     tmsh modify ltm pool srv members modify { $node_full { state user-up  } }
   fi
   if [ "$ratio" != "$curratio" ]; then
     tmsh  modify ltm pool srv members modify { $node_full { ratio $ratio } }
   fi
   # Remove the pidfile before the script echoes anything to stdout and is killed by bigd      
   rm -f $pidfile
   echo "up"

elif [ "$alive" = "-1" ]
then
#   echo "alive is -1"
   # Alive is present but with -1 value. Immediate shutdown of the member with no keeping of persistent session
   tmsh modify ltm pool srv members modify { $node_full { state user-down } }

else
    # Alive not present or with wrong value. Down
#   echo "alive is not present"
   tmsh modify ltm pool srv members modify { $node_full { state user-down } }
fi

# Remove the pidfile before the script ends
rm -f $pidfile

Ressources

The following links contains useful information around external monitors:

https://community.f5.com/t5/technical-articles/ltm-external-monitors-the-basics/ta-p/277128

https://community.f5.com/t5/technical-articles/external-monitor-information-and-templates/ta-p/289008

https://community.f5.com/t5/technical-articles/ltm-external-monitors-troubleshooting/ta-p/277124

https://support.f5.com/csp/article/K71282813