####Objective:

**EDIT(29/10/2017): Download BIGIP-API.py and mainip.txt
Old draft: Python script MAIN.PY

Multiple F5 BIG-IP systems are to be upgraded without downtime. To achieve this, one system is put on forced offline state and upgrade procedure is followed following with a reboot of machine, to ultimately boot into upgraded version. The system is then brought to online state. The offline state makes sure no incoming traffic is allowed, in a BIG-IP Pool, other BIG-IP systems which are online will take the incoming traffic. In BIG-IP 11.1.0 and later, the Force Offline status persists through system reboots and upgrades. So, it is needed to be put online again after upgradation/reboot.

With one system upgraded, next system is brought offline, then upgraded and booted into, and finally put online. The recursion follows until last system is brought online with latest version running.

To effectively follow upgrade procedure. We need to make sure the current installed version is compatible to be upgraded, so not to lose existing configuration of the system. For eg, a 10.X BIG-IP cannot be upgraded directly to 13, however, 11 can be directly upgraded to 13. Re-activation of license is not required after upgradation, however a check should be followed that a license exists, else, losing of configuration files is imminent on upgradation.

Version upgrading toVersions we can directly upgrade from
13.X13.x
12.x
11.x
12.X12.x
11.x
10.1.x - 10.2.x
11.X 11.x
10.x
10.X10.x
9.6.x
9.4.x
9.3.x
When we perform the software installation to upgrade the BIG-IP system, the installation process does not validate the service check date with the license check date of the version being installed. If the service check date is missing or is earlier than the license check date, the BIG-IP system will fail to load the configuration when we attempt to boot into the upgraded software slot.
ProductVersionLicense Check Date (yyyy/mm/dd)
BIG-IP13.0.02017-01-13
BIG-IP12.1.0 - 12.1.22016-03-18
BIG-IP12.0.02015-08-03

Check Current Version

Check what version the BIG-IP system is running currently.

    tmsh show /sys license | grep "Licensed Version" | tail -c 7

Following conditions can be used to perform upgrades for specific version to a specific version. Jumping directly from 10.X to 13.X would lose the configuration. When recursively done, 10.X will be upgraded to 11.X, which the program checks again, will upgrade the 11.X to 13.X(latest).

    remote=$(tmsh show /sys license | grep "Licensed Version" | tail -c 7)

    if [ $remote " "12.1.1" ]; 
        then <COMMAND TO UPGRADE FROM 12.1.1 to 13>; 
    elif [ $remote " "11.1.0" ];
        <COMMAND TO UPGRADE TO 12>; 
    fi

service date

Command:

    grep "Service check date" /config/bigip.license

shall output the service date. For eg., to upgrade to 13.X, the service date should be “2017-01-13” or later, in our case, the service date as mentioned in picture is “2017-10-04”, since it is a later date than the service date, we can continue with the upgradation from 12.X to 13.X.

serial no.BIG-IP System IPusernameuser passwordadmin user (API privileges)admin pass
mainip.txt
    1,192.168.1.110,root,default,admin,admin
    2,192.168.1.111,root,default,admin,admin
BIGIP-API.PY
#!/bin/python2
import time
import subprocess
import os
import commands
from datetime import datetime

#####
##  Requirements: Python2, sshpass, pssh
##  Todo: Implement pssh or clusterit for parallel download of BIGIP ISO file in BIGIP systems. 
##        Log everything to a file
#####

def Main(ip,user,passw,admin_user,admin_pass):
    global BIGIP_IP, USER_NAME, USER_PASS, A_USER, A_PASS, ADMIN_USER_PASS, SSH_MAIN, BIGIPISO, VOLUME_LOCATION
    BIGIP_IP = ip
    USER_NAME = user
    USER_PASS = passw

    A_USER = admin_user                     # For REST API, admin credentials are required, unless a normal user is granted admin privileges.
    A_PASS = admin_pass
    ADMIN_USER_PASS = A_USER + ':' + A_PASS

    BIGIPISO = "BIGIP-13.0.0.0.0.1645.iso"  # Setting V13, change to another version or hotfix
    VOLUME_LOCATION = "HD1.3"               # Set to HD1.3, change HD1.3 to a different install location as desired

    SSHPassSyntax = 'sshpass -p '+ USER_PASS +' ssh -o StrictHostKeyChecking=no '
    USER_NAME_PASS = USER_NAME + '@' + BIGIP_IP

    ServiceCheckDate = ' \'grep "Service check date" /config/bigip.license\''
    CheckVersion = ' \'tmsh show /sys license | grep "Licensed Version" | tail -c 7\''
    
    SSH_MAIN = SSHPassSyntax + USER_NAME_PASS

    DATE = commands.getoutput(SSH_MAIN + ServiceCheckDate).strip().split(' ')[-1]   # Eg. output 'Service Check Date 20171004' 
                                                                                #Stripping only the required date: '20171004'

    # Converting '20171004' to Date object '2017/10/04' to compare dates easily
    DateObject = str(DATE)[:4] + '/' + str(DATE)[4:6] + '/' + str(DATE)[6:]         
    CompareableDate = time.strptime(DateObject, "%Y/%m/%d")


    SystemVersion = commands.getoutput(SSH_MAIN + CheckVersion)
    if SystemVersion != '13.0.0':
        print '*'*42
        print '\n(Device: ' + str(serialNum) + ') BIG-IP (' + BIGIP_IP + ') current version: ' + SystemVersion
        print '\nChecking Service Check Date of BIG-IP...\n'

        if CompareableDate >= time.strptime("2017/01/03", "%Y/%m/%d") :
            print 'BIG-IP (' + BIGIP_IP + ')' +' Service Check Date: ' + DateObject + ' >= ' + '2017/01/03 (13.0.0)'
            print 'Upgrade can proceed without losing existing configuration.\n'
            print '*'*17
            print 'Backup initiated: '
            print '*'*17

            print "Checking whether device " + BIGIP_IP + " is alive...\n"
            
            # TODO: Check before whether deivce is alive, if yes, then proceed with uograde procedue, else the script returns error and exit.
            #       If device is not alive, traverse to next IP, check if device alive, then proceed accordingly.
            if checkDeviceAlive() " 1:
                # Device Alive
                print ""
            elif checkDeviceAlive() " 0:
                # Device not alive
                print ""
            
            #generateConfigBackup()
            installISO()

        else:
            print 'BIG-IP(' + BIGIP_IP + ')' +' Service Check Date: ' + DateObject + ' < ' + '2017/01/03 (13.0.0)'
            print 'Proceeding with upgrade will not recover current configuration.'
            print 'Enforcing upgrade will do a fresh install of BIGIP\n\n'
    else:
        print 'default' 


def generateConfigBackup():
    FQDN = commands.getoutput(SSH_MAIN + ' dnsdomainname -f ')
    print  'System FQDN: ' + FQDN
    print '\nInvoked UCS generation.\nTakes few minutes...'
    
    commands.getoutput('mkdir -p Backup/' + FQDN)
    backup_SSH_MAIN = ' tmsh save /sys ucs ' + FQDN + '.ucs'
    
    BACKUP = commands.getstatusoutput(SSH_MAIN + backup_SSH_MAIN)
    QKVIEW = commands.getstatusoutput(SSH_MAIN + ' qkview')
    if BACKUP[0] " 0:
        print (BACKUP[1])
        print '\nCopying UCS file locally...'
        scp = 'sshpass -p ' + USER_PASS + ' scp -pr ' + USER_NAME + '@' + BIGIP_IP + ':/var/local/ucs/' + FQDN + '.ucs ' + ' ./Backup/' + FQDN + '/' 

        doSCP = commands.getstatusoutput(scp)
        if doSCP[0] " 0:
            print (doSCP[1])
            print 'Backup saved locally: \'Backup/' + FQDN + '/' + FQDN + '.ucs\'\n'
        else:
            print (doSCP[1])
            print 'Error copying Backup file from remote machine \'' + BIGIP_IP + '\':/var/local/ucs/' + FQDN + '.ucs locally in ./Backup/' + FQDN + '/' 
    else:
        print (BACKUP[1])

    if QKVIEW[0] " 0:
        print (QKVIEW[1])
        scp = 'sshpass -p ' + USER_PASS + ' scp -pr ' + USER_NAME + '@' + BIGIP_IP + ':/var/tmp/' + FQDN + '.qkview ' + ' ./Backup/'+ FQDN + '/' 

        doSCP = commands.getstatusoutput(scp)
        if doSCP[0] " 0:
            print (doSCP[1])
            print 'Backup saved locally: \'Backup/' + FQDN + '/' + FQDN + '.qkview\''
            pass
        else:
            print (doSCP[1])
            print 'Error copying Backup file from remote machine \'' + BIGIP_IP + '\':/var/tmp/' + FQDN + '/' + FQDN + '.qkview locally in ./Backup/' + FQDN + '/' 
    else:
        print (QKVIEW[1])

def installISO():
    print '*'*60
    print 'Installing \'' + BIGIPISO + '\' to new volume \'' + VOLUME_LOCATION + '\''
    print '*'*60

    INSTALL_JSON_DATA = '{"command":"install","options":[{"create-volume":true},{"reboot":true}],"name":\"' +  BIGIPISO  + '\","volume":"'+ VOLUME_LOCATION +'"}'
    ### Following JSON data passed through REST API using cURL` command, invokes ISO installation, switboots to the newly installed ISO/upgraded version, and reboots automaticaly to it. 
    ### It sums up three commands: "tmsh install sys software image BIGIP-13.0.0.0.0.1645.iso create-volume volume HD1.3" +  "switchboot -b HD1.3" + "reboot"

    CURL_CMD = 'curl -sk -u ' + ADMIN_USER_PASS + ' -H "Content-Type: application/json" -X POST -d \'' + INSTALL_JSON_DATA + '\' https://'+ BIGIP_IP + '/mgmt/tm/sys/software/image'
    commands.getoutput(CURL_CMD)

    print "Progress of installation will be displayed in 60s\r"
    time.sleep(60)
    checkIfInstalled = " tmsh show sys software | grep \'" + VOLUME_LOCATION + "\'"

    while True:
        if filter(None, commands.getoutput(SSH_MAIN + checkIfInstalled).strip().split(' '))[-1] != 'complete':
            time.sleep(60)
            print BIGIPISO + " installation progress:" + filter(None, commands.getoutput(SSH_MAIN + checkIfInstalled).strip().split(' '))[-2] + "% Completed " + " \r",
            break
# End of function, traversing to next IP

def fetch_upgrade_iso():
# Needed to be configured manually here
# SFTP can be used alternativley to SCP
# Fetch MD5 and ISO to verify integrity, before the ISO is used to upgrade.
# pssh/clusterit/fabric(python) can be used to parallely fetch the ISO in multiple systems at a time (eg. 10 BIG-IP systems downloading ISO simultaneously)
    scp = 'scp ' + scp_USER_NAME + '@' + server_address + ':/some/remote/directory/\{' + BIGIPISO + '.md5, ' + BIGIPISO + '.ISO\} /shared/images/'  

def reboot():
    reboot = SSH_MAIN + " reboot"

def checkDeviceAlive():

    result = subprocess.Popen(["ping", "-c", "1", "-n", "-W", "2", BIGIP_IP]).wait()

    if result:                              # Check Ping fails, proceed accordingly or traverse to next IP
        for counter in range(0,29):         # 1st ping fails due to ARP resolution and rest of the tries for caution, just in case
            checkDeviceAlive()
            counter += 1
            time.sleep(3)                   # 30 tries x 3 seconds = 135s = 2:25min = enough tries to check for BIG-IP alive state, as it takes time to boots up after upgradation.
                                    # If device is not alive, break the loop and go to traverseDeviceIP, to check another device.
        #serialNum += 1
        return 0
    else:
        return 1
        #print "\n\t Device: " + BIGIP_IP + " active"



def traverseDeviceIP():
    # To keep count of devices under upgrade procedure
    result = subprocess.Popen(["ping", "-c", "1", "-n", "-W", "2", BIGIP_IP]).wait() # For checking whether device is alive, script fails when commands are sent to a dead device.
    
    global serialNum
    serialNum = 1
    # Read CSV txt file having a list of IP address, USER_NAME, password.
    # Save them in a variable, pass them to Function 'Main', which follows
    with open('mainip.txt','r') as f:
        for lines in f:
            BIGIP_IP = lines.strip().split(',') [1]
            USER_NAME = lines.strip().split(',') [2]
            USER_PASS = lines.strip().split(',') [3]
            ADMIN_USER = lines.strip().split(',') [4]
            ADMIN_PASS = lines.strip().split(',') [5]

            ServiceCheckDate = Main(BIGIP_IP,USER_NAME,USER_PASS,ADMIN_USER,ADMIN_PASS)

            #checkDeviceAlive()
            
            serialNum += 1

    print "\nEnd of program\n"
traverseDeviceIP()

asciicast