Palo Alto : Upgrade High Availability (HA) Pair

Over the last 3 weeks since the Christmas and New Year Holidays,  I have been upgrading all of our firewalls globally, many of them are an High Availability Pair. This means they are redundant and being redundant allows me to upgrade them individually while the site stays full up and functional.

The  instructions for upgrading an HA pair are recommended because:

      • It verifies HA functionality before starting the upgrade.
      • It ensures the upgrade is successfully applied to the first device before starting the upgrade on the second.
      • At any point in the procedure, if any issue arises, the upgrade can be seamlessly reverted without any expected downtime (unless you are having any dynamic routing protocols line OSPF/BGP).
      • When finished, the final active/passive device state will be the same as it was before the upgrade with the fewest number of fail overs possible (2).

Before you Begin :

Take backup of the configuration as well as Tech Support from both HA Peers. Give proper names to each file, here is how:

Device > Setup > Operations > Save Named Configuration Snapshot

Device > Setup > Operations > Export Named configuration Snapshot

Device > Setup > Operations > Export Device State (If device managed from panorama

Device > Support > Generate Tech Support File, and then download it. (Might be required if any issues)

(Optional but recommended) Disable preemption on High Availability settings to avoid the possibility of unwanted failovers. Disabling preempt configuration change must be committed on both peers. Likewise, once completed, re-enabling must be committed on both peers.

To disable preempt, go to

Device > High Availability > Election Settings and uncheck Preemptive.
Then, perform a commit.

 

 

 

If upgrade is between major versions (4.1 -> 5.0 OR 5.0-> 6.0), it is advisable to disable TCP-Reject-Non-SYN, so that sessions can failover even when they are not in sync. : Do this on both Firewalls from the CLI:

 # set deviceconfig setting session tcp-reject-non-syn no
 # commit

 

(Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is again to help recover from any unexpected situation where we are unable to login to the firewall. If you have a Terminal Server awesome, if not a simple Cell Phone tethered to a Laptop with RDP is also fine.

 

The Upgrade Process

Suspend Backup Device 

From the CLI

 > request high-availability state suspend

From the GUI

Go to Device > High Availability > Operational Commands  > Suspend local device

Install the new PAN-OS on the suspended device

Device > Software > Install

Reboot the device to complete the install.

When the upgraded device is rebooted, check the dashboard to check the version, wait for all the interfaces to come backup green.

If the device is still in suspended state make it functional again

From the CLI

> request high-availability state functional

From the GUI

Go to Device > High Availability > Operational Commands  > Make Local Device Functional

 

Repeat steps on other firewall.

 

Suspend Primary Device

From CLI

> request high-availability state suspend

From the GUI

Go to Device > High Availability > Operational Commands  > Suspend local device.

 

*The Backup Firewall will become Active – it does take 30-45 seconds so don’t panic

 

Install the new PAN-OS on the suspended device:

Device > Software > Install

Reboot the device to complete the install.

When the upgraded device is rebooted, check the dashboard to check the version, wait for all the interfaces to come backup green.

If the device is still in suspended state make it functional again

From the CLI

> request high-availability state functional

From the GUI

Go to Device > High Availability > Operational Commands  > Make Local Device Functional

To Get Primary Back to Primary by suspending the backup (current active) firewall (The Original Backup Firewall)

From the GUI,

Go to Device > High Availability > Operational Commands  > Suspend local device.

Once the Primary became active again, enable the suspended backup firewall

Enable TCP-Reject-Non-SYN, so that sessions can failover even when they are not in sync. : (Do this on both Firewalls)

# set deviceconfig setting session tcp-reject-non-syn yes
# commit

 

Re-Enable preempt configuration change must be committed on both peers. To re-enable preempt, go to Device > High Availability > Election Settings and uncheck Preemptive.  Then, perform a commit.

 

How to Downgrade

If an issue occurs on the new version and a downgrade is necessary:

To revert to the previous PAN-OS screen, run the following CLI command:

# debug swm revert

This causes the firewall to boot from the partition in use prior to the upgrade. Nothing will be uninstalled and no configuration change will be made.

 

However please be aware while running this command –

After rebooting from a SWM revert, the configuration active at the time before upgrade will be loaded with the activation of the previous partion. Any configuration changes made after upgrade will not be accounted for and will need to be manually recovered by loading the latest configuration version and committing the changes.