The Packet Wizard : Work Travel

I am home! I have been travelling for work for the best part of the past 5 weeks. I was in Boston doing a network refresh the week before Easter, which included replacing all the network cables, installing new Palo Alto Firewalls and removing Cisco ASA’s. I also removed all Cisco Switches and installed a new stack of Ruckus 7250, replace the core switches with 2 new Arista’s. I then came home for 2 days and I left again for Singapore for 3 weeks. I was in Singapore integrating a new company we bought into our network, this was a team effort as we had other sites to bring online within 48 hours. Copenhagen and a small site in Kaohsiung, Taiwan. I have learned a lot over the past 2 month. I have some articles to write on what I have learned but for now, I just wanted to give a quick update. Here is some cable porn from the Boston Network Refresh.

Before:

After:

 

 

Palo Alto : Upgrade High Availability (HA) Pair

Over the last 3 weeks since the Christmas and New Year Holidays,  I have been upgrading all of our firewalls globally, many of them are an High Availability Pair. This means they are redundant and being redundant allows me to upgrade them individually while the site stays full up and functional.

The  instructions for upgrading an HA pair are recommended because:

      • It verifies HA functionality before starting the upgrade.
      • It ensures the upgrade is successfully applied to the first device before starting the upgrade on the second.
      • At any point in the procedure, if any issue arises, the upgrade can be seamlessly reverted without any expected downtime (unless you are having any dynamic routing protocols line OSPF/BGP).
      • When finished, the final active/passive device state will be the same as it was before the upgrade with the fewest number of fail overs possible (2).

Before you Begin :

Take backup of the configuration as well as Tech Support from both HA Peers. Give proper names to each file, here is how:

Device > Setup > Operations > Save Named Configuration Snapshot

Device > Setup > Operations > Export Named configuration Snapshot

Device > Setup > Operations > Export Device State (If device managed from panorama

Device > Support > Generate Tech Support File, and then download it. (Might be required if any issues)

(Optional but recommended) Disable preemption on High Availability settings to avoid the possibility of unwanted failovers. Disabling preempt configuration change must be committed on both peers. Likewise, once completed, re-enabling must be committed on both peers.

To disable preempt, go to

Device > High Availability > Election Settings and uncheck Preemptive.
Then, perform a commit.

 

 

 

If upgrade is between major versions (4.1 -> 5.0 OR 5.0-> 6.0), it is advisable to disable TCP-Reject-Non-SYN, so that sessions can failover even when they are not in sync. : Do this on both Firewalls from the CLI:

 # set deviceconfig setting session tcp-reject-non-syn no
 # commit

 

(Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is again to help recover from any unexpected situation where we are unable to login to the firewall. If you have a Terminal Server awesome, if not a simple Cell Phone tethered to a Laptop with RDP is also fine.

 

The Upgrade Process

Suspend Backup Device 

From the CLI

 > request high-availability state suspend

From the GUI

Go to Device > High Availability > Operational Commands  > Suspend local device

Install the new PAN-OS on the suspended device

Device > Software > Install

Reboot the device to complete the install.

When the upgraded device is rebooted, check the dashboard to check the version, wait for all the interfaces to come backup green.

If the device is still in suspended state make it functional again

From the CLI

> request high-availability state functional

From the GUI

Go to Device > High Availability > Operational Commands  > Make Local Device Functional

 

Repeat steps on other firewall.

 

Suspend Primary Device

From CLI

> request high-availability state suspend

From the GUI

Go to Device > High Availability > Operational Commands  > Suspend local device.

 

*The Backup Firewall will become Active – it does take 30-45 seconds so don’t panic

 

Install the new PAN-OS on the suspended device:

Device > Software > Install

Reboot the device to complete the install.

When the upgraded device is rebooted, check the dashboard to check the version, wait for all the interfaces to come backup green.

If the device is still in suspended state make it functional again

From the CLI

> request high-availability state functional

From the GUI

Go to Device > High Availability > Operational Commands  > Make Local Device Functional

To Get Primary Back to Primary by suspending the backup (current active) firewall (The Original Backup Firewall)

From the GUI,

Go to Device > High Availability > Operational Commands  > Suspend local device.

Once the Primary became active again, enable the suspended backup firewall

Enable TCP-Reject-Non-SYN, so that sessions can failover even when they are not in sync. : (Do this on both Firewalls)

# set deviceconfig setting session tcp-reject-non-syn yes
# commit

 

Re-Enable preempt configuration change must be committed on both peers. To re-enable preempt, go to Device > High Availability > Election Settings and uncheck Preemptive.  Then, perform a commit.

 

How to Downgrade

If an issue occurs on the new version and a downgrade is necessary:

To revert to the previous PAN-OS screen, run the following CLI command:

# debug swm revert

This causes the firewall to boot from the partition in use prior to the upgrade. Nothing will be uninstalled and no configuration change will be made.

 

However please be aware while running this command –

After rebooting from a SWM revert, the configuration active at the time before upgrade will be loaded with the activation of the previous partion. Any configuration changes made after upgrade will not be accounted for and will need to be manually recovered by loading the latest configuration version and committing the changes.

Cisco : IOS Upgrade to Denali

CATALYST IOS UPGRADE TO DENALI

You can copy the files to the router from your local pc with the Fenix web server.

Right click on the file in the Fenix web browser > copy link > use the copy command on the device.

Copy OS file to flash:

Copy usbflash0:<filename> flash:<filename>

                  verify /md5 bootflash:<image_file>

On XE 3.x

software install file flash:<filename> new force

On Denali 16.x

request platform software package install switch all file flash: <filename> auto-copy

Troubleshooting

Clean OS’s in Denali

Request platform software package clean switch all file flash:

If Versions are Mismatches in XE

% Switch # is running incompatible software.

Compatible software must be installed on this switch before performing the current operation.

Switch#software auto-upgrade

If Versions are Mismatches in Denali

device(config)#software auto-upgrade enable

UPGRADE 3.6+ IF MASTER SWITCH IS ON DENALI 16.3

request platform software package install autoupgrade

Cisco : IOS Catalyst Upgrade

CATALYST IOS UPGRADE

From the switch, do the following

Get an MD5 hash to verify the image is intact

                verify /md5 bootflash:<image_file>

Set the config to boot from the new image

               boot system flash bootflash:<image_file>

Save the config and reload

              write mem
              reload

The switch comes back up with the new image

             sh version

Save the config

             write mem