Upgrading ISE does not have to be difficult or complex in any way. There is a very well-defined process that should be followed to the letter and the upgrades these days are relatively safe. This article will discuss the process that should be followed.
Step 1 – General Health Check
Strangely, this first step is the one that is overlooked the most. Just like any other software application ever built, it’s always a good idea to ensure everything is working and the alerts are completely understood before an upgrade is attempted. Of course, at times, the upgrade may be the fix to a problem, but when that is not the case you should begin the upgrade after all alerts and errors have been dealt with.
There are a lot of little caveats this first step eliminates. For example, an expired certificate is a deal breaker which kind of makes sense. What does not really make sense is it is a deal breaker even when the expired certificate is not in use.
In addition, during this step you should verify your hardware or VMs are compatible with the new version of ISE. This is especially true if your appliances are EOL or the same VMs have been running for a while. This is also a good time to ensure ample disk space is available.
Of course, you can find the resource information and any known caveats in the ISE Release Notes. For example, here are the release notes for v2.7.
Within the release notes, you can also confirm you are attempting a supported upgrade path.
Finally, this is also a good time to confirm all nodes in the deployment are fully synced and enough available bandwidth exists for the upgrade.
Step 2 – Backups and Information
Probably goes without saying, but it is always a good idea to run an ad-hoc backup of ISE immediately prior to running an upgrade or patch. More than once I’ve been called to assist with a failed upgrade only to find out the customer forgot the password to their backup files. Ug! If doom strikes and a full restore is required, you are sunk without the backup password.
Also, it’s important to note the automated backup does not capture the certificates. All certs in use must be backed up separately and manually. Do not skip this step! I have seen certificates disappear even after a successful upgrade.
Finally, make sure you know the following information and it is 100% ready to go:
GUI/CLI Local Admin Password: One of the most common problems with upgrades is the AD integration goes out-of-sync and has to be rejoined. If you are used to logging in with your AD credentials, you have a sad story if the admin account is locked out or you simply forgot the password. Test this! In addition, the CLI is seldom used, and the password is not necessarily the same as the GUI Password. It’s not uncommon for a customer to not login to the CLI for 6 months! It’s easy to forget. Plus, nearly all failed upgrades have to be corrected on the CLI. This is an important step.
Active Directory Account: Ensure you have an AD account with permissions to create machine objects ready to go. On occasion, especially with older versions of ISE, you have to rejoin AD after the upgrade. By having the account ready to go, you won’t have to scramble to find someone with the necessary AD privileges.
Vmware Admin: If you are running VMs, make sure you have someone available in case console access is necessary. It seldom is, so an “on-call” engineer works well.
CIMC Information: Ensure you know the IPs of each CIMC if you have appliances and know the credentials.
Patching
While not a strict requirement, it is best practice to patch every node in the deployment to the latest version prior to an upgrade. I realize it is slow and painful, but so is a broken ISE deployment😊.
Stage the Media
Before you can run the upgrade, the upgrade software has to be within an ISE repository. Upload the media to the server and confirm ISE can see it.
Pull the Trigger
There’s not much to this piece. Navigate to Administration > System > Upgrade
Choose the media from the repository and have at it!
Fire In The Barn! When a good plan goes bad!
You did everything right, the process starts with no problems, and then 45 minutes later you see the first node failed. Bummer.
With modern versions of ISE, this process is a lot better than it used to be. In the old days, the process would just kind of die, you couldn’t cancel, and you couldn’t move forward either. Once this nightmare was realized, TAC would, without skipping a beat, say “Do a fresh install and then a restore”. SCREAM!!!!! The fresh install process is anything but fast, and then once that’s done, you also have to do a full restore, then restore the certificates, then rejoin AD, and then take all your switchports out of Critical Auth. Nightmare. Of course, then you have to do the entire upgrade again!
The good news is I have not seen that in quite a while. However, you should still understand the manual process which resolves nearly all problems.
The order of operations comes into play during a manual CLI based upgrade. In a distributed deployment, ISE effectively breaks the deployment up into the “old” deployment and the “new” deployment. Let’s assume you have 2xAdmin, 2xM&T, and 2xPSN.
Step 1: Upgrade the secondary Admin. At this point, the secondary Admin drops out of the existing “old” deployment and creates a brand “new” deployment. The “old” secondary admin becomes the primary admin on the “new” deployment. Once done and you can get to the GUI of the upgraded node, you can move onto step 2.
Note: The Primary Admin of your “old” deployment is still up and running, thus, so are your users!
Step 2: Upgrade the secondary M&T. Keep in mind, a Cisco ISE deployment requires 1xAdmin, 1xM&T, and 1xPSN to function. The secondary M&T will become the primary M&T of the new deployment. The old deployment is still unaffected, and the new deployment has 2 out of 3 required personas.
Step 3: Upgrade one PSN. The PSN will join the new deployment. At this point, you should have two fully functioning deployments. Thus, you can fully test the upgraded deployment before breaking the old one.
Step 4: Once you are satisfied with the new deployment, you can upgrade all remaining PSNs. You are not concerned about down time as the new deployment is already tested.
Step 5: Upgrade the “old” Primary M&T.
Step 6: Upgrade the “old” Primary Admin. Keep in mind, the only catastrophic event is when you lose both Admin nodes at the same time. Up until this point, it is relatively trivial to build a new node of any flavor, join it to your good Admin Node, and off you go. After this step, it is extremely difficult to go backwards so make sure your “new” deployment is running perfectly.
Finally, keep in mind ISE is modular. If the new deployment seems a bit shaky or you are not 100% certain, simply, don’t upgrade the old admin node yet. Add the secondary admin persona to one of the upgraded PSNs to maintain redundancy, and then upgrade the old admin node when you are completely comfortable.
The final step in the upgrade is to patch and put the personas back the way you want them.
If you would like assistance with your Cisco ISE upgrade, contact us today!