In a part 3 (Part 1 and Part 2) we will observe the most complicated upgrade of CISCO CRS-1 (aka HFR (huge f**cking router)) running IOS-XR. Competing with Junos in flexibility and expanded capabilities which are absent in a standart IOS, IOS-XR requires a lot of time and moves during upgrade procedure. Same time adding some SMU-s (bugfix) to the current software may require reload of the box and lead to service interruption for about 30 minutes. ISSU functionality is not an option at all, especially for major release upgrades.
So, let's cross the fingers and start upgrade procedure:
IOS XR Upgrade Procedures on CRS-1 with Redundant RP-s
RP/0/RP0/CPU0:#sh hfr
So, let's cross the fingers and start upgrade procedure:
IOS XR Upgrade Procedures on CRS-1 with Redundant RP-s
Example will be based on CRS-1 upgrade from 4.1.2 to 4.2.4
PX image.
First let’s obtain required files from CCO:
1.
PIE file, in our case composite one with k9 : CRS-iosxr-px-k9-4.2.4.tar
2.
SMU files, in our case recommended one : 4.2.4_hfr-px_REC_SMUS_2013-06-10.tar
Next untar archives on PC and copy all the files to router’s
RP0 and RP1 (harddisk:)
Check free space :
RP/0/RP0/CPU0:#dir harddisk:
RP/0/RP0/CPU0:#dir harddisk: location 0/RP1/CPU0
Before copy operation new directory has to be created on
harddisks :
RP/0/RP0/CPU0:#admin mkdir harddisk:/iosxr424
RP/0/RP0/CPU0:#admin mkdir harddisk:/iosxr424 location 0/RP1/CPU0
Copy files via ftp to CRS:
RP/0/RP0/CPU0:#copy ftp://1.1.1.1/4.2.4.files harddisk:/iosxr424
RP/0/RP0/CPU0:#copy ftp://1.1.1.1/4.2.4.files harddisk:/iosxr424 location 0/RP1/CPU0
Remark : 4.2.4.files are all the files from TAR archives
Also we can copy between RP-s with the wildcard in filename, if files are already saved to RP0 :
RP/0/RP0/CPU0:#copy harddisk:/iosxr424/* location 0/RP0/CPU0 harddisk:/iosxr424/ location 0/RP1/CPU0
Perform necessary system check:
show platform
show install active
cfs check
admin show license
RP/0/RP0/CPU0:#sh hfr
Node Type PLIM State Config State
------------- ----------------- ------------------ --------------- ---------------
0/0/2 FP40(SPA) 10X1GE OK PWR,NSHUT,MON
0/0/4 FP40(SPA) OC192RPR-XFP OK PWR,NSHUT,MON
0/4/CPU0 FP40 2-10GbE-FLX IOS XR RUN PWR,NSHUT,MON
0/4/0 FP40(SPA) 1x10GE OK PWR,NSHUT,MON
0/4/4 FP40(SPA) OC192RPR-XFP OK PWR,NSHUT,MON
0/6/CPU0 FP40 4-10GbE IOS XR RUN PWR,NSHUT,MON
0/7/CPU0 FP40 4-10GbE IOS XR RUN PWR,NSHUT,MON
0/RP0/CPU0 RP(Active) N/A IOS XR RUN PWR,NSHUT,MON
0/RP1/CPU0 RP(Standby) N/A IOS XR RUN PWR,NSHUT,MON
and so on…
Check free space on flash cards (disk0):
RP/0/RP0/CPU0:#show media
Media Information for 0/RP0/CPU0.
Image Current Part
Mountpoint FsType FsType Size State DrvrPid Mirror Flags
================================================================================
/disk0: FAT16 FAT32 3.4G Mounted 0036891 Enabled
/disk0a: FAT16 FAT16 0.5G Mounted 0036891
/disk1: FAT16 (?) Not Present
/disk1a: FAT16 (?) Not Present
/harddisk: QNX4 QNX4 33.5G Mounted 0032791
/harddiska: QNX4 QNX4 11.2G Mounted 0032791
/harddiskb: FAT32 FAT32 11.2G Mounted 0032791
/lcdisk0: FAT32 (?) Not Present
/lcdisk0a: FAT32 (?) Not Present
RP/0/RP0/CPU0:#show media location 0/RP1/CPU0
Media Information for 0/RP1/CPU0.
Image Current Part
Mountpoint FsType FsType Size State DrvrPid Mirror Flags
================================================================================
/disk0: FAT16 FAT32 3.4G Mounted 0036891 Enabled
/disk0a: FAT16 FAT16 0.5G Mounted 0036891
/disk1: FAT16 (?) Not Present
/disk1a: FAT16 (?) Not Present
/harddisk: QNX4 QNX4 33.5G Mounted 0028696
/harddiska: QNX4 QNX4 11.2G Mounted 0028696
/harddiskb: FAT32 FAT32 11.2G Mounted 0028696
/lcdisk0: FAT32 (?) Not Present
/lcdisk0a: FAT32 (?) Not Present
RP/0/RP0/CPU0#show filesystem disk0:
Model: UNIGEN FLASH
Firmware: 30/06/03
BIOS Geometry: 16 Heads, 63 Sectors
Drive Geometry: 16 Heads, 8150 Tracks, 63 Sectors
Drive Capacity: 8215200 Cur Sctrs, 8215200 User
Sctrs, Extd
Address Mode: LBA
PIO mode: 2
Multimode Blocks/Transfer: 32
Capacity:
8215201 Sectors, Total 4206182912 Bytes, (512 Bytes/sector)
RP/0/RP0/CPU0#show filesystem disk0: location 0/RP1/CPU0
Model: UNIGEN FLASH
Firmware: 30/06/03
BIOS Geometry: 16 Heads, 63 Sectors
Drive Geometry: 16 Heads, 8150 Tracks, 63 Sectors
Drive Capacity: 8215200 Cur Sctrs, 8215200 User
Sctrs, Extd
Address Mode: LBA
PIO mode: 2
Multimode Blocks/Transfer: 32
Capacity:
8215201 Sectors, Total 4206182912 Bytes, (512 Bytes/sector)
If there’s not enough space on disk0 (at least 1.5GB), then inactive packages
should be removed:
RP/0/RP0/CPU0:#admin show install inactive
if found some inactive packages then do next:
RP/0/RP0/CPU0:# admin install remove inactive
RP/0/RP0/CPU0:# admin install commit
For the upgrade process to flow without interruptions it’s
recommended to offload the traffic and shutdown routing protocols (BGP, OSPF,
ISIS, LDP etc.) Even better is to reload whole box before upgrade:
RP/0/RP0/CPU0:#admin reload location all
Also let the FPD-s upgraded automatically during the new
image installation:
RP/0/RP0/CPU0:#admin
RP/0/RP0/CPU0(admin):#conf t
RP/0/RP0/CPU0(admin):# fpd auto-upgrade
RP/0/RP0/CPU0(admin):# commit
Let the upgrade process begin with the text scripts
prepared in notepad:
RP/0/RP0/CPU0:# admin install add harddisk:/424/hfr-mpls-px.pie-4.2.4 harddisk:/424/hfr-services-px.pie-4.2.4 harddisk:/424/hfr-fpd-px.pie-4.2.4 harddisk:/424/hfr-mcast-px.pie-4.2.4 harddisk:/424/hfr-mini-px.pie-4.2.4 harddisk:/424/hfr-k9sec-px.pie-4.2.4 harddisk:/424/hfr-diags-px.pie-4.2.4 harddisk:/424/hfr-mgbl-px.pie-4.2.4 harddisk:/424/hfr-doc-px.pie-4.2.4 sync
Info: The following packages are now available to be activated:
Info:
Info: disk0:hfr-mpls-px-4.2.4
Info: disk0:hfr-services-px-4.2.4
Info: disk0:hfr-fpd-px-4.2.4
Info: disk0:hfr-mcast-px-4.2.4
Info: disk0:hfr-mini-px-4.2.4
Info: disk0:hfr-k9sec-px-4.2.4
Info: disk0:hfr-diags-px-4.2.4
Info: disk0:hfr-mgbl-px-4.2.4
Info: disk0:hfr-doc-px-4.2.4
Info:
Info: The packages can be activated across the entire router.
Let’s add the recommended SMU-s also:
RP/0/RP0/CPU0:# admin install add harddisk:/424/hfr-px-4.2.4.CSCue53201.pie harddisk:/424/hfr-px-4.2.4.CSCug09031.pie harddisk:/424/hfr-px-4.2.4.CSCug20386.pie harddisk:/424/hfr-px-4.2.4.CSCue55783.pie harddisk:/424/hfr-px-4.2.4.CSCue71114.pie harddisk:/424/hfr-px-4.2.4.CSCue04603.pie harddisk:/424/hfr-px-4.2.4.CSCue19011.pie harddisk:/424/hfr-px-4.2.4.CSCuc56287.pie harddisk:/424/hfr-px-4.2.4.CSCue21974.pie harddisk:/424/hfr-px-4.2.4.CSCud41972.pie sync
Info: The following packages are now available to be activated:
Info:
Info: disk0:hfr-px-4.2.4.CSCue53201-1.0.0
Info: disk0:hfr-px-4.2.4.CSCug09031-1.0.0
Info: disk0:hfr-px-4.2.4.CSCug20386-1.0.0
Info: disk0:hfr-px-4.2.4.CSCue55783-1.0.0
Info: disk0:hfr-px-4.2.4.CSCue71114-1.0.0
Info: disk0:hfr-px-4.2.4.CSCue04603-1.0.0
Info: disk0:hfr-px-4.2.4.CSCue19011-1.0.0
Info: disk0:hfr-px-4.2.4.CSCuc56287-1.0.0
Info: disk0:hfr-px-4.2.4.CSCue21974-1.0.0
Info: disk0:hfr-px-4.2.4.CSCud41972-1.0.0
Info:
Info: The packages can be activated across the entire router.
Time to activate new XR:
RP/0/RP0/CPU0:# admin install activate disk0:hfr-mpls-px-4.2.4 disk0:hfr-doc-px-4.2.4 disk0:hfr-services-px-4.2.4 disk0:hfr-fpd-px-4.2.4 disk0:hfr-mcast-px-4.2.4 disk0:hfr-mini-px-4.2.4 disk0:hfr-k9sec-px-4.2.4 disk0:hfr-diags-px-4.2.4 disk0:hfr-mgbl-px-4.2.4 sync test
“test” keyword is needed to perform virtual installation and
check if any errors can be found. After success remove “test” and perform real
activation…
Info: This operation will reload the following nodes in parallel:
Info: 0/0/SP (MSC-DRP-SP) (Admin Resource)
Info: 0/4/SP (MSC-DRP-SP) (Admin Resource)
Info: 0/6/SP (MSC-DRP-SP) (Admin Resource)
Info: 0/7/SP (MSC-DRP-SP) (Admin Resource)
Info: 0/0/CPU0 (LC) (SDR: Owner)
Info: 0/4/CPU0 (LC) (SDR: Owner)
Info: 0/6/CPU0 (LC) (SDR: Owner)
Info: 0/7/CPU0 (LC) (SDR: Owner)
Info: 0/RP0/CPU0 (HRP) (SDR: Owner)
Info: 0/SM0/SP (140G-Fabric-SP-B) (Admin Resource)
Info: 0/SM1/SP (140G-Fabric-SP-B) (Admin Resource)
Info: 0/SM2/SP (140G-Fabric-SP-B) (Admin Resource)
Info: 0/SM3/SP (140G-Fabric-SP-B) (Admin Resource)
Proceed with this install operation (y/n)? [y]
Info: Install Method: Parallel Reload
….
Info: The changes made to software configurations will not be persistent
Info: across system reloads. Use the command '(admin) install commit' to
Info: make changes persistent.
Info: Please verify that the system is consistent following the software
Info: change using the following commands:
Info: show system verify
Info: install verify packages
Install operation 53 completed successfully at 00:00:00
Router will reload itself upon completing of installation
and activate new image:
Do important check of the new system :
RP/0/RP0/CPU0:#show version
RP/0/RP0/CPU0:#show configuration failed startup
RP/0/RP0/CPU0:#admin show configuration failed startup
RP/0/RP0/CPU0:#show platform
If everything is fine apply the final commit for the new image
:
RP/0/RP0/CPU0:# admin install commit
Install operation 54 '(admin) install commit' started by user ‘root’ via CLI at
| 100% complete: The operation can no longer be aborted (ctrl-c for options)RP/0/RP0/CPU0: instdir[251]: %INSTALL-INSTMGR-4-ACTIVE_SOFTWARE_COMMITTED_INFO : The currently active software is now the same as the committed software.
Install operation 54 completed successfully at 00:00:00
Activating SMU-s are absolutely the same process as
activating PIE-s, just create new text script and paste it into the box. If the
some of the SMU-s requires reload then at the end of the activation box will be
rebooted. To finalize type “admin install commit”
one more time for the SMU-s.
Downtime during each reload is about 20-30 minutes. “Add” operation
takes about 1 hour, activation with reboot +30 minutes. Almost the same timings
are for the SMU-s add+activation.
Total
outage expected is 4+ hours!
TIP: Before the start of the upgrade
procedure physically remove (with the help of OIR) RP1 from the router. With
this operation you can do the fast rollback to the old XR version saved on RP1.
After the complete upgrade to new XR on RP0 and putting system in service just insert
the RP1 back to the box and it will perform its upgrade to the new image
automatically. First it’ll download required files via tftp from the RP0 and
then add+activate them. This process will take additional 4 hours for RP1, but
the box during that time can be in production and work without any
interruption.
PS: Check the licenses are in place after upgrade: “admin show license”.
Small comparison table of upgrade levels:
Small comparison table of upgrade levels:
Vendor
|
Model
|
Upgrade files' count
|
Total files' size
|
Outage time
|
Difficulty Level
|
Cisco
|
C7600
|
2
|
200MB
|
15-20 min
|
Easy
|
Juniper
|
MX960
|
1
|
420MB
|
2 min
|
Moderate
|
Cisco
|
CRS-1
|
10+
|
1GB+
|
4 + hours
|
Challenging
|
