HOWTO: Flexible Netflow with IOS-XE

 Time to time network engineers are facing the issue, when simple SNMP monitoring of interface load is not enough for the raised needs. Especially when network is experiencing DDoS attacks, or some kind of abnormal traffic flows are detected. So how to check what's going around. The answer is simple and almost as old as an IP protocol itself, visualization and further analysis with just implementation of Netflow functionality on border/core routers allowing to feed flow collector software.

With the introduction of flexible Netflow V9 which gives a lot of configuration options, engineers are given possibilities to have receive exact network traffic patterns view

 Even the entry level routers from Cisco Systems running IOS-XE are capable for Netflow implementation.

To configure flexible Netflow on IOS-XE there's need for 3 main and one optional components:

1. "flow record", 2. "flow exporter", 3. "flow monitor" and optional "sampler".

"flow monitor" ties "flow record" and "flow exporter" together and then configuration can be attached to the interesting interface/sub-interface for input or output direction to collect transit traffic stats. Optional (but highly recommended) "sampler" gives possibility to randomly choose one packet from "x" count from the flow to avoid router and collector overload. e.g. 1 packet out of 1000 etc.

Additionally IPv6 support can be added to configuration if network runs dual stack of IPv4/IPv6.

Here we go with the most simple and complete configuration chunk, where flow destination IP address and port should be changed to the appropriate ones in individual network, also source interface for export must be chosen (usually Loopback0 if existing). Next flow monitor can be attached to any of the interesting interfaces.

!
flow record FLOW_RECORD_V4
 match ipv4 tos
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 collect counter bytes long
 collect counter packets long
 collect interface input

!
flow exporter FLOW_EXPORTER
 destination 192.168.100.10
 source loopback0
 transport udp 5555
 export-protocol netflow-v9
 option interface-table
 option sampler-table

!
flow monitor FLOW-MONITOR
 record FLOW_RECORD_V4
 exporter FLOW_EXPORTER
 cache timeout active 60

!
sampler FLOW_SAMPLER
 mode random 1 out-of 1000

!
interface Gi0/0/0
 ip flow monitor FLOW-MONITOR sampler FLOW_SAMPLER input 

 

To check if everything works as expected:

 

router-xe#show flow monitor FLOW-MONITOR statistics
  Cache type:                               Normal (Platform cache)
  Cache size:                               200000
  Current entries:                              96
  High Watermark:                              778

  Flows added:                             2568995
  Flows aged:                              2568899
    - Active timeout      (    60 secs)      32661
    - Inactive timeout    (    15 secs)    2536238 

 

router-xe#show flow monitor FLOW-MONITOR cache
  Cache type:                               Normal (Platform cache)
  Cache size:                               200000
  Current entries:                             100
  High Watermark:                              778

  Flows added:                             2569229
  Flows aged:                              2569129
    - Active timeout      (    60 secs)      32664
    - Inactive timeout    (    15 secs)    2536465

IPV4 SRC ADDR    IPV4 DST ADDR    TRNS SRC PORT  TRNS DST PORT  IP TOS  IP PROT  intf input                      bytes long             pkts long
===============  ===============  =============  =============  ======  =======  ====================  ====================  ====================
192.168.1.2  10.9.7.2             16610           8548  0x00         17  Gi0/0/0                                400                     2
192.168.1.3  10.9.7.3             18260          23830  0xB8         17  Gi0/0/0                           400                     2
192.168.1.4  10.9.7.4             41914           5060  0x00         17  Gi0/0/0                           665                     1

 

 

 "option interface-table" feeds collector with interface names together with SNMP ifindex for convenient view.
 "option sampler-table" gives collector hint of sampling ratio for correct multiply factor of bandwidth calculation and visualization.

ElastiFlow can be recommended as an open source software for a flow collector.

NetFlow Collector Installation guide from my good friend:

https://github.com/lsopromadze/netflowcollector-elk

 

Segment Routing and LDP islands interconnection between IOS XE and JUNOS

Intro

Given the new trends of transport protocol named Segment Routing I decide to make some labbing involving IOS XE and JUNOS devices running standard MPLS framework with LDP underlay and then to try smoothly move from LDP to SR.

Segment Routing with IPv6 extended headers currently is not feasible and easily achievable, so only choice left is MPLS dataplane, which also helps to interwork with LDP control and dataplane.

With the simple topology consisting of three IOS XE and three JUNOS routers the target is to have L3VPN connectivity from R1 Loopback1 11.11.11.11/32 to R6 lo0.1 66.66.66.66/32 in vrf CE1.

/31-s subnets for inter-links are derived from 10.0.0.0/24. ISIS configured as Level2 only. Loopbacks are added as passive interfaces. In initial configuration LDP is deployed accross the network. BGP is in AS1 with R2 and R4 as route-reflectors and other routers as clients serving VPNv4 AFI/SAFI for L3VPN connectivity.

General topology:

In the next section LDP will be completely disabled on R1 leaving only Segment Routing MPLS. R2 and R3 will interconnect SR and LDP islands, so running both protocols and serving as mapping servers to pass R4/5/6 loopbacks to SR only network (R1).

SR/LDP topology:

For LDP island to reach SR island SRMS (Segment Routing Mapping Server) comes into play. Manual mappings on one or more routers for redundancy (even not on the data path) gives ISIS ability to advertise prefixes with the related labels for the MPLS dataplane to do the right action of label operation (pop/push/swap). For example SRMS advertises R4 loopback with the SID label 16004 in SR domain. Checking R1 MPLS forwarding table shows prefix with the correct label to be impositioned.

R1#sh mpls forwarding-table 4.4.4.4   
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
16004      16004      4.4.4.4/32       0             Gi1.12     10.0.0.1    


CEF also programmed with the label push:


R1#sh ip cef 4.4.4.4/32 det
4.4.4.4/32, epoch 2
  sr local label info: global/16004 [0x1B]
  nexthop 10.0.0.1 GigabitEthernet1.12 label 16004() 

LDP only network

Standard LDP label operations for packet from R1 to R6 via R2&R4 and for packet from R6 to R1 via R5&R3.

Let's check R1 FIB. For the R6 Lo0 outgoing label to be pushed is 24:

R1#sh mpls forwarding-table 6.6.6.6
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop  
Label      Label      or Tunnel Id     Switched      interface            
22         24         6.6.6.6/32       0             Gi1.12     10.0.0.1 


R2 will swap label 24 to label 299856:

R2#sh mpls forwarding-table 6.6.6.6
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop  
Label      Label      or Tunnel Id     Switched      interface            
24         299856     6.6.6.6/32       12588         Gi1.24     10.0.0.5   



Finally R4 will pop label as PHP router and forward pure data to R6:


root@R4# run show route table mpls.0 label 299856   

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299856             *[LDP/9] 21:29:18, metric 1
                    > to 10.0.0.9 via ge-0/0/0.46, Pop   





For the connectivity between emulated CE-s let's check VPN label allocation on R1 and R6.
From R1 to R6 two labels should be pushed, bottom VPN label 299840 and top transport label 20 or 24.
From R6 to R1 bottom VPN label is 25 and top transport label is 299792 or 299824.




R1#sh ip ro vrf CE1 66.66.66.66

Routing Table: CE1
Routing entry for 66.66.66.66/32
  Known via "bgp 1", distance 200, metric 0, type internal
  Last update from 6.6.6.6 1d17h ago
  Routing Descriptor Blocks:
  * 6.6.6.6 (default), from 2.2.2.2, 1d17h ago
      Route metric is 0, traffic share count is 1
      AS Hops 0
      MPLS label: 299840


R1#sh ip cef vrf CE1 66.66.66.66
66.66.66.66/32
  nexthop 10.0.0.1 GigabitEthernet1.12 label 24(elc,) 299840()
  nexthop 10.0.0.3 GigabitEthernet1.13 label 20(elc,) 299840()




root@R6# run show route table CE1 11.11.11.11/32

CE1.inet.0: 2 destinations, 3 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

11.11.11.11/32     *[BGP/170] 1d 17:24:55, MED 0, localpref 100, from 2.2.2.2
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.8 via ge-0/0/0.46, Push 25, Push 299792(top)
                      to 10.0.0.10 via ge-0/0/0.56, Push 25, Push 299824(top)

LDP and SR hybrid network

 

Now it's time for the Segment Routing configuration on IOS XE.

First of all NODE SID-s must be assigned to routers:

R1#sh run | s segment
segment-routing mpls
 !
 connected-prefix-sid-map
  address-family ipv4
   1.1.1.1/32 absolute 16001 range 1
  exit-address-family

R2#sh run | s segment
segment-routing mpls
 !
 connected-prefix-sid-map
  address-family ipv4
   2.2.2.2/32 absolute 16002 range 1
  exit-address-family


R3#sh run | s segment
segment-routing mpls
 !
 connected-prefix-sid-map
  address-family ipv4
   3.3.3.3/32 absolute 16003 range 1
  exit-address-family 

Activation of SR gives nice log message in console :
07:50:40.312: %SR-6-SR_STATE_LOG: Segment Routing MPLS ENABLED 




SRMS (Segment Routing Mapping Server) functionality for the LDP interworking can be added to any of the router in SR domain:

segment-routing mpls
 !
 mapping-server
  !
  prefix-sid-map
   address-family ipv4
    4.4.4.4/32 absolute 16004 range 1
    5.5.5.5/32 absolute 16005 range 1
    6.6.6.6/32 absolute 16006 range 1
   exit-address-family

!

router isis 1
segment-routing prefix-sid-map advertise-local

For ISIS to start use of SR MPLS one simple line should be added:

router isis 1
 segment-routing mpls

 

So from this point SR is active in network in parallel with LDP. Two transport protocols gives two different labels per the same FEC and both can be used without any consequences:


R1#sh mpls forwarding-table 6.6.6.6
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
22         24         6.6.6.6/32       0             Gi1.12     10.0.0.1   
           20         6.6.6.6/32       0             Gi1.13     10.0.0.3   
16006      16006      6.6.6.6/32       0             Gi1.12     10.0.0.1   
           16006      6.6.6.6/32       0             Gi1.13     10.0.0.3   



R1#traceroute mpls ipv4 6.6.6.6/32
 
Tracing MPLS Label Switched Path to 6.6.6.6/32, timeout is 2 seconds
Type escape sequence to abort.
 
  0 10.0.0.0 MRU 1500 [Labels: 24 Exp: 0]
L 1 10.0.0.1 MRU 1500 [Labels: 299856 Exp: 0] 15 ms
L 2 4.4.4.4 MRU 1522 [Labels: implicit-null Exp: 0] 59 ms
! 3 6.6.6.6 5 ms


LDP is preffered by default. For the SR to override LDP preference next configuration can be added for the target router "sr-label-preferred" (In my virtual environment somehow it doesn't influence SR preference, so LDP was just completely disabled on R1):

 
R1#sh run | s segment
segment-routing mpls
 !
 set-attributes
  address-family ipv4
   sr-label-preferred



R1#sh isis segment-routing
 ISIS protocol is registered with MFI
 ISIS MFI Client ID:0x63
 Tag 1 - Segment-Routing:
   SR State:SR_ENABLED
   Number of SRGB:1
   SRGB Start:16000, Range:8000, srgb_handle:0x7F2D1539C7B8, srgb_state: created
   Address-family IPv4 unicast SR is configured
     Operational state:Enabled
     Receive is enabled
     Advertise local is disabled
     Explicit null is disabled
     SR label preferred is enabled



R1#trace mpls ipv4 6.6.6.6/32
Tracing MPLS Label Switched Path to 6.6.6.6/32, timeout is 2 seconds

Type escape sequence to abort.
  0 10.0.0.0 MRU 1500 [Labels: 16006 Exp: 0]
L 1 10.0.0.1 MRU 1500 [Labels: 299856 Exp: 0] 12 ms
L 2 4.4.4.4 MRU 1522 [Labels: implicit-null Exp: 0] 62 ms
! 3 6.6.6.6 3 ms



R1#trace vrf CE1 66.66.66.66 probe 1
Type escape sequence to abort.
Tracing the route to 66.66.66.66
VRF info: (vrf in name/id, vrf out name/id)
  1 10.0.0.3 [MPLS: Labels 16006/299840 Exp 0] 5 msec
  2 10.0.0.5 [MPLS: Labels 299856/299840 Exp 0] 5 msec
  3 10.0.0.11 [MPLS: Label 299840 Exp 0] 79 msec
  4 66.66.66.66 3 msec


And from R6 back to R1:

root@R6# run traceroute routing-instance CE1 11.11.11.11
 traceroute to 11.11.11.11 (11.11.11.11), 30 hops max, 40 byte packets
 1  10.0.0.10 (10.0.0.10)  4.039 ms 10.0.0.8 (10.0.0.8)  2.811 ms  2.862 ms
     MPLS Label=299792 CoS=0 TTL=1 S=0
     MPLS Label=25 CoS=0 TTL=1 S=1
 2  10.0.0.6 (10.0.0.6)  5.460 ms  6.734 ms 10.0.0.4 (10.0.0.4)  4.834 ms
     MPLS Label=16 CoS=0 TTL=1 S=0
     MPLS Label=25 CoS=0 TTL=1 S=1
 3  11.11.11.11 (11.11.11.11)  9.722 ms  7.675 ms  5.061 ms







The interesting thing worth to have attention is that FEC 1.1.1.1/32 have SR Node SID 16001 advertised by itself as well as LDP Label 16 advertised by R2 and R3 for LDP dataplane reachability. Because of this LSP from LDP island swaps label with 16 and not 16001.




R2#sh mpls forwarding-table 1.1.1.1
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
16         Pop Label  1.1.1.1/32       106904        Gi1.12     10.0.0.0   
16001      Pop Label  1.1.1.1/32       106904        Gi1.12     10.0.0.0   

Final SR/LDP dataplane

 

 

So how to get from R1 Lo1 to R6 lo0.1 and vice versa. Let's check all the related tables.






R1#sh ip ro vrf CE1 66.66.66.66

Routing Table: CE1
Routing entry for 66.66.66.66/32
  Known via "bgp 1", distance 200, metric 0, type internal
  Last update from 6.6.6.6 1d21h ago
  Routing Descriptor Blocks:
  * 6.6.6.6 (default), from 2.2.2.2, 1d21h ago
      Route metric is 0, traffic share count is 1
      AS Hops 0
      MPLS label: 299840
      MPLS Flags: MPLS Required






R1#sh bgp vpnv4 unicast vrf CE1 66.66.66.66/32
BGP routing table entry for 1:1:66.66.66.66/32, version 13
Paths: (2 available, best #2, table CE1)
  Not advertised to any peer
  Local
    6.6.6.6 (metric 30) (via default) from 2.2.2.2 (2.2.2.2)
      Origin IGP, localpref 100, valid, internal, best
      Extended Community: RT:1:1
      Originator: 6.6.6.6, Cluster list: 2.2.2.2
      mpls labels in/out nolabel/299840
      rx pathid: 0, tx pathid: 0x0




R1#sh ip ro 6.6.6.6
Routing entry for 6.6.6.6/32
  Known via "isis", distance 115, metric 30, type level-2
--- output ommited ---
    10.0.0.1, from 6.6.6.6, 00:24:49 ago, via GigabitEthernet1.12, merge-labels
      Route metric is 30, traffic share count is 1
      MPLS label: 16006
      MPLS Flags: NSF




R1#sh ip cef 6.6.6.6 det
6.6.6.6/32, epoch 2, per-destination sharing
  sr local label info: global/16006 [0x3B]
  1 RR source [no flags]
  nexthop 10.0.0.1 GigabitEthernet1.12 label 16006()




Two labels has been discovered, one for VPN (299840) and one for transport (16006), so R1 is pushing both of them on top of IP packet and passes it to R2, which check FIB and swaps transport label with the other one (299856).

 
R2#sh mpls forwarding-table labels 16006
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
16006      299856     6.6.6.6/32       28770         Gi1.24     10.0.0.5   



R4 running JUNOS inspects it's mpls.0 table and popping top label, leaving VPN label intact.


root@R4# run show route table mpls.0 label 299856

mpls.0: 14 destinations, 14 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299856             *[LDP/9] 1d 02:37:57, metric 1
                    > to 10.0.0.9 via ge-0/0/0.46, Pop     
299856(S=0)        *[LDP/9] 1d 02:37:57, metric 1
                    > to 10.0.0.9 via ge-0/0/0.46, Pop     





Packet arrives to R6 with the VPN label only and after popping it pure IP packet lends in routing-instance (VRF)  CE1.




root@R6# run show route table mpls.0 label 299840

mpls.0: 14 destinations, 14 routes (14 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299840             *[VPN/170] 1d 21:37:00
                      receive table CE1.inet.0, Pop     





Same things happens on the way back, R6 pushes two labels, R5 swaps transport label, R3 pops it and brings packet with VPN label only to R1.



root@R6# run show route table CE1.inet.0 11.11.11.11/32

CE1.inet.0: 2 destinations, 3 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

11.11.11.11/32     *[BGP/170] 1d 21:38:19, MED 0, localpref 100, from 2.2.2.2
                      AS path: ?, validation-state: unverified
                    > to 10.0.0.10 via ge-0/0/0.56, Push 25, Push 299824(top)




R1#sh mpls forwarding-table labels 25
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
25         Pop Label  11.11.11.11/32[V]   \
                                       10332         aggregate/CE1 




 

Remarks

* One issue worth to be mentioned is that upon enabling SR FIB in R2 and R3 wasn't programmed sucessfully and only interface triggering with shutdown/no shutdown gives correct label operation order ("No Label" and "drop" breaks dataplane in MPLS environment):


R2#sh mpls for
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
16004      No Label   4.4.4.4/32       0             drop      
16006      No Label   6.6.6.6/32       0             drop    



R3#sh mpls for
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop   
Label      Label      or Tunnel Id     Switched      interface             
16005      No Label   5.5.5.5/32       0             drop      
16006      No Label   6.6.6.6/32       0             drop         

Software upgrade procedures on the routers from different vendors - Part 3

In a part 3  (Part 1 and Part 2) we will observe the most complicated upgrade of CISCO CRS-1 (aka HFR (huge f**cking router)) running IOS-XR. Competing with Junos in flexibility and expanded capabilities which are absent in a standart IOS, IOS-XR requires a lot of time and moves during upgrade procedure. Same time adding some SMU-s (bugfix) to the current software may require reload of the box and lead to service interruption for about 30 minutes. ISSU functionality is not an option at all, especially for major release upgrades.
So, let's cross the fingers and start upgrade procedure:






IOS XR Upgrade Procedures on CRS-1 with Redundant RP-s

Example will be based on CRS-1 upgrade from 4.1.2 to 4.2.4 PX image.

First let’s obtain required files from CCO:

1.       PIE file, in our case composite one with k9 : CRS-iosxr-px-k9-4.2.4.tar

2.       SMU files, in our case recommended one :  4.2.4_hfr-px_REC_SMUS_2013-06-10.tar

Next untar archives on PC and copy all the files to router’s RP0 and RP1  (harddisk:)

Check free space :

RP/0/RP0/CPU0:#dir harddisk:

RP/0/RP0/CPU0:#dir harddisk: location 0/RP1/CPU0

Before copy operation new directory has to be created on harddisks :

RP/0/RP0/CPU0:#admin mkdir harddisk:/iosxr424

RP/0/RP0/CPU0:#admin mkdir harddisk:/iosxr424 location 0/RP1/CPU0

Copy files via ftp to CRS:

RP/0/RP0/CPU0:#copy ftp://1.1.1.1/4.2.4.files harddisk:/iosxr424

RP/0/RP0/CPU0:#copy ftp://1.1.1.1/4.2.4.files harddisk:/iosxr424  location 0/RP1/CPU0 

Remark : 4.2.4.files are all the files from TAR archives

Also we can copy between RP-s with the wildcard in filename, if files are already saved to RP0 : 

RP/0/RP0/CPU0:#copy harddisk:/iosxr424/* location 0/RP0/CPU0  harddisk:/iosxr424/  location 0/RP1/CPU0

Perform necessary system check:

show platform

show install active

cfs check

admin show license

 

 

RP/0/RP0/CPU0:#sh hfr
Node          Type              PLIM               State           Config State
------------- ----------------- ------------------ --------------- ---------------
0/0/2         FP40(SPA)         10X1GE             OK              PWR,NSHUT,MON
0/0/4         FP40(SPA)         OC192RPR-XFP       OK              PWR,NSHUT,MON
0/4/CPU0      FP40              2-10GbE-FLX        IOS XR RUN      PWR,NSHUT,MON
0/4/0         FP40(SPA)         1x10GE             OK              PWR,NSHUT,MON
0/4/4         FP40(SPA)         OC192RPR-XFP       OK              PWR,NSHUT,MON
0/6/CPU0      FP40              4-10GbE            IOS XR RUN      PWR,NSHUT,MON
0/7/CPU0      FP40              4-10GbE            IOS XR RUN      PWR,NSHUT,MON
0/RP0/CPU0    RP(Active)        N/A                IOS XR RUN      PWR,NSHUT,MON
0/RP1/CPU0    RP(Standby)       N/A                IOS XR RUN      PWR,NSHUT,MON

 

and so on…

Check free space on flash cards (disk0):

RP/0/RP0/CPU0:#show media

 

Media Information for 0/RP0/CPU0.

                   Image   Current  Part

  Mountpoint       FsType  FsType   Size   State         DrvrPid  Mirror  Flags

================================================================================

  /disk0:          FAT16   FAT32    3.4G Mounted         0036891  Enabled

  /disk0a:         FAT16   FAT16    0.5G Mounted         0036891         

  /disk1:          FAT16   (?)           Not Present                     

  /disk1a:         FAT16   (?)           Not Present                      

  /harddisk:       QNX4    QNX4    33.5G Mounted         0032791         

  /harddiska:      QNX4    QNX4    11.2G Mounted         0032791         

  /harddiskb:      FAT32   FAT32   11.2G Mounted         0032791         

  /lcdisk0:        FAT32   (?)           Not Present                     

  /lcdisk0a:       FAT32   (?)           Not Present                     

RP/0/RP0/CPU0:#show media location 0/RP1/CPU0

 

Media Information for 0/RP1/CPU0.

                   Image   Current  Part

  Mountpoint       FsType  FsType   Size   State         DrvrPid  Mirror  Flags

================================================================================

  /disk0:          FAT16   FAT32    3.4G Mounted         0036891  Enabled

  /disk0a:         FAT16   FAT16    0.5G Mounted         0036891         

  /disk1:          FAT16   (?)           Not Present                     

  /disk1a:         FAT16   (?)           Not Present                     

  /harddisk:       QNX4    QNX4    33.5G Mounted         0028696         

  /harddiska:      QNX4    QNX4    11.2G Mounted         0028696         

  /harddiskb:      FAT32   FAT32   11.2G Mounted         0028696         

  /lcdisk0:        FAT32   (?)           Not Present                      

  /lcdisk0a:       FAT32   (?)           Not Present                    

RP/0/RP0/CPU0#show filesystem  disk0:

   Model:                       UNIGEN FLASH                          

   Firmware:                       30/06/03

   BIOS Geometry:               16 Heads, 63 Sectors

   Drive Geometry:              16 Heads, 8150 Tracks, 63 Sectors

   Drive Capacity:              8215200 Cur Sctrs, 8215200 User Sctrs, Extd

   Address Mode:                LBA

   PIO mode:                    2

   Multimode Blocks/Transfer:   32

   Capacity:    8215201 Sectors, Total 4206182912 Bytes, (512 Bytes/sector)

RP/0/RP0/CPU0#show filesystem  disk0: location 0/RP1/CPU0

   Model:                       UNIGEN FLASH                          

   Firmware:                       30/06/03

   BIOS Geometry:               16 Heads, 63 Sectors

   Drive Geometry:              16 Heads, 8150 Tracks, 63 Sectors

   Drive Capacity:              8215200 Cur Sctrs, 8215200 User Sctrs, Extd

   Address Mode:                LBA

   PIO mode:                    2

   Multimode Blocks/Transfer:   32

   Capacity:    8215201 Sectors, Total 4206182912 Bytes, (512 Bytes/sector)

If there’s not enough space on disk0 (at least 1.5GB), then inactive packages should be removed:

RP/0/RP0/CPU0:#admin show install inactive

if found some inactive packages then do next:

RP/0/RP0/CPU0:# admin install remove inactive

RP/0/RP0/CPU0:# admin install commit

For the upgrade process to flow without interruptions it’s recommended to offload the traffic and shutdown routing protocols (BGP, OSPF, ISIS, LDP etc.) Even better is to reload whole box before upgrade:

 

RP/0/RP0/CPU0:#admin reload location all

Also let the FPD-s upgraded automatically during the new image installation:

RP/0/RP0/CPU0:#admin

RP/0/RP0/CPU0(admin):#conf  t

RP/0/RP0/CPU0(admin):# fpd auto-upgrade

RP/0/RP0/CPU0(admin):# commit

Let the upgrade process begin with the text scripts prepared in notepad:

RP/0/RP0/CPU0:# admin install add harddisk:/424/hfr-mpls-px.pie-4.2.4 harddisk:/424/hfr-services-px.pie-4.2.4 harddisk:/424/hfr-fpd-px.pie-4.2.4 harddisk:/424/hfr-mcast-px.pie-4.2.4 harddisk:/424/hfr-mini-px.pie-4.2.4 harddisk:/424/hfr-k9sec-px.pie-4.2.4 harddisk:/424/hfr-diags-px.pie-4.2.4 harddisk:/424/hfr-mgbl-px.pie-4.2.4 harddisk:/424/hfr-doc-px.pie-4.2.4 sync

 

Info:     The following packages are now available to be activated:

Info:     

Info:         disk0:hfr-mpls-px-4.2.4

Info:         disk0:hfr-services-px-4.2.4

Info:         disk0:hfr-fpd-px-4.2.4

Info:         disk0:hfr-mcast-px-4.2.4

Info:         disk0:hfr-mini-px-4.2.4

Info:         disk0:hfr-k9sec-px-4.2.4

Info:         disk0:hfr-diags-px-4.2.4

Info:         disk0:hfr-mgbl-px-4.2.4

Info:         disk0:hfr-doc-px-4.2.4

Info:     

Info:     The packages can be activated across the entire router.

Let’s add the recommended SMU-s also:

RP/0/RP0/CPU0:# admin install add harddisk:/424/hfr-px-4.2.4.CSCue53201.pie harddisk:/424/hfr-px-4.2.4.CSCug09031.pie harddisk:/424/hfr-px-4.2.4.CSCug20386.pie harddisk:/424/hfr-px-4.2.4.CSCue55783.pie harddisk:/424/hfr-px-4.2.4.CSCue71114.pie harddisk:/424/hfr-px-4.2.4.CSCue04603.pie harddisk:/424/hfr-px-4.2.4.CSCue19011.pie harddisk:/424/hfr-px-4.2.4.CSCuc56287.pie harddisk:/424/hfr-px-4.2.4.CSCue21974.pie harddisk:/424/hfr-px-4.2.4.CSCud41972.pie sync

 

Info:     The following packages are now available to be activated:

Info:     

Info:         disk0:hfr-px-4.2.4.CSCue53201-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCug09031-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCug20386-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCue55783-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCue71114-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCue04603-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCue19011-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCuc56287-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCue21974-1.0.0

Info:         disk0:hfr-px-4.2.4.CSCud41972-1.0.0

Info:     

Info:     The packages can be activated across the entire router.

Time to activate new XR:

RP/0/RP0/CPU0:# admin install activate disk0:hfr-mpls-px-4.2.4 disk0:hfr-doc-px-4.2.4 disk0:hfr-services-px-4.2.4 disk0:hfr-fpd-px-4.2.4 disk0:hfr-mcast-px-4.2.4 disk0:hfr-mini-px-4.2.4 disk0:hfr-k9sec-px-4.2.4 disk0:hfr-diags-px-4.2.4 disk0:hfr-mgbl-px-4.2.4 sync test

“test” keyword is needed to perform virtual installation and check if any errors can be found. After success remove “test” and perform real activation…

Info:     This operation will reload the following nodes in parallel:

Info:         0/0/SP (MSC-DRP-SP) (Admin Resource)

Info:         0/4/SP (MSC-DRP-SP) (Admin Resource)

Info:         0/6/SP (MSC-DRP-SP) (Admin Resource)

Info:         0/7/SP (MSC-DRP-SP) (Admin Resource)

Info:         0/0/CPU0 (LC) (SDR: Owner)

Info:         0/4/CPU0 (LC) (SDR: Owner)

Info:         0/6/CPU0 (LC) (SDR: Owner)

Info:         0/7/CPU0 (LC) (SDR: Owner)

Info:         0/RP0/CPU0 (HRP) (SDR: Owner)

Info:         0/SM0/SP (140G-Fabric-SP-B) (Admin Resource)

Info:         0/SM1/SP (140G-Fabric-SP-B) (Admin Resource)

Info:         0/SM2/SP (140G-Fabric-SP-B) (Admin Resource)

Info:         0/SM3/SP (140G-Fabric-SP-B) (Admin Resource)

Proceed with this install operation (y/n)? [y]

Info:     Install Method: Parallel Reload

 

….

 

Info:     The changes made to software configurations will not be persistent

Info:     across system reloads. Use the command '(admin) install commit' to

Info:     make changes persistent.

Info:     Please verify that the system is consistent following the software

Info:     change using the following commands:

Info:         show system verify

Info:         install verify packages

Install operation 53 completed successfully at 00:00:00

Router will reload itself upon completing of installation and activate new image:

Do important check of the new system :

RP/0/RP0/CPU0:#show version

RP/0/RP0/CPU0:#show configuration failed startup

RP/0/RP0/CPU0:#admin show configuration failed startup

RP/0/RP0/CPU0:#show platform

If everything is fine apply the final commit for the new image :

RP/0/RP0/CPU0:# admin install commit

Install operation 54 '(admin) install commit' started by user ‘root’ via CLI at

| 100% complete: The operation can no longer be aborted (ctrl-c for options)RP/0/RP0/CPU0: instdir[251]: %INSTALL-INSTMGR-4-ACTIVE_SOFTWARE_COMMITTED_INFO : The currently active software is now the same as the committed software.

Install operation 54 completed successfully at 00:00:00

Activating SMU-s are absolutely the same process as activating PIE-s, just create new text script and paste it into the box. If the some of the SMU-s requires reload then at the end of the activation box will be rebooted. To finalize type “admin install commit” one more time for the SMU-s.

Downtime during each reload is about 20-30 minutes. “Add” operation takes about 1 hour, activation with reboot +30 minutes. Almost the same timings are for the SMU-s add+activation.

Total outage expected is 4+ hours!

TIP: Before the start of the upgrade procedure physically remove (with the help of OIR) RP1 from the router. With this operation you can do the fast rollback to the old XR version saved on RP1. After the complete upgrade to new XR on RP0 and putting system in service just insert the RP1 back to the box and it will perform its upgrade to the new image automatically. First it’ll download required files via tftp from the RP0 and then add+activate them. This process will take additional 4 hours for RP1, but the box during that time can be in production and work without any interruption.

PS: Check the licenses are in place after upgrade: “admin show license”.



Small comparison table of upgrade levels:

Vendor	Model	Upgrade files' count	Total files' size	Outage time	Difficulty Level
Cisco	C7600	2	200MB	15-20 min	Easy
Juniper	MX960	1	420MB	2 min	Moderate
Cisco	CRS-1	10+	1GB+	4 + hours	Challenging