Infrastructure Monitoring
IP Service Level Agreement (IP SLA)
IP SLA allows you to probe different key performance indicators of the network such as reachability, latency, jitter, etc.
For the purpose of CCIE Datacenter, we are only going to enable feature sla sender. In particular, I am going to explain the ICMP echo configuration.
N9K01(config)# feature sla sender N9K01(config)# ip sla 111 N9K01(config-ip-sla)# icmp-echo 1.1.1.1 source-interface e1/1 N9K01(config-ip-sla-echo)# threshold 40 N9K01(config-ip-sla-echo)# timeout 120 N9K01(config-ip-sla-echo)# frequency 2 N9K01(config-ip-sla-echo)# ip sla schedule 111 life forever start-time now
- Frequency: The seconds between each operation.
- Timeout: The number of milliseconds that operation waits for a response to report the IP SLA Up state.
- Threshold: Same as timeout but the milliseconds which generates syslog/trap. It needs to be less than or equal to the value of timeout.
After IP SLA configuration, you need to schedule it to start.
Object Tracker (Track)
You cannot employ IP SLA to do something for you based on it is state with Cisco NX-OS. You need to “track” an SLA, then use that track object somewhere which supports track object. For example, you can use tracks in FHRPs, EEM, or a static route.
An object tracker monitors the state of an interface or an IP SLA probe. The snippet below shows how track object can track the state of an interface and the state of an IP SLA.
! An example that tracks the state of an interface N9K01(config)# track 1 interface e1/1 line-protocol ! An example which shows how a track is leveraged to track the state of an SLA N9K01(config)# track 111 ip sla 111 reachability
A track can also monitor another track. For example,
N9K01(config)# track 100 N9K01(config-track)# object 111
In the example above, track 100 tracks track 111. So, in the end of the day, they will have the same result. Well, that is not much useful for us. Let us create a track object which tracks multiple other tracks.
N9K01(config)# feature sla sender N9K01(config)# ip sla 111 N9K01(config-ip-sla)# icmp-echo 1.1.1.1 source-interface e1/1 N9K01(config-ip-sla-echo)# threshold 40 N9K01(config-ip-sla-echo)# timeout 120 N9K01(config-ip-sla-echo)# frequency 2 N9K01(config-ip-sla-echo)# ip sla schedule 111 life forever start-time now N9K01(config)# track 111 ip sla 111 reachability N9K01(config)# ip sla 112 N9K01(config-ip-sla)# icmp-echo 2.2.2.2 source-interface e1/1 N9K01(config-ip-sla-echo)# threshold 40 N9K01(config-ip-sla-echo)# timeout 120 N9K01(config-ip-sla-echo)# frequency 2 N9K01(config-ip-sla-echo)# ip sla schedule 112 life forever start-time now N9K01(config)# track 112 ip sla 112 reachability N9K01(config)# ip sla 113 N9K01(config-ip-sla)# icmp-echo 2.2.2.2 source-interface e1/1 N9K01(config-ip-sla-echo)# threshold 40 N9K01(config-ip-sla-echo)# timeout 120 N9K01(config-ip-sla-echo)# frequency 2 N9K01(config-ip-sla-echo)# ip sla schedule 113 life forever start-time now N9K01(config)# track 113 ip sla 113 reachability N9K01(config)# track 100 list threshold percentage N9K01(config-track)# object 111 N9K01(config-track)# object 112 N9K01(config-track)# object 113 N9K01(config-track)# threshold percentage down 40 up 60 N9K01(config-track)# delay down 6
As an exercise, experiment with track NUMBER list boolean command.
Embedded Event Manager (EEM)
EEM monitors the events (event statement: for example, interface shuts down) that occur on your Nexus switch and then takes action (action statement: generate syslog/trap, send email, run a series of command, no shut the interface) upon that event.
To configure the EEM you need to go through the following steps:
event manager applet NAMEevent event_statementaction number action_statement
You can also define variables for EEM with command event manager environment variable_name variable_value
As an Example, Let us ask NX-OS if the track 100 is down, it sends us a syslog message.
N9K01(config)# event manager applet TRACK_100_DOWN N9K01(config-applet)# event track 100 state down N9K01(config-applet)# action 01.0 snmp-trap strdata "SNMP: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore." N9K01(config-applet)# action 02.0 syslog msg SYSLOG: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore.
Should you need more information about the above examples including EEM, I strongly suggest that you review the workshop in this post.
Workshop
In this workshop, we are going to simulate Automatic BGP switchover in case that not directly connected link to ISP fails but if some targets in the Internet fail as per IP SLA perspective. This can happen if the upstream service provider is DOSed for example. I implemented the original solution in production with IOS-XE to switchover the BGP back after one hour of stability. But with NX-OS since there was no event timer countdown command, I suggest you use the up delay which is restricted to maximum of 3 minutes.
N9K01:
hostname N9K01 feature bgp feature sla sender ip prefix-list AS_90 seq 5 permit 192.190.0.0/24 ip prefix-list DEFAULT_ROUTE seq 5 permit 0.0.0.0/0 route-map AS_PREPEND permit 10 match ip address prefix-list AS_90 route-map LOCAL_PREF permit 10 match ip address prefix-list DEFAULT_ROUTE set local-preference 150 interface Ethernet1/1 ip address 91.91.91.9/24 no shutdown interface Ethernet1/2 ip address 169.254.0.1/30 no shutdown interface loopback0 ip address 192.190.0.1/24 router bgp 90 address-family ipv4 unicast network 192.190.0.0/24 neighbor 91.91.91.1 remote-as 111 address-family ipv4 unicast route-map LOCAL_PREF in route-map AS_PREPEND out neighbor 169.254.0.2 remote-as 90 address-family ipv4 unicast next-hop-self ip access-list R1-DIRECT-INTERNET 10 permit icmp 91.91.91.9/32 1.1.1.1/32 20 permit icmp 91.91.91.9/32 2.2.2.2/32 30 permit icmp 91.91.91.9/32 3.3.3.3/32 route-map R1-DIRECT-INTERNET permit 10 match ip address R1-DIRECT-INTERNET set ip next-hop 91.91.91.1 ip local policy route-map R1-DIRECT-INTERNET feature sla sender ip sla 111 icmp-echo 1.1.1.1 source-interface e1/1 threshold 40 timeout 120 frequency 2 ip sla schedule 111 life forever start-time now track 111 ip sla 111 reachability ip sla 112 icmp-echo 2.2.2.2 source-interface e1/1 threshold 40 timeout 120 frequency 2 ip sla schedule 112 life forever start-time now track 112 ip sla 112 reachability ip sla 113 icmp-echo 3.3.3.3 source-interface e1/1 threshold 40 timeout 120 frequency 2 ip sla schedule 113 life forever start-time now track 113 ip sla 113 reachability track 100 list threshold percentage object 111 object 112 object 113 threshold percentage down 40 up 60 delay down 6 delay up 180 event manager applet TRACK_100_DOWN event track 100 state down action 01.0 snmp-trap strdata "SNMP: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore." action 02.0 syslog msg SYSLOG: At least two of IP SLAs 111, 112, and 113 are not receiving ICMP echo replies on N9K01 - 91.91.91.9 anymore. no event manager applet TRACK_ISP_UP event manager applet TRACK_ISP_UP event track 100 state up action 0.9 event-default action 1.0 cli conf t action 2.4 cli router bgp 90 action 2.5 cli neighbor 91.91.91.1 action 2.6 cli no shutdown action 2.8 cli end action 2.9 cli syslog msg BGP neighbor 91.91.91.1 is UP by EEM script action 3.0 cli snmp-trap strdata "BGP neighbor 91.91.91.1 is UP by EEM" no event manager applet eBGP-DOWN event manager applet eBGP-DOWN event track 100 state down action 0.9 event-default action 1.0 cli conf t action 2.0 cli no event manager applet eBGP_UP action 4.0 cli router bgp 90 action 5.0 cli neighbor 91.91.91.1 action 5.1 cli shutdown action 6.0 cli end action 7.0 syslog msg BGP neighbor 91.91.91.1 is shutdown by EEM script action 8.0 snmp-trap strdata "BGP neighbor 91.91.91.1 is shutdown by EEM"
N9K02:
hostname N9K02
feature bgp
ip prefix-list AS_90 seq 5 permit 192.190.0.0/24
ip prefix-list DEFAULT_ROUTE seq 5 permit 0.0.0.0/0
route-map AS_PREPEND permit 10
match ip address prefix-list AS_90
set as-path prepend 90 90
route-map LOCAL_PREF permit 10
match ip address prefix-list DEFAULT_ROUTE
set local-preference 90
interface Ethernet1/1
ip address 92.92.92.9/24
no shutdown
interface Ethernet1/2
ip address 169.254.0.2/30
no shutdown
interface loopback0
ip address 192.190.0.2/24
router bgp 90
address-family ipv4 unicast
network 192.190.0.0/24
neighbor 92.92.92.2
remote-as 222
address-family ipv4 unicast
route-map LOCAL_PREF in
route-map AS_PREPEND out
neighbor 169.254.0.1
remote-as 90
address-family ipv4 unicast
next-hop-self