Powered by Max Banner Ads 

@fryguy

Archive for November, 2010|Monthly archive page

Clearing hung TCP session on a Cisco router

In Uncategorized on November 19, 2010 at 08:34

While going through some of my old notes I came across some notes from a few years ago on hung TCP sessions on some of our routers. These hung sessions where due to the latency differences on a Multi-link T1 connection that was in place.  Below are the notes (generalized IP and Names) on how to detect and reset these connections.It is my hope for you to never have to use these commands, but since there is little out there on them I figured I would share.

1) We saw this error (connection queue limit reached) on the terminating gateway (10.10.3.5) during an incoming call attempt:

R1#
R1#debug ip tcp trans
TCP special event debugging is on
R1#term mon
R1#
Mar  6 04:49:55 PHL: TCP0: bad seg from 10.11.3.20 — connection queue limit reached: seq 3379916701 ack 0 rcvnxt 0 rcvwnd 4128 len 0
Mar  6 04:49:57 PHL: TCP0: bad seg from 10.11.3.20 — connection queue limit reached: seq 3379916701 ack 0 rcvnxt 0 rcvwnd 4128 len 0
Mar  6 04:50:01 PHL: TCP0: bad seg from 10.11.3.20 — connection queue limit reached: seq 3379916701 ack 0 rcvnxt 0 rcvwnd 4128 len 0
Mar  6 04:50:09 PHL: TCP0: bad seg from 10.11.3.20 — connection queue limit reached: seq 3379916701 ack 0 rcvnxt 0 rcvwnd 4128 len 0

R1#

2) Take a look at: 

R1#sh tcp brief
TCB       Local Address   Foreign Address        (state)
6353F5E8 10.10.3.5.1720   10.11.3.24.12871      SYNRCVD
63555A14 10.10.3.5.1720   10.11.3.23.12814      SYNRCVD
6353AEFC 10.10.3.5.1720   10.11.3.24.12872      SYNRCVD
6350B2DC 10.10.3.5.1720   10.11.3.24.12875      SYNRCVD
63488D44  10.10.3.5.23    10.11.3.23.11265      ESTAB
63571718 10.10.3.5.1720   10.11.3.24.12914      SYNRCVD
All of the TCB's associated with TCP port 1720 are 'hung' (SYNRCVD).
The associated TCP Control Block (TCB) are highlighted in a PINKISH colour.

3) To clear them, clear the associated TCB:

R1#clear tcp tcb 6353F5E8
[confirm]y
[OK]

R1#
R1#clear tcp tcb 63555A14
[confirm]y
[OK]

R1#
R1#clear tcp tcb 6353AEFC
[confirm]y
[OK]

R1#
R1#clear tcp tcb 6350B2DC
[confirm]y
[OK]

R1#
R1#clear tcp tcb 63571718
[confirm]y
[OK]

R1#

And the router started processing calls again.

R1#

Type 7 password decryption via IOS router (? bonus)

In Uncategorized on November 18, 2010 at 08:28

I have seen a few posts out there about this as well as links to sites that will decrypt a Cisco type 7 password.  Since I am trying to share what I know I figured I might as well throw my hat into the ring and include this information here.

If you have a type 7 enable password such as : enable password 7 00331A0A087D071F012F7F5B1B0F0C011754 and want to decrpyt it, all you need is another router that you can get to enable (priv 15) access on and configuration mode.

Enter configuration mode and create a key-chain. For the key-string, tell it type 7 and cut-and-past the encrypted key

Rack1R6(config)#key chain DecrpytThis
Rack1R6(config-keychain)#key 1
Rack1R6(config-keychain-key)#key-string 7 00331A0A087D071F012F7F5B1B0F0C011754

Then all you have to do from a command prompt is issue show key chain

Rack1R6#sh key chain
Key-chain DecrpytThis:
key 1 — text “WillFlynnSurvive?
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]
Rack1R6#

As you can see here, the password is WillFlynnSurvive?

This works for any Type 7 password, including users.  Below I have done a show run | inc user and they added Key 2 with he user type 7 password.

Rack1R6#sh run | inc user
username Flynn privilege 15 password 7 0478071303245F5D
Rack1R6#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Rack1R6(config)#key chain DecryptThis
Rack1R6(config-keychain)#key 2
Rack1R6(config-keychain-key)#key-string 7 0478071303245F5D
Rack1R6(config-keychain-key)#^Z
Rack1R6#sh key chain
Key-chain DecrpytThis:
key 1 — text “WillFlynnSurvive?”
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]
Key-chain DecryptThis:
key 2 — text “Cluless
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]
Rack1R6#

As you can see, the password for Key 2 ( the user one we just added ) is Cluless

 

Now, I am not sure if you noticed but for the first example the password is WillFlynnSurvive? – yes, that is a question mark in the password.  If you want to use a question mark as part of your password, you need to enter ctrl-v prior to typing the question mark in the password or the IOS will think you are looking for help. Old dirty CCIE lab prep tricks there.

 

 

Cisco Embedded Event Manager and its usefullness for TAC

In Uncategorized on November 16, 2010 at 12:49

A few years ago I had a TAC case open with Cisco.  The case was centered around an unusually high CPU condition on a Cisco 3800ISR series router.  This router was only routing traffic between the Ethernet interfaces, doing some SNAT (stateful NAT), and basic BGP routing – so the CPU should not have been in the 90%+ range.  Well, TAC wanted me to run some commands every 60 minutes 24×7 for a few days to see what was going on.  Now we all now that that is not an easy thing to do, so what I decided to do was grab an existing EEM script from Cisco’s website and modify it to provide all the information that the TAC engineer wanted.

Needless to say, the case was solved about 2 days later when we upgraded the code per their suggestion because of the identification of the bug. :)   I am guessing that having a router send you the logs and command output every hour either provided them the information they needed, or more likely, the engineer realized that  – well, you know.

So, for this post I have decided to post the EEM configuration as well as the EEM script in case you ever need to do something similar.  If I can save you some time, I am all for that.  I still use this script today – albeit in a different manner.  I now use it to get “Daily Health Checks” from the routers in the field.  Every day at noon they send me standard output commands so that we can check on them without having to log into each and every one of them.

So, without further typing – here is the script broken down: (to download this script, you can click here – HealthCheck_BLOG.tcl

Router Configuration:

First thing we will need to do is configure the Event Manager variables on the router:
The variables that we will use are:

  • Mail Server (SMTP) – 10.11.11.254
  • Receipt E-Mail address – Flynn@Domain.Com
  • From E-Mail address – CluRouter@Domain.Com
  • TCL file location – flash:/
  • User who runs the eem – eem_user

First we configure the EEM Environment using the event manager commands
Router(config)#event manager environment _email_server 10.11.11.254
Router(config)#event manager environment _email_to Flynn@Domain.Com
Router(config)#event manager environment _email_from CluRouter@Domain.Com
Router(config)#event manager directory user policy “flash:/”
Router(config)#event manager session cli username “eem_user”
!
Next we define the scheduling of this command (here at 16:00 every day)
Router(config)#kron occurrence Daily at 16:00 recurring
Router(config)#policy-list HealthCheck_BLOG
!

Now we can device that happens when this KRON job is run.
Router(config)#kron policy-list HealthCheck_BLOG
Router(config)#cli event manager run HealthCheck_BLOG.tcl
!

Lastly we can define the policy
Router(config)#event manager policy HealthCheck_BLOG.tcl
!

———————————————————————-
EEM Script section
———————————————————————-

This .tcl (originally was sendmail.tcl) file was taken from Cisco’s website and then modified.  I will explain only what I have modified in order to accomplish the task of sending logs.

The sections that I have modified are:
#———————– “show commands” —————-
#———————– send mail ———————-

Under the Show Commands section:
To display the clock and store the results in $result:

if [catch {cli_exec $cli(fd) "show clock"} result] {
error $result $errorInfo
}
set show_clock $result

To display the show ip interfaces and store the results in show_ip_interfaces

if [catch {cli_exec $cli(fd) "show ip interfaces"} result] {
error $result $errorInfo
}
set show_ip_interfaces $result

To show the command show align and store the results in the show_align variable
if [catch {cli_exec $cli(fd) "show align"} result] {
error $result $errorInfo
}
set show_align $result

To execute the command show buffers gig 0/0 dump and store in show_buffers_gig_0_0
if [catch {cli_exec $cli(fd) "show buffers gig 0/0 dump"} result] {
error $result $errorInfo
}
set show_buffers_gig_0_0 $result

Now to send the mail most of the variables where defined in the EEM environment originally, all that I had to do was the subject and body

#———————– send mail ———————-
#
# create mail form
action_syslog msg “Creating mail header…”
set body [format "Mailservername: %s" "$_email_server"]
set body [format "%snFrom: %s" "$body" "$_email_from"]
set body [format "%snTo: %s" "$body" "$_email_to"]
set _email_cc “”
set body [format "%snCc: %s" "$body" ""]

#setting the subject
set body [format "%snSubject: %sn" "$body" "SR TAC CASE NUMBER - $routername..."]

#outputting the command output into the e-mail body
set body [format "%sn%s" "$body" "The body of your msg goes here..."]
set body [format "%sn%s" "$body" "Report Summary:"]
set body [format "%sn%s" "$body" "   - Show Clock"]
set body [format "%sn%s" "$body" "   - Show ip interfaces"]
set body [format "%sn%s" "$body" "   - Show align"]
set body [format "%sn%s" "$body" "   - Show buffers gig 0/0 dump"]
set body [format "%snn%s" "$body" "---------- Show Clock----------"]
set body [format "%snn%s" "$body" "$show_clock"]
set body [format "%snn%s" "$body" "---------- Show IP Interfaces ----------"]
set body [format "%snn%s" "$body" "$show_ip_interfaces "]
set body [format "%snn%s" "$body" "---------- Show Align ----------"]
set body [format "%snn%s" "$body" "$show_align"]
set body [format "%snn%s" "$body" "---------- Show buffers gig 0/0 dump ----------"]
set body [format "%snn%s" "$body" "$show_buffers_gig_0_0"]

Below is a pure dump of the script if you want to look at it in its raw format

###############################################################################################################
#
#     Daily Health Check TCL Script taken from Cisco.Com web site
#
#    Update by Jeff Fry – 9/23/2008
#
###############################################################################################################

# Useful event registration tcl command extensions
# None
::cisco::eem::event_register_none queue_priority low nice 1 maxrun 600
# Watchdog Timer
#::cisco::eem::event_register_timer watchdog name errimt time $errim_period queue_priority low nice 1
# Syslog
#::cisco::eem::event_register_syslog occurs 1  pattern .*STANDBY.*STATECHANGE.* maxrun 90 queue_priority low nice 1
# Object Tracking
#::cisco::eem::event_register_track 1 state up queue_priority low nice 1
# Interface
#::cisco::eem::event_register_interface name $intf parameter txload entry_op ge entry_val 192 entry_val_is_increment FALSE queue_priority low nice 1
# Cron Job
#::cisco::eem::event_register_timer cron name test cron_entry “0 * * * *” queue_priority low nice 1 maxrun 20

#
# Namespace imports
#
namespace import ::cisco::eem::*
namespace import ::cisco::lib::*

#— Check required environment variable(s) has been defined

if {![info exists _email_server]} {
set result “EEM Policy Error: variable $_email_server has not been set”
error $result $errorInfo
}

if {![info exists _email_to]} {
set result “EEM Policy Error: variable $_email_to has not been set”
error $result $errorInfo
}

if {![info exists _email_from]} {
set result “EEM Policy Error: variable $_email_from has not been set”
error $result $errorInfo
}

#——————  hostname        ——————-
set routername [info hostname]

#
#——————-   ” cli open”   ——————-
#
if [catch {cli_open} result] {
error $result $errorInfo
} else {
array set cli $result
}

#————— end of  “cli open”   ——————-

#
#———————– “show commands” —————-
#
if [catch {cli_exec $cli(fd) "enable"} result] {
error $result $errorInfo
}

if [catch {cli_exec $cli(fd) "show clock"} result] {
error $result $errorInfo
}
set show_clock $result

if [catch {cli_exec $cli(fd) "show ip interfaces"} result] {
error $result $errorInfo
}
set show_ip_interfaces $result

if [catch {cli_exec $cli(fd) "show align"} result] {
error $result $errorInfo
}
set show_align $result

if [catch {cli_exec $cli(fd) "show buffers gig 0/0 dump"} result] {
error $result $errorInfo
}
set show_buffers_gig_0_0 $result

#———————- end of show commands ————

#
#———————– “enable” ———————-
#
if [catch {cli_exec $cli(fd) "enable"} result] {
error $result $errorInfo
}

#
#———————– send mail ———————-
#
# create mail form
action_syslog msg “Creating mail header…”
set body [format "Mailservername: %s" "$_email_server"]
set body [format "%snFrom: %s" "$body" "$_email_from"]
set body [format "%snTo: %s" "$body" "$_email_to"]
set _email_cc “”
set body [format "%snCc: %s" "$body" ""]
set body [format "%snSubject: %sn" "$body" "SR TAC CASE NUMBER - $routername..."]

set body [format "%sn%s" "$body" "The body of your msg goes here..."]
set body [format "%sn%s" "$body" "Report Summary:"]
set body [format "%sn%s" "$body" "   - Show Clock"]
set body [format "%sn%s" "$body" "   - Show ip interfaces"]
set body [format "%sn%s" "$body" "   - Show align"]
set body [format "%sn%s" "$body" "   - Show buffers gig 0/0 dump"]
set body [format "%snn%s" "$body" "---------- Show Clock----------"]
set body [format "%snn%s" "$body" "$show_clock"]
set body [format "%snn%s" "$body" "---------- Show IP Interfaces ----------"]
set body [format "%snn%s" "$body" "$show_ip_interfaces "]
set body [format "%snn%s" "$body" "---------- Show Align ----------"]
set body [format "%snn%s" "$body" "$show_align"]
set body [format "%snn%s" "$body" "---------- Show buffers gig 0/0 dump ----------"]
set body [format "%snn%s" "$body" "$show_buffers_gig_0_0"]

if [catch {smtp_send_email $body} result] {
action_syslog msg “smtp_send_email: $result”
}

action_syslog msg “E-mail sent!”
#—————— end of send mail ——————–

#
#——————— cli close ————————
#
cli_close $cli(fd) $cli(tty_id)

# eeeeeeeeeeeeeeeeeeeeeeeeeeee    End of sendmail.tcl eeeeeeeeeeeeeeeeeeeeee

%d bloggers like this: