Tags

, ,

MyView
Had a small problem today that was interesting, but not unique.  I have seen this problem before, but realized that maybe most people might not know how to fix it. I think the normal approach that people take is a reload of the device, not always a good thing to do on a production network in the middle of the day.
So the  depicts is a VTY session that is unclearable via the normal clear command route.  When you do a who, you see that someone is connected to VTY 0 for an extended period of time:

Switch1#who
    Line       User       Host(s)              Idle       Location
   1 vty 0                idle                     6w4d HackerHost.virtua.com.br
*  2 vty 1     jfry       idle                 00:00:00 192.168.0.1
  Interface      User        Mode                     Idle     Peer Address
Switch1#

They are not logged in as there is no User listed, but you can see they have been idle for 6 weeks and 4 days.
So the first think that you would do would be to clear the session and do a who again to make sure they are gone.

Switch1#clear line vty 0
[confirm]y [OK]
Switch1#who
    Line       User       Host(s)              Idle       Location
   1 vty 0                idle                     6w4d HackerHost.virtua.com.br
*  2 vty 1     jfry       idle                 00:00:00 192.168.0.1
  Interface      User        Mode                     Idle     Peer Address
Switch1#

Odd, they are still there.  Well, maybe its VTY 1 and not 0, lets clear that one instead:

Switch1#clear line vty 1
% Not allowed to clear current line
Switch1#who
    Line       User       Host(s)              Idle       Location
   1 vty 0                idle                     6w4d HackerHost.virtua.com.br
*  2 vty 1     jfry       idle                 00:00:00 192.168.0.1
  Interface      User        Mode                     Idle     Peer Address
Switch1#

Nope, that is my line – just like I thought.  Ok, so now what?
Since we know this is a TCP based session to the switch, we should be able to gather some TCP information. This will include their remote ip (10.0.13.37) ports (56667) as well as all our information 192.168.100.254 port 23 as you can see via the show tcp vty 0 command.

Switch1#sh tcp vty 0
tty1, virtual tty from host HackerHost.HackerDom.com
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 192.168.100.254, Local port: 23
Foreign host: 10.0.13.37, Foreign port: 56667
Enqueued packets for retransmit: 1, input: 0  mis-ordered: 1 (0 bytes)
Event Timers (current time is 0xBD23926C4):
Timer          Starts    Wakeups            Next
Retrans             1          0             0x0
TimeWait            0          0             0x0
AckHold             0          0             0x0
SendWnd         66409      66408     0xBD239AA5C
KeepAlive           0          0             0x0
GiveUp              0          0             0x0
PmtuAger            0          0             0x0
DeadWait            0          0             0x0
iss: 1865937424  snduna: 1865937425  sndnxt: 1865937426     sndwnd:      0
irs:  949507069  rcvnxt:  949507070  rcvwnd:       4128  delrcvwnd:      0
SRTT: 247 ms, RTTO: 3727 ms, RTV: 3480 ms, KRTT: 119264 ms
minRTT: 1980 ms, maxRTT: 1980 ms, ACK hold: 200 ms
Flags: passive open, higher precedence, retransmission timeout
  timing out
Datagrams (max data segment is 536 bytes):
Rcvd: 30 (out of order: 1), with data: 0, total data bytes: 0
Sent: 3 (retransmit: 66407), with data: 1, total data bytes: 1

So, what does this tell us – just some triage information.  At least before we do anything, we can actually gather information of the IP of the remote host.  Now to fix the problem.
Welcome to Transmission Control Block commands – ie tcb
What is a TCB you might ask, well I found a good explanation here:
http://puck.nether.net/lists/cisco-nsp/6073.html

       Internet Protocol Control Block Structure (inpcb{}): PCB’s are used
at the transport layer by TCP (and UDP) to hold various pieces of
information needed by TCP. They hold: TCP state information, IP address
information, port numbers, IP header prototype and options and a pointer
to the routing table entry for the destination address. PCB’s are
created for a given TCP based server when the server calls listen()

So what does this have to do with our problem of this hung session?  Well, since it is a TCP session to the router, there is an associated TCB that we can clear!
To find out the TCB information for the session, we can use the command show tcp brief

Switch1#sh tcp brief
TCB       Local Address           Foreign Address        (state)
02DC0AB0  192.168.253.254..22     192.168.0.1.51917     ESTAB
0155CBF0  192.168.253.254..23     HackerHost.hac..56667 ESTAB

Now that we have the associated TCB session – 0155CBF0 – and  we can clear it.

Switch1#clear tcp tcb 0155CBF0
[confirm]y
 [OK]
Switch1#

Now we can make sure the session is gone:

Switch1#who
    Line       User       Host(s)              Idle       Location
*  2 vty 1     jfry       idle                 00:00:00 192.168.0.1
  Interface      User        Mode                     Idle     Peer Address
Switch1#

There you go, the session is cleared and the remote connection is now gone!
If you still see the session there, give it a moment to clear – but it will go away!