0 Replies Latest reply on Aug 12, 2015 1:08 PM by yerkogofes

    OSPF adjacency loss

    yerkogofes

      Hi team,

       

      Please, your help

       

      Incident description:

       

      Our customer has a McAfee firewall, model S3008, between a Cisco VPN router and a Cisco ASA firewall. The topology is:

       

      VPN router                          McAfee S3008 FW                    ASA firewall

                     ------------------- 1-1          1-0 -------------------

      IP 192.168.13.65                   .66          .130                   IP 192.168.13.129

       

      The McAfee firewall is directly connected to the VPN router through its 1-1 interface, running at 100 Mbps, and directly connected to the ASA through its 1-0 interface, also at 100 Mbps. The three boxes are running OSPF, area 102.

      Last night, at 23:03:52, for unknown reasons, the McAfee firewall lost OSPF protocol adjacency with its neighbors, even when the interfaces seemed to be up. This adjacency was not reestablished until a kernel watchdog event, ocurred at 00:26:54. After this event, the adjacency with both boxes was reestablished.

       

      The client wants an explanation about this issue, because it disrupted its normal operation, so we have the following questions about this:

      - Is this problem hardware or software related?

      - Under what circumstances does a kernel watchdog of this type happens?

      - Why did it took so long to the watchdog to detect an anomaly?

      - Could this event happen again?

       

      Please, let us know if you require more information.

       

      These are the relevant system logs:

       

      1. Messages.log

       

      Jun 23 23:04:19 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 23 23:12:53 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 23 23:18:11 corpbanca2 sfagent[1447]: Failed to look up host 'list.smartfilter.com': error 8

      Jun 23 23:18:11 corpbanca2 sfagent[1447]: The download function returned a permanent error

      Jun 23 23:18:11 corpbanca2 sfagent[1447]: Failed to download a list

      Jun 23 23:18:11 corpbanca2 sfagent[1447]: SmartFilter was unable to download the Internet Database on corpbanca2.rbi.cl.^M ^M Reason: Invalid hostname (list.smartfilter.com) for download server^M ^M SmartFilter will retry the download in 30 minutes.  You can also try to manually download the Internet Database:  On the SmartFilter Administration Console toolbar, click the Download Internet Database button.^M ^M If this problem continues, contact Secure Computing Technical Support online at www.securecomputing.com.  To contact Technical Support directly, call +1.800.700.8328 or +1.651.628.1500.^M

      Jun 23 23:21:27 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 23 23:30:00 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 23 23:38:34 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 23 23:47:07 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 23 23:48:17 corpbanca2 sfagent[1447]: Failed to look up host 'list.smartfilter.com': error 8

      Jun 23 23:48:17 corpbanca2 sfagent[1447]: The download function returned a permanent error

      Jun 23 23:48:17 corpbanca2 sfagent[1447]: Failed to download a list

      Jun 23 23:48:17 corpbanca2 sfagent[1447]: SmartFilter was unable to download the Internet Database on corpbanca2.rbi.cl.^M ^M Reason: Invalid hostname (list.smartfilter.com) for download server^M ^M SmartFilter will retry the download in 30 minutes.  You can also try to manually download the Internet Database:  On the SmartFilter Administration Console toolbar, click the Download Internet Database button.^M ^M If this problem continues, contact Secure Computing Technical Support online at www.securecomputing.com.  To contact Technical Support directly, call +1.800.700.8328 or +1.651.628.1500.^M

      Jun 23 23:55:41 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 24 00:04:14 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 24 00:18:23 corpbanca2 sfagent[1447]: Failed to look up host 'list.smartfilter.com': error 8

      Jun 24 00:18:23 corpbanca2 sfagent[1447]: The download function returned a permanent error

      Jun 24 00:18:23 corpbanca2 sfagent[1447]: Failed to download a list

      Jun 24 00:18:23 corpbanca2 sfagent[1447]: SmartFilter was unable to download the Internet Database on corpbanca2.rbi.cl.^M ^M Reason: Invalid hostname (list.smartfilter.com) for download server^M ^M SmartFilter will retry the download in 30 minutes.  You can also try to manually download the Internet Database:  On the SmartFilter Administration Console toolbar, click the Download Internet Database button.^M ^M If this problem continues, contact Secure Computing Technical Support online at www.securecomputing.com.  To contact Technical Support directly, call +1.800.700.8328 or +1.651.628.1500.^M

      Jun 24 00:21:19 corpbanca2 ntpd[1444]: sendto(192.168.5.66): No route to host

      Jun 24 00:26:54 corpbanca2 kernel: igb0: Watchdog timeout -- resetting

      Jun 24 00:26:54 corpbanca2 kernel: igb0: Queue(0) tdh = 2990, hw tdt = 3672

      Jun 24 00:26:54 corpbanca2 kernel: igb0: TX(0) desc avail = 8,Next TX to Clean = -256

      Jun 24 00:29:32 corpbanca2 sshd[7082]: Connection closed by 163.250.240.95 [preauth]

      Jun 24 00:31:48 corpbanca2 sshd[7090]: Disconnecting: Change of username or service not allowed: (interside,ssh-connection) -> (HDDG2636,ssh-connection) [preauth]

      Jun 24 00:32:56 corpbanca2 sshd[7099]: Disconnecting: Change of username or service not allowed: (interside,ssh-connection) -> (HRRG8203,ssh-connection) [preauth]

      Jun 24 00:33:14 corpbanca2 sshd[7101]: Disconnecting: Change of username or service not allowed: (interside,ssh-connection) -> (HRRG8203,ssh-connection) [preauth]

      Jun 24 00:34:04 corpbanca2 sshd[7105]: fatal: Read from socket failed: Connection reset by peer [preauth]

      Jun 24 00:36:08 corpbanca2 login: 3 LOGIN FAILURES ON tty??

      Jun 24 00:38:18 corpbanca2 sshd[7136]: Received disconnect from 163.250.240.55: 13: The user canceled authentication.  [preauth]

      Jun 24 00:40:24 corpbanca2 login: 3 LOGIN FAILURES ON tty??

      Jun 24 00:48:23 corpbanca2 sfagent[1447]: Failed to download a list

       

       

       

       

      1. daemon.log

       

      Tue Jun 23 02:00:00 2015 corpbanca2.rbi.cl rollaudit[5669]: logfile turned over

      Jun 23 23:03:52 corpbanca2 ospfd[1468]: nsm_change_state(192.168.13.65, Full -> Init): scheduling new router-LSA origination

      Jun 23 23:03:52 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 0.0.0.0

      Jun 23 23:03:52 corpbanca2 ospfd[1468]: DR-Election[1st]: DR     192.168.13.66

      Jun 23 23:03:54 corpbanca2 ospfd[1468]: *** sendmsg in ospf_write failed to 192.168.13.129, id 0, off 0, len 80, interface 1-0, mtu 1500: Host is down

      Jun 23 23:03:54 corpbanca2 ospfd[1468]: nsm_change_state(192.168.13.162, Full -> Init): scheduling new router-LSA origination

      Jun 23 23:03:54 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 0.0.0.0

      Jun 23 23:03:54 corpbanca2 ospfd[1468]: DR-Election[1st]: DR     192.168.13.130

      Jun 23 23:18:11 corpbanca2 named[1452]: error (host unreachable) resolving 'list.smartfilter.com/A/IN': 192.168.5.66#53

      Jun 23 23:18:11 corpbanca2 named[1452]: error (host unreachable) resolving 'list.smartfilter.com/A/IN': 192.168.5.66#53

      Jun 23 23:48:17 corpbanca2 named[1452]: error (host unreachable) resolving 'list.smartfilter.com/A/IN': 192.168.5.66#53

      Jun 23 23:48:17 corpbanca2 named[1452]: error (host unreachable) resolving 'list.smartfilter.com/A/IN': 192.168.5.66#53

      Jun 24 00:18:23 corpbanca2 named[1452]: error (host unreachable) resolving 'list.smartfilter.com/A/IN': 192.168.5.66#53

      Jun 24 00:18:23 corpbanca2 named[1452]: error (host unreachable) resolving 'list.smartfilter.com/A/IN': 192.168.5.66#53

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 0.0.0.0

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.66

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: Packet[DD]: Neighbor 192.168.13.65: Initial DBD from Slave, ignoring.

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: *** sendmsg in ospf_write failed to 192.168.13.65, id 0, off 0, len 52, interface 1-1, mtu 1500: Host is down

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 192.168.13.65

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.66

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 192.168.13.65

      Jun 24 00:26:54 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.66

      Jun 24 00:26:59 corpbanca2 ospfd[1468]: Packet[DD]: Neighbor 192.168.13.65: Initial DBD from Slave, ignoring.

      Jun 24 00:26:59 corpbanca2 ospfd[1468]: Packet[DD]: Neighbor 192.168.13.65 Negotiation done (Master).

      Jun 24 00:26:59 corpbanca2 ospfd[1468]: nsm_change_state(192.168.13.65, Loading -> Full): scheduling new router-LSA origination

      Jun 24 00:27:03 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 0.0.0.0

      Jun 24 00:27:03 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.129

      Jun 24 00:27:03 corpbanca2 ospfd[1468]: DR-Election[2nd]: Backup 192.168.13.130

      Jun 24 00:27:03 corpbanca2 ospfd[1468]: DR-Election[2nd]: DR 192.168.13.129

      Jun 24 00:27:03 corpbanca2 ospfd[1468]: Packet[DD]: Neighbor 192.168.13.162 Negotiation done (Slave).

      Jun 24 00:27:03 corpbanca2 ospfd[1468]: nsm_change_state(192.168.13.162, Loading -> Full): scheduling new router-LSA origination

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 192.168.13.129

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.129

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: DR-Election[2nd]: Backup 192.168.13.129

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: DR-Election[2nd]: DR 192.168.13.129

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: interface 192.168.13.130 [1] leave AllDRouters Multicast group.

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 192.168.13.129

      Jun 24 00:27:07 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.129

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 192.168.13.130

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.129

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: DR-Election[2nd]: Backup 192.168.13.130

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: DR-Election[2nd]: DR 192.168.13.129

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: interface 192.168.13.130 [1] join AllDRouters Multicast group.

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: DR-Election[1st]: Backup 192.168.13.130

      Jun 24 00:27:17 corpbanca2 ospfd[1468]: DR-Election[1st]: DR 192.168.13.129

      Jun 24 00:47:48 corpbanca2 ospfd[1468]: Vty connection from 127.0.0.1

      Jun 24 00:48:13 corpbanca2 ospfd[1468]: Vty connection from 127.0.0.1

       

       

       

      • ospfd.conf

       

      router ospf

      ospf router-id 192.168.13.66

      network 192.168.13.64/27 area 0.0.0.102

      network 192.168.13.128/27 area 0.0.0.102

      !

       

       

      Best regards,