3 Replies Latest reply on Dec 3, 2015 9:59 AM by andy777

    Restart detection for a Linux server

    ravismallah

      Hi ,

       

      I have created a rule to detect whether any of the linux server in my datacenter is restarted. In case any such event occurs the rule should fire, for this i used normalized rule of system shut down and abnormal shutdown.

       

      This was working well when a serve which has weblogic running on it went down but recently my other server wend down an we didnt get any alarm for it. Can any one help me in suggesting any other method to detect shutting down or restart of a linux server. Is there a uniform log we receive from linux when the system goes down.

       

      Regards

       

      Ravi Mallah

        • 1. Re: Restart detection for a Linux server
          andy777

          I ran some tests across a couple of distributions and I don't see a consistent and reliable indicator in the logs that a system is going down every time. And that's when I'm intentionally shutting it down. I have no chance to get a log when there is a power outage or other external factors. I think system availability is a great use case however I think that the data source should be collected from a 3rd party device like the network monitoring tool. Do you use something like Nagios on your network that could generate up/down events for you?

          • 2. Re: Restart detection for a Linux server
            ravismallah

            Hi Andy

             

            I don't tools like Nagios , but i do have a test setup and i have tried shutting the system down with command and by turning the power off also, similar to your finding even i am unable to detect any uniform patter or service which may be used to trigger alarm for system going down.

             

            Have you ever tried the out of the box normalized rule for detection of system shutdown/Restart.

            • 3. Re: Restart detection for a Linux server
              andy777

              Thank you for that idea. You're able to go into the Policy Editor and filter the rules under both ASP and Data Source for the Normalized ID and see all of the events that are mapped to it. I don't see any events that are in the Linux rule set so we know the SIEM agrees there's not a common Linux shutdown log.

               

              shutdown-normalization.PNG

               

              As I said though, I like the use case even if we need to get a little creative with it. Some of the options to consider might be:

               

              1. Consider any boot-up logs that you see consistently. Is there anything unique enough that you would only see it on an actual boot as opposed to just a service restart? For instance, one thing I do consistently see is syslog starting back up, but I get the same messages when I restart the service so it's not definitive.

               

              2. You could use a shell alias in the login scripts:

              alias reboot="logger -n x.x.x.x $(echo "REBOOT issued for $HOSTNAME by $USER from $SSH_CLIENT");reboot"

              alias halt="logger -n x.x.x.x $(echo "HALT issued for $HOSTNAME by $USER from $SSH_CLIENT");reboot"

              alias shutdown="logger -n x.x.x.x $(echo "SHUTDOWN issued for $HOSTNAME by $USER from $SSH_CLIENT");reboot"


              3. You could use a wrapper around the binaries.

              cat /sbin/reboot

              #!/bin/bash

              logger -n x.x.x.x $(echo "REBOOT issued for $HOSTNAME by $USER from $SSH_CLIENT")

              /sbin/real-reboot

               

              Then a simple regex to parse the line might look like:

              (\w+)\sissued\sfor\s([A-Za-z0-9\-\_]+)\sby(\w+)\sfrom\s((?:\d{1,3}\x2e){3}\d{1,3})