1 2 Previous Next 16 Replies Latest reply on Jan 20, 2016 3:12 AM by tiwake

    Last Time Values more than one hour in the future

    pepelepuu

      I have been getting LOTS and LOTS of alerts stating 'Last Time' values more than one hour in the future.

      HEre are the steps taken so far and other specific information:

      • ALL devices(ESM & ELM) and data sources in the environment are configured for GMT, as show below in the screen shot.
      • Runinng version 9.3.2 20140408
      • Restarted the ELM, ESM and the sample datasource
      • Rebuild Tables

       

       

      log.jpg

       

      datasource timezone.jpg

        • 1. Re: Last Time Values more than one hour in the future
          pepelepuu

          Oh btw, I also opened a ticket with McAfee Platinum support..no luck there

          • 2. Re: Last Time Values more than one hour in the future
            Scott Taschler

            Most often this means that you have misconfigured the time zone in the configurtion of the data source.  Your SIEM devices (ESM, REC, ELM etc.) should all have their local system time set to GMT (which it sounds like you've done correctly).  The data sources should be configured with the time zone of the logs that are being received.  If the timestamp in the Cisco ASA logs is in Eastern US time, then that's what you should have in the data source config above. 

             

            Scott

            • 3. Re: Last Time Values more than one hour in the future
              penoffd

              Same issue as I posted previously.  Time settings are correct for both of the devices (Cisco ASA and Imprivata appliance) and match the setting of the ELM.  I get consistent errors in the ELM logs every five minutes for both devices.

              • 4. Re: Last Time Values more than one hour in the future
                pepelepuu

                @Scott

                Yes, I have confirmed that ALL the devices are configured for GMT. Coincidentally, all of the devices are in the same rackspace as well.

                I did a show clock to confirm, when I first noticed the issue.

                a# sh clock

                15:41:12.067 UTC Fri Apr 25 2014

                ra#

                ra#

                ra# sh ntp status

                Clock is synchronized, stratum 2, reference is 10.40.7.30

                nominal freq is 99.9984 Hz, actual freq is 99.9958 Hz, precision is 2**6

                reference time is d704fd72.b3e79c36 (15:27:46.702 UTC Fri Apr 25 2014)

                clock offset is -2.6682 msec, root delay is 5.28 msec

                root dispersion is 21.97 msec, peer dispersion is 18.91 msec

                 

                Also worth noting, all device use the same NTP servers as well

                 

                Message was edited by: pepelepuu on 5/7/14 1:44:53 PM CDT
                • 5. Re: Last Time Values more than one hour in the future
                  Scott Taschler

                  As a next troubleshooting step, I would look hard for these mysterious logs from the future.  One way to do this is to select a view with a time-distribution panel (for example, Event Views/Distribution).  Set your time filter for "Today", or perhaps "This Week".  It's important that you don't select "Last 24 hours" or similar timeframe.  This shows events from the most recent past, but you want to see the events that are inserted in the future.

                   

                  You should see a sharp drop off in the events at the current time.  If you zoom in on the Y axis, you should be able to see if there are any logs in the database with future timestamps. Drilling into these logs should allow you to get a handle on the problem.

                   

                  One other point: in your message above, I note that your device has a time of 15:41 UTC, and the NTP server has a time of 15:27 UTC.  Perhaps there was a time delay between when you executed the "sh clock" and "sh ntp status" commands.  If not, then there is something very odd indeed with time settings on your network.

                   

                  Scott

                  1 of 1 people found this helpful
                  • 6. Re: Last Time Values more than one hour in the future
                    penoffd

                    Scott,

                     

                    I used the method you described to identify the "future" log events.  In my case they are originating from a Cisco VPN.

                     

                    When I look at the details, the "first time" and "last time" values are in the future, typically three hours ahead of the ELM time.  The packet contents look like this:

                     

                     

                    <164>May 08 2014 11:01:06: %ASA-4-722051: Group <ANYCONNECT> User <greenpa> IP <166.142.254.87> IPv4 Address <172.19.1.114> IPv6 address <::> assigned to session

                     

                     

                    The time listed in the packet is correct. I can only assume that these are some sort of log event that the ELM can't interpret, but why it would "bump" the event's time ahead by three hours seems strange.

                     

                    Thanks,

                     

                    Dan

                    • 7. Re: Last Time Values more than one hour in the future
                      Scott Taschler

                      It's not entirely clear to me that we are talking about things in the same way.  I will try to be very explicit; sorry if it comes across as basic.

                       

                      ELM: Enterprise log manager.  The ELM manages the store of raw logs.  It does not interpret or change these logs in any way.  The ELM system time must always be GMT.  The logs will have timestamps unchanged from how the device sent them.  It is not at all uncommon for the logs to come in with timestamps that are in a different time zone from the ELM. 

                       

                      ESM: Enterprise Security Manager.  The ESM manages the database of parsed logs.  The ESM *does* deal in parsed, interpreted logs.  The logs that are stored in the ESM are converted from the local time reported in the log (and reflected in the data source config) to GMT.  All logs stored in the ESM database are normalized to GMT.  When they are extracted from the database and displayed for the user, the "First Time" and "Last Time" fields are then displayed in the time zone that the user has selected as their preferred time zone (via the Options menu in the top-right corner)

                       

                      In the packet above (which represents exactly what we got from the VPN), we see a timestamp of 11:01am.  What time (in GMT) was it when that log was generated/received by the Receiver (and ultimately, the ESM?) 

                       

                      If you've confirmed that

                      a) the logs are coming in with timestamps that are definitely reporting time in GMT

                      b) the data source configuration shows that the data source is properly configured for GMT

                      c) The First time and Last time are displaying as hours in the future.

                       

                      Then something very unexpected is happening.  At this point it would probably be best to give support a call and get some deeper troubleshooting assistance.

                       

                      Scott

                      1 of 1 people found this helpful
                      • 8. Re: Last Time Values more than one hour in the future
                        pepelepuu

                        Scott,

                        Interestingly enough, the problematic datasource for me is also Cisco VPN concentrator. I have confirmed numerous times:

                         

                        a) the logs are coming in with timestamps that are definitely reporting time in GMT

                        b) the data source configuration shows that the data source is properly configured for GMT

                        c) The First time and Last time are displaying as hours in the future.

                         

                        I've had McAfee technician confirm that as well. I already opened a ticket with McAfee Platinum with no results. Which why I'm on here now. I just got to frustrated.

                        SR 4-5731085426

                        • 9. Re: Last Time Values more than one hour in the future
                          penoffd

                          Exactly as we are experiencing as well.

                           

                          I have a G2M set up with our local SE, and will report back our findings later today.

                          1 of 1 people found this helpful
                          1 2 Previous Next