On the receivers, if you hit the flag icon there is an option to pull a time delta report.
When I look at the output, I see that Host123 is +100 minutes for example.
At first I thought this meant I had a NTP out of sync issue, but as I check my devices I see they are synced correctly to our centralized NTP pool..
Then I investigated for a configuration issue, but the ELM/ESM is configured correctly with UTC, and my device configured with it's local time zone which is EDT.
So where is this time delta being generated from and what is it relation too? Is it even something I need to concern myself about if I have verified my NTP settings?
All time on the ESM is stored in UTC. Sometimes a device will appear to have the correct time, but have the wrong time zone. Is the Data Source also configured with EDT on the ESM? If you do a live stream or tcpdump on the Receiver, do your logs come in with the timestamp that you expect?
tcpdump -nni eth0 -X host x.x.x.x | less
I do realize that if the timezone of the device is EDT and I set it on the Receiver as CDT there will be a skew. Event if you set it to UTC, because the source is EDT it will be off. As I have been going through this exercise I thought I understood the function of the time delta report.
What got me to ask this question was there are devices in the report that are several days of time off even after validating the time zones.
The report will tell me a device is +3000 minutes off for example. In theory you can't have a live source have more than 24 hours of time difference but that would equate to 50 hours.
The only idea I can come up with is a historical log getting loaded into the Receiver.
That's a good catch.
If you find that there are specific set of devices "only" which are showing time delta difference, then first which I would prefer to check is, whether the logs are received "ALL TIME" from that specific data source or not. In case if they are not, then possibility is with the services (collection/parsing) not running as expected.
After digging around a bit I have noticed that the events from sources that are ftp'ed or similar (not WMI not Syslog) into the receiver are the ones that are farthest off in time.
They are disparate devices and do not necessarily share an operating system or other common factor that I can see.
Syslog sources have been correctly by matching time zones on source and device receiver.
Windows devices have been corrected by fixing NTP settings locally on the hosts.
One little wrinkle that can make this interesting is if different services on the data source are sending independently and using different time zones. For example, one is using UTC and the other local wall clock time. If you can identify this as the cause and cannot correct it then playing with the source IP or the port the service is sending it to can allow you to differentiate the time zones on the receiver by setting them up as different data sources.
Just suggesting this because I've run into it.