Hi,I am running search for previous month in ELM, under ELM Properties>Data section, where under the "containing the following string" I am putting Ip only "x.x.x.x" and under the Device I am selecting all the receivers,Time search limit 24 hrs, file limit 1024.After starting the search it get time out after running 24 hrs and even if I select 3-4 receivers its getting time out.
The requirement is to search for and IP address if there is any communication happened last month.Please let me know if there is any way we can get this search completed.
Hi, I split the query into weeks and it get completed in ~11 hrs - with 263 matches but this is too much delay.
The estimated raw data collection in our environment for one week is ~6 GB.
It took 11 hours to find only 263 matches?
How about you try again with same query (the IP address), but instead of selecting the whole receiver, pick only the data sources which have the events of your interest. e.g. you are looking for IP address, so I assume FW traffic logs, so just pick the firewall data sources.
Run the query and see if it finishes quicker. This way the ELM should just search through the events that's generated by the FW data sources, not others like WMI.
sssyyy , do you know know how many queries we can run in once in ELM query i.e. under ELM>properties>data
Here what I noticed - I already have 10 queries running and there are 3 more queries which are still not come in running status even after 2 hrs as there are already 10 queries running.
Is there any limit of maximum running queries in ELM ?
Yep it took 11 hrs to find only 263 matches for 1 week query.
The requirement is to reach if there is any communication happened for the given IP - so I've to search all the data sources - endpoints, domain, fw,proxy,gw etc.
We have around 9 RC's which may have 20-30% deviation in EPS..approximately 70 GB per receiver/week (Its ~600 GB/week from all receivers - apologies I mentioned wrong numbers in my last reply ) so I can try same query on each receiver/week ? let me know if that is fine and should give me the quick results.Also is it fine if I run same query for remaining receivers at the same time ? as ELM can handle multiple query at the same time - so there will be total 9 queries running at the same time. I am fine if there will be 20-30% delay in outcome.
600GB/week! If you need to search every single event for that IP address, then I don't think there is any faster way of doing it. Just need to chew through them one by one. It's going to be a long process.
I understand that search has to go through billions of events, as I mentioned I ran the same query on just single receiver ~ 70 GB / week which get completed in 4 hrs without any match.I guess ELM is designed for this kind of aggressive search.Let me know if that is not the case and there is no way we can get the faster out by running the same type of search in ELM. just an fyi - we have Splunk also for two of our customers and we ran the same search there as well on ~ 250 GB/month - the difference is like in ELM we just give the ip address i.e. x.x.x.x in search filed and in Splunk we have to type srcip=x.x.x.x or dstip=x.x.x.x but again it also has to search the billions of events but we get the output in 30 min. I am not doing any product comparison here - just want to understand if there is any better way to get the quick results.
Not sure. I have never had to search across that amount of data, but can imagine it will take a while. I heard elastic search has better performance which is available in version 10, pretty sure splunk also uses elastic search mechanism.