1 2 Previous Next 10 Replies Latest reply on Aug 14, 2017 3:24 AM by minki

    ELM search

    minki

      Hi,I am running search for previous month in ELM, under ELM Properties>Data section, where under the "containing the following string" I am putting Ip only "x.x.x.x" and under the Device I am selecting all the receivers,Time search limit 24 hrs, file limit 1024.After starting the search it get time out after running 24 hrs and even if I select 3-4 receivers its getting time out.

      The requirement is to search for and IP address if there is any communication happened last month.Please let me know if there is any way we can get this search completed.

        • 1. Re: ELM search
          sssyyy

          If there are actually that large number of matching results, I would think you should split the search query into individual weeks.

          • 2. Re: ELM search
            minki

            Hi, I split the query into weeks and it get completed in ~11 hrs - with 263 matches but this is too much delay.

            The estimated raw data collection in our environment for one week is ~6 GB.

            • 3. Re: ELM search
              sssyyy

              It took 11 hours to find only 263 matches?

               

              How about you try again with same query (the IP address), but instead of selecting the whole receiver, pick only the data sources which have the events of your interest. e.g. you are looking for IP address, so I assume FW traffic logs, so just pick the firewall data sources.

               

              Run the query and see if it finishes quicker. This way the ELM should just search through the events that's generated by the FW data sources, not others like WMI.

              • 4. Re: ELM search
                minki

                Yep it took 11 hrs to find only 263 matches for 1 week query.

                The requirement is to reach if there is any communication happened for the given IP - so I've to search all the data sources - endpoints, domain, fw,proxy,gw etc.

                We have around 9 RC's which may have 20-30% deviation in EPS..approximately 70 GB per receiver/week (Its ~600 GB/week from all receivers - apologies I mentioned wrong numbers in my last reply ) so I can try same query on each receiver/week ? let me know if that is fine and should give me the quick results.Also is it fine if I run same query for remaining receivers at the same time ? as ELM can handle multiple query at the same time - so there will be total 9 queries running at the same time. I am fine if there will be 20-30% delay in outcome.

                • 5. Re: ELM search
                  minki

                  Hi, I run the same query on single receiver and it again took around 4 hrs with zero match.

                  • 6. Re: ELM search
                    sssyyy

                    600GB/week! If you need to search every single event for that IP address, then I don't think there is any faster way of doing it. Just need to chew through them one by one. It's going to be a long process.

                    • 7. Re: ELM search
                      minki

                      I understand that search has to go through billions of events, as I mentioned I ran the same query on just single receiver ~ 70 GB / week  which get completed in 4 hrs without any match.I guess ELM is designed for this kind of aggressive search.Let me know if that is not the case and there is no way we can get the faster out by running the same type of search in ELM. just an fyi - we have Splunk also for two of our customers and we ran the same search there as well on ~ 250 GB/month - the difference is like in ELM we just give the ip address i.e. x.x.x.x in search filed and in Splunk we have to type srcip=x.x.x.x or dstip=x.x.x.x but again it also has to search the billions of events but we get the output in 30 min. I am not doing any product comparison here - just want to understand if there is any better way to get the quick results.

                      • 8. Re: ELM search
                        sssyyy

                        Not sure. I have never had to search across that amount of data, but can imagine it will take a while. I heard elastic search has better performance which is available in version 10, pretty sure splunk also uses elastic search mechanism.

                        • 9. Re: ELM search
                          minki

                          I guess ESL and Splunk both use Map reduce algo to search.There is not much much information available on ESL so far

                          For ESL I've another query in community but that is no answer yet -- ESM 10.x ELS features and installation recommendations

                          It will be help full if you can share some document related to ESL in case you manage to get it from samewhere. In PG 10.x and youtube video its very limited information.

                          1 2 Previous Next