7 Replies Latest reply on Feb 20, 2015 9:55 AM by vagner.silva

    McAfee SIEM - ESM Data lost

    vagner.silva


      Hello guys,

       


      Recently I faced it 2 times with 2 different clients.

       

      The data disappeared after some problems and I'd like to know if you guys are facing it or have this terrible experience before.

       

      Now I have only 5 days of data:
      last 60 days, but I only have 5.JPG

       

      and for only that I have 250GB consumed so I ask, what can files are atoring the data lost? Where is it?

       

      df h.JPG

       

      Look my Data folder:

      McAfee-ENMELM-VM12 /usr/local/ess/data # ls

      ADGroup.blob                        ExtDeviceAttr.blob                NotificationEMailGroups.data   TagSevBits.index

      ADGroup.data                        ExtDeviceAttr.data                NotificationEMailGroups.index  TagUpdateException.data

      ADGroup.index                       ExtDeviceAttr.index               NotificationMembers.data       TagUpdateException.index

      ADGroupSM.data                      ExternalDevice.data               NotificationMembers.index      Theme.data

      ADGroupSM.index                     ExternalDevice.index              NotificationUser.data          Theme.index

      Access.data                         GeoLoc.data                       NotificationUser.index         ThirdPartyConfig.blob

      Access.index                        GeoLoc.index                      OS.data                        ThirdPartyConfig.data

      Action.data                         GetRedundantSettings.sql          OS.index                       ThirdPartyConfig.index

      Action.index                        Groups.blob                       Obfuscation.blob               ThirdPartyType.data

      AggException.data                   Groups.data                       Obfuscation.data               ThirdPartyType.index

      AggException.index                  Groups.index                      Obfuscation.index              Timezone.blob

      Alert1.blob_p35                     HCFilters.blob                    PluginData.blob                Timezone.data

      Alert1.data_p35                     HCFilters.data                    PluginData.data                Timezone.index

      Alert_AlertID_1.index_p35           HCFilters.index                   PluginData.index               TriggeredAlarm.blob

      Alert_DstIP_1.index_p35             HealthStatusChanges.data          Plugins.data                   TriggeredAlarm.data

      Alert_DstMac_1.index_p35            Hosts.data                        Plugins.index                  TriggeredAlarm.index

      Alert_DstPort_1.index_p35           Hosts.index                       PortApps.blob                  TriggeredCondition.data

      Alert_GUID1_1.index_p35             ICMPType.data                     PortApps.data                  TriggeredCondition.index

      Alert_GUID2_1.index_p35             ICMPType.index                    PortApps.index                 UCFA2U.data

      Alert_ID_1.index_p35                IPS.blob                          Ports.data                     UCFA2U.index

      Alert_SigIDDstIP_1.index_p35        IPS.data                          Ports.index                    UCFC2U.data

      Alert_SigIDSrcIP_1.index_p35        IPS.index                         Preprocess.blob                UCFC2U.index

      Alert_SigID_1.index_p35             IPSBlob.blob                      Preprocess.data                UCFN2U.data

      Alert_SrcIP_1.index_p35             IPSBlob.data                      Preprocess.index               UCFN2U.index

      Alert_StaticStrings1.bloom_p35      IPSBlob.index                     PreprocessException.data       UCFName.blob

      Asset.data                          IPSChange.data                    PreprocessException.index      UCFName.data

      Asset.index                         IPSChange.index                   PreprocessGroup.blob           UCFName.index

      AssetGroup.data                     IPSCheck.data                     PreprocessGroup.data           US.data

      AssetGroup.index                    IPSCheck.index                    PreprocessGroup.index          US.index

      AssetGroupXRef.data                 ItemRights.data                   Profile.data                   UpdateBlob.blob

      AssetGroupXRef.index                ItemRights.index                  Profile.index                  UpdateBlob.data

      AssetVendor.data                    Job.data                          Query.blob                     UpdateBlob.index

      AssetVendor.index                   Job.index                         Query.data                     Usage.data

      AssetVulnerability.data             LocaleString.blob                 Query.index                    Usage.index

      AssetVulnerability.index            LocaleString.data                 RemoteAction.data              UserField.data

      AutoCreateRule.data                 LocaleString.index                RemoteAction.index             UserField.index

      AutoCreateRule.index                LocaleString_StrValue.bloom       RemoteActionAttr.data          UserFieldUse.data

      AutoCreateRuleCriteria.data         Log.blob_p2                       RemoteActionAttr.index         UserFieldUse.index

      AutoCreateRuleCriteria.index        Log.data_p2                       RemoteCommandAttr.blob         UserFilterList.data

      Blacklist.data                      Log.index_p2                      ReportComponent.blob           UserFilterList.index

      Blacklist.index                     LogCategory.data                  ReportComponent.data           UserIPSIDJoin.data

      BlacklistBuffer.data                LogCategory.index                 ReportComponent.index          UserIPSIDJoin.index

      BlacklistBuffer.index               MessageTemplate.blob              ReportFolder.data              UserLicense.data

      CaseEvents.data                     MessageTemplate.data              ReportFolder.index             UserLicense.index

      CaseEvents.index                    MessageTemplate.index             Reports.blob                   UserViewExclusion.data

      CaseMgt.blob                        NDDevice.data                     Reports.data                   UserViewExclusion.index

      CaseMgt.data                        NDDevice.index                    Reports.index                  User_IPS.data

      CaseMgt.index                       NDDeviceAddresses.data            Rights.blob                    User_IPS.index

      CaseMgt_Name.bloom                  NDDeviceAddresses.index           Rights.data                    Users.blob

      CaseMgt_Notes.bloom                 NDDeviceInterface.data            Rights.index                   Users.data

      CaseMgt_Viewed.bloom                NDDeviceInterface.index           RightsAssignment.data          Users.index

      CaseOrg.data                        NDDeviceVLAN.data                 RightsAssignment.index         UsersPW.data

      CaseOrg.index                       NDDeviceVLAN.index                Rule.blob                      UsersPW.index

      CaseStatus.data                     NDEPDevices.data                  Rule.data                      Var.blob

      CaseStatus.index                    NDEPDevices.index                 Rule.index                     Var.data

      ChangeLog.blob                      NDEPParams.data                   RuleParam.blob                 Var.index

      ChangeLog.data                      NDEPParams.index                  RuleParam.data                 VarException.blob

      ChangeLog.index                     NDEndPointIP.data                 RuleParam.index                VarException.data

      Class.blob                          NDEndPointIP.index                RuleParamChange.blob           VarException.index

      Class.data                          NDEndPointIPHistory.data          RuleParamChange.data           View.blob

      Class.index                         NDEndPointIPHistory.index         RuleParamChange.index          View.data

      Condition.blob                      NDEndPoints.data                  RuleUseException.data          View.index

      Condition.data                      NDEndPoints.index                 RuleUseException.index         ViewComponent.blob

      Condition.index                     NDEndPointsHistory.data           RuleVA.data                    ViewComponent.data

      Connection1.blob_p1                 NDEndPointsHistory.index          RuleVA.index                   ViewComponent.index

      Connection1.data_p1                 NDFolder.blob                     RuleVIN.data                   ViewFolder.data

      Connection_ConnectionID_1.index_p1  NDFolder.data                     RuleVIN.index                  ViewFolder.index

      Connection_DstIPDur_1.index_p1      NDFolder.index                    SMXRef.data                    Vulnerability.data

      Connection_DstPort_1.index_p1       NDFolderDevice.data               SMXRef.index                   Vulnerability.index

      Connection_ID_1.index_p1            NDFolderDevice.index              Scoring.blob                   WMIType.data

      Connection_LocIDDst_1.index_p1      NDIPLoc.data                      Scoring.data                   WMIType.index

      Connection_Prot_1.index_p1          NDIPLoc.index                     Scoring.index                  WatchListValues1.data

      Connection_SrcIPDur_1.index_p1      NDNeighbors.data                  ScoringSource.blob             WatchListValues1.index

      Connection_SrcPort_1.index_p1       NDNeighbors.index                 ScoringSource.data             WatchLists.blob

      Connection_StaticStrings1.bloom_p1  NDParams.data                     ScoringSource.index            WatchLists.data

      Connection_User16_1.index_p1        NDParams.index                    SelectFieldName.data           WatchLists.index

      DataEnrichment.blob                 NDParamsDetail.data               SelectFieldName.index          Zone.data

      DataEnrichment.data                 NDParamsDetail.index              SendEMail.blob                 Zone.index

      DataEnrichment.index                NDParamsExclusion.data            SendEMail.data                 ZoneIPMap.data

      DataEnrichmentFields.data           NDParamsExclusion.index           SendEMail.index                ZoneIPMap.index

      DataEnrichmentFields.index          NDPortControl.data                SendSyslog.blob                connect_esm.sql

      DataEnrichmentIPSID.data            NDPortControl.index               SendSyslog.data                finalpartitionlist.sql

      DataEnrichmentIPSID.index           NDProcess.data                    SendSyslog.index               ngcp.cfd

      DeviceFolder.blob                   NDProcess.index                   StringMap1.data                ngcp.cfd_old

      DeviceFolder.data                   NDSearchResults.data              StringMap1.index               ngcp.cfg

      DeviceFolder.index                  NDSearchResults.index             StringMap_Name1.bloom          ngcp.cfg_old

      DeviceFolderIPSJoin.data            NitroError.Log                    SysSettings.blob               ngcp.cpy

      DeviceFolderIPSJoin.index           Notes.blob                        SysSettings.data               ngcp.cpy_old

      DistributedESM.data                 Notes.data                        SysSettings.index              ngcp.dfl

      DistributedESM.index                Notes.index                       TPTypeApplication.data         ngcp.dfl1407848751

      EMail.data                          Notification.blob                 TPTypeApplication.index        ngcp.dfl1410804574

      EMail.index                         Notification.data                 Tag.data                       ngcp.dfl_1399392877

      EMailGroup.data                     Notification.index                Tag.index                      ngcp.old

      EMailGroup.index                    NotificationAction.data           TagAsset.data                  old_sa/

      EMailGroupEMailAddress.data         NotificationAction.index          TagAsset.index                 packet1.blob_p1

      EMailGroupEMailAddress.index        NotificationActionAttr.blob       TagAssetException.data         packet1.blob_p2

      ESMFilters.blob                     NotificationActionAttr.data       TagAssetException.index        packet1.data_p1

      ESMFilters.data                     NotificationActionAttr.index      TagAssetGroup.data             packet1.data_p2

      ESMFilters.index                    NotificationCheck.data            TagAssetGroup.index            packet1.index_p1

      EventForwarding.blob                NotificationCheck.index           TagRule.data                   packet1.index_p2

      EventForwarding.data                NotificationEMailAddresses.data   TagRule.index                  partitionlist.sql

      EventForwarding.index               NotificationEMailAddresses.index  TagSevBits.data

       

      I noticed many *old files ... but none seems the database lost.

       

      In attachment some logs.

       

      I appreciate your on that, I don't want to face it for the thrid time hehe. Tks

        • 1. Re: McAfee SIEM - ESM Data lost
          aszotek

          check for detached database partitions:

           

          nquery -d '/usr/local/ess/data/ngcp.dfl|:::1|1111' -q 'show partitions from alert'

          • 2. Re: McAfee SIEM - ESM Data lost
            vagner.silva

            This is what I get:

            McAfee-ENMELM-VM12 ~ # nquery -d '/usr/local/ess/data/ngcp.dfl|:::1|1111' -q 'show partitions from alert'

            executing queryToRun: [show partitions from alert]

            queryToRun 100% complete, 00:00.000 elapsed

            SHOW PARTITIONS

            Partition     35 | 02/14/2015 00:00:00.000 to 02/19/2015 23:59:59.999 |attached  |      18,000,259 recs|v193956454654519|mod 02/19/2015 16:58:47|open

             

            Seems everthing is attached?

            • 3. Re: McAfee SIEM - ESM Data lost
              vagner.silva

              Sorry, now I see. Only partitions from 02/14/2015 to 02/19/2015 are attached. How can I see old partitions and add them?

              • 4. Re: McAfee SIEM - ESM Data lost
                aszotek

                You don't have any detached partitions, I'm afraid.

                They may be elsewhere, have you checked backups?

                Also, as you mentioned "recreation of data source" in another thread, deleting old data source, wipes all the data received from this data source from ESM database.

                • 5. Re: McAfee SIEM - ESM Data lost
                  vagner.silva


                  Unfortunately I don't have any events backup but there's no problem to recover the data since those partitions are not there. My problem now it's these amount of data stored in somewhere that I cannot see and taking up space in disk .

                  • 6. Re: McAfee SIEM - ESM Data lost
                    aszotek

                    'du' Linux command is your friend to pinpoint directories consuming precious disk space.

                    From the files that you listed above, check their filesizes.

                    • 7. Re: McAfee SIEM - ESM Data lost
                      vagner.silva

                      Thanks aszotek, seems the problem is not only in the ./usr/local/ess/data folder: I have a lot of files as well in /var/log .

                       

                      102052  ./usr/lib/locale

                      104604  ./tmp

                      112756  ./usr/share

                      125164  ./usr/lib/perl5

                      130536  ./var/log/shm/collector/p/rpcclient

                      142940  ./var/log/shm/collector/p

                      142960  ./var/log/shm/collector

                      144252  ./var/log/shm

                      147508  ./var/www/html/help

                      153148  ./var/log/data/inline/tmp

                      153336  ./etc

                      160080  ./root/update_db_backups

                      161304  ./root

                      184204  ./var/www/html

                      185496  ./var/www

                      188412  ./usr/java/jre1.6.0_26-i586/lib

                      189764  ./usr/java/jre1.6.0_26-i586

                      217924  ./usr/local/bin

                      280980  ./usr/java

                      306228  ./var/log/data/inline/thirdparty.logs

                      516144  ./usr/lib64

                      881764  ./usr/local/ess/dbbackup

                      907020  ./usr/lib

                      1084484 ./var/log/data/autodisc/syslog-syslog/input

                      1084620 ./var/log/data/autodisc/syslog-syslog

                      1084700 ./var/log/data/autodisc

                      1196088 ./usr/local/ess/update/archive

                      1249924 ./usr/local/ess/update/updates

                      2446020 ./usr/local/ess/update

                      3177668 ./var/log/httpd

                      5074604 ./usr/local/ace/incoming

                      5157552 ./usr/local/ace

                      77777600        ./usr/local/ess/data

                      81431336        ./usr/local/ess

                      87070864        ./usr/local

                      89045636        ./usr

                      157540960       ./var/log/data/inline

                      158625676       ./var/log/data

                      162400212       ./var/log

                      162608528       ./var

                      252209780       .

                      McAfee-ENMELM-VM12 / #

                       

                      I have a lot of log folders storing a huge size. I'll check with McAfee support if I can delete something inside of it .