cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted

McAfee SIEM - ESM Data lost

Jump to solution


Hello guys,


Recently I faced it 2 times with 2 different clients.

The data disappeared after some problems and I'd like to know if you guys are facing it or have this terrible experience before.

Now I have only 5 days of data:
last 60 days, but I only have 5.JPG

and for only that I have 250GB consumed so I ask, what can files are atoring the data lost? Where is it?

df h.JPG

Look my Data folder:

McAfee-ENMELM-VM12 /usr/local/ess/data # ls

ADGroup.blob                        ExtDeviceAttr.blob                NotificationEMailGroups.data   TagSevBits.index

ADGroup.data                        ExtDeviceAttr.data                NotificationEMailGroups.index  TagUpdateException.data

ADGroup.index                       ExtDeviceAttr.index               NotificationMembers.data       TagUpdateException.index

ADGroupSM.data                      ExternalDevice.data               NotificationMembers.index      Theme.data

ADGroupSM.index                     ExternalDevice.index              NotificationUser.data          Theme.index

Access.data                         GeoLoc.data                       NotificationUser.index         ThirdPartyConfig.blob

Access.index                        GeoLoc.index                      OS.data                        ThirdPartyConfig.data

Action.data                         GetRedundantSettings.sql          OS.index                       ThirdPartyConfig.index

Action.index                        Groups.blob                       Obfuscation.blob               ThirdPartyType.data

AggException.data                   Groups.data                       Obfuscation.data               ThirdPartyType.index

AggException.index                  Groups.index                      Obfuscation.index              Timezone.blob

Alert1.blob_p35                     HCFilters.blob                    PluginData.blob                Timezone.data

Alert1.data_p35                     HCFilters.data                    PluginData.data                Timezone.index

Alert_AlertID_1.index_p35           HCFilters.index                   PluginData.index               TriggeredAlarm.blob

Alert_DstIP_1.index_p35             HealthStatusChanges.data          Plugins.data                   TriggeredAlarm.data

Alert_DstMac_1.index_p35            Hosts.data                        Plugins.index                  TriggeredAlarm.index

Alert_DstPort_1.index_p35           Hosts.index                       PortApps.blob                  TriggeredCondition.data

Alert_GUID1_1.index_p35             ICMPType.data                     PortApps.data                  TriggeredCondition.index

Alert_GUID2_1.index_p35             ICMPType.index                    PortApps.index                 UCFA2U.data

Alert_ID_1.index_p35                IPS.blob                          Ports.data                     UCFA2U.index

Alert_SigIDDstIP_1.index_p35        IPS.data                          Ports.index                    UCFC2U.data

Alert_SigIDSrcIP_1.index_p35        IPS.index                         Preprocess.blob                UCFC2U.index

Alert_SigID_1.index_p35             IPSBlob.blob                      Preprocess.data                UCFN2U.data

Alert_SrcIP_1.index_p35             IPSBlob.data                      Preprocess.index               UCFN2U.index

Alert_StaticStrings1.bloom_p35      IPSBlob.index                     PreprocessException.data       UCFName.blob

Asset.data                          IPSChange.data                    PreprocessException.index      UCFName.data

Asset.index                         IPSChange.index                   PreprocessGroup.blob           UCFName.index

AssetGroup.data                     IPSCheck.data                     PreprocessGroup.data           US.data

AssetGroup.index                    IPSCheck.index                    PreprocessGroup.index          US.index

AssetGroupXRef.data                 ItemRights.data                   Profile.data                   UpdateBlob.blob

AssetGroupXRef.index                ItemRights.index                  Profile.index                  UpdateBlob.data

AssetVendor.data                    Job.data                          Query.blob                     UpdateBlob.index

AssetVendor.index                   Job.index                         Query.data                     Usage.data

AssetVulnerability.data             LocaleString.blob                 Query.index                    Usage.index

AssetVulnerability.index            LocaleString.data                 RemoteAction.data              UserField.data

AutoCreateRule.data                 LocaleString.index                RemoteAction.index             UserField.index

AutoCreateRule.index                LocaleString_StrValue.bloom       RemoteActionAttr.data          UserFieldUse.data

AutoCreateRuleCriteria.data         Log.blob_p2                       RemoteActionAttr.index         UserFieldUse.index

AutoCreateRuleCriteria.index        Log.data_p2                       RemoteCommandAttr.blob         UserFilterList.data

Blacklist.data                      Log.index_p2                      ReportComponent.blob           UserFilterList.index

Blacklist.index                     LogCategory.data                  ReportComponent.data           UserIPSIDJoin.data

BlacklistBuffer.data                LogCategory.index                 ReportComponent.index          UserIPSIDJoin.index

BlacklistBuffer.index               MessageTemplate.blob              ReportFolder.data              UserLicense.data

CaseEvents.data                     MessageTemplate.data              ReportFolder.index             UserLicense.index

CaseEvents.index                    MessageTemplate.index             Reports.blob                   UserViewExclusion.data

CaseMgt.blob                        NDDevice.data                     Reports.data                   UserViewExclusion.index

CaseMgt.data                        NDDevice.index                    Reports.index                  User_IPS.data

CaseMgt.index                       NDDeviceAddresses.data            Rights.blob                    User_IPS.index

CaseMgt_Name.bloom                  NDDeviceAddresses.index           Rights.data                    Users.blob

CaseMgt_Notes.bloom                 NDDeviceInterface.data            Rights.index                   Users.data

CaseMgt_Viewed.bloom                NDDeviceInterface.index           RightsAssignment.data          Users.index

CaseOrg.data                        NDDeviceVLAN.data                 RightsAssignment.index         UsersPW.data

CaseOrg.index                       NDDeviceVLAN.index                Rule.blob                      UsersPW.index

CaseStatus.data                     NDEPDevices.data                  Rule.data                      Var.blob

CaseStatus.index                    NDEPDevices.index                 Rule.index                     Var.data

ChangeLog.blob                      NDEPParams.data                   RuleParam.blob                 Var.index

ChangeLog.data                      NDEPParams.index                  RuleParam.data                 VarException.blob

ChangeLog.index                     NDEndPointIP.data                 RuleParam.index                VarException.data

Class.blob                          NDEndPointIP.index                RuleParamChange.blob           VarException.index

Class.data                          NDEndPointIPHistory.data          RuleParamChange.data           View.blob

Class.index                         NDEndPointIPHistory.index         RuleParamChange.index          View.data

Condition.blob                      NDEndPoints.data                  RuleUseException.data          View.index

Condition.data                      NDEndPoints.index                 RuleUseException.index         ViewComponent.blob

Condition.index                     NDEndPointsHistory.data           RuleVA.data                    ViewComponent.data

Connection1.blob_p1                 NDEndPointsHistory.index          RuleVA.index                   ViewComponent.index

Connection1.data_p1                 NDFolder.blob                     RuleVIN.data                   ViewFolder.data

Connection_ConnectionID_1.index_p1  NDFolder.data                     RuleVIN.index                  ViewFolder.index

Connection_DstIPDur_1.index_p1      NDFolder.index                    SMXRef.data                    Vulnerability.data

Connection_DstPort_1.index_p1       NDFolderDevice.data               SMXRef.index                   Vulnerability.index

Connection_ID_1.index_p1            NDFolderDevice.index              Scoring.blob                   WMIType.data

Connection_LocIDDst_1.index_p1      NDIPLoc.data                      Scoring.data                   WMIType.index

Connection_Prot_1.index_p1          NDIPLoc.index                     Scoring.index                  WatchListValues1.data

Connection_SrcIPDur_1.index_p1      NDNeighbors.data                  ScoringSource.blob             WatchListValues1.index

Connection_SrcPort_1.index_p1       NDNeighbors.index                 ScoringSource.data             WatchLists.blob

Connection_StaticStrings1.bloom_p1  NDParams.data                     ScoringSource.index            WatchLists.data

Connection_User16_1.index_p1        NDParams.index                    SelectFieldName.data           WatchLists.index

DataEnrichment.blob                 NDParamsDetail.data               SelectFieldName.index          Zone.data

DataEnrichment.data                 NDParamsDetail.index              SendEMail.blob                 Zone.index

DataEnrichment.index                NDParamsExclusion.data            SendEMail.data                 ZoneIPMap.data

DataEnrichmentFields.data           NDParamsExclusion.index           SendEMail.index                ZoneIPMap.index

DataEnrichmentFields.index          NDPortControl.data                SendSyslog.blob                connect_esm.sql

DataEnrichmentIPSID.data            NDPortControl.index               SendSyslog.data                finalpartitionlist.sql

DataEnrichmentIPSID.index           NDProcess.data                    SendSyslog.index               ngcp.cfd

DeviceFolder.blob                   NDProcess.index                   StringMap1.data                ngcp.cfd_old

DeviceFolder.data                   NDSearchResults.data              StringMap1.index               ngcp.cfg

DeviceFolder.index                  NDSearchResults.index             StringMap_Name1.bloom          ngcp.cfg_old

DeviceFolderIPSJoin.data            NitroError.Log                    SysSettings.blob               ngcp.cpy

DeviceFolderIPSJoin.index           Notes.blob                        SysSettings.data               ngcp.cpy_old

DistributedESM.data                 Notes.data                        SysSettings.index              ngcp.dfl

DistributedESM.index                Notes.index                       TPTypeApplication.data         ngcp.dfl1407848751

EMail.data                          Notification.blob                 TPTypeApplication.index        ngcp.dfl1410804574

EMail.index                         Notification.data                 Tag.data                       ngcp.dfl_1399392877

EMailGroup.data                     Notification.index                Tag.index                      ngcp.old

EMailGroup.index                    NotificationAction.data           TagAsset.data                  old_sa/

EMailGroupEMailAddress.data         NotificationAction.index          TagAsset.index                 packet1.blob_p1

EMailGroupEMailAddress.index        NotificationActionAttr.blob       TagAssetException.data         packet1.blob_p2

ESMFilters.blob                     NotificationActionAttr.data       TagAssetException.index        packet1.data_p1

ESMFilters.data                     NotificationActionAttr.index      TagAssetGroup.data             packet1.data_p2

ESMFilters.index                    NotificationCheck.data            TagAssetGroup.index            packet1.index_p1

EventForwarding.blob                NotificationCheck.index           TagRule.data                   packet1.index_p2

EventForwarding.data                NotificationEMailAddresses.data   TagRule.index                  partitionlist.sql

EventForwarding.index               NotificationEMailAddresses.index  TagSevBits.data

I noticed many *old files ... but none seems the database lost.

In attachment some logs.

I appreciate your on that, I don't want to face it for the thrid time hehe. Tks

1 Solution

Accepted Solutions
Highlighted
Level 10
Report Inappropriate Content
Message 7 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

'du' Linux command is your friend to pinpoint directories consuming precious disk space.

From the files that you listed above, check their filesizes.

View solution in original post

8 Replies
Highlighted
Level 10
Report Inappropriate Content
Message 2 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

check for detached database partitions:

nquery -d '/usr/local/ess/data/ngcp.dfl|:::1|1111' -q 'show partitions from alert'

Highlighted

Re: McAfee SIEM - ESM Data lost

Jump to solution

This is what I get:

McAfee-ENMELM-VM12 ~ # nquery -d '/usr/local/ess/data/ngcp.dfl|:::1|1111' -q 'show partitions from alert'

executing queryToRun: [show partitions from alert]

queryToRun 100% complete, 00:00.000 elapsed

SHOW PARTITIONS

Partition     35 | 02/14/2015 00:00:00.000 to 02/19/2015 23:59:59.999 |attached  |      18,000,259 recs|v193956454654519|mod 02/19/2015 16:58:47|open

Seems everthing is attached?

Re: McAfee SIEM - ESM Data lost

Jump to solution

Sorry, now I see. Only partitions from 02/14/2015 to 02/19/2015 are attached. How can I see old partitions and add them?

Highlighted
Level 10
Report Inappropriate Content
Message 5 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

You don't have any detached partitions, I'm afraid.

They may be elsewhere, have you checked backups?

Also, as you mentioned "recreation of data source" in another thread, deleting old data source, wipes all the data received from this data source from ESM database.

Highlighted

Re: McAfee SIEM - ESM Data lost

Jump to solution


Unfortunately I don't have any events backup but there's no problem to recover the data since those partitions are not there. My problem now it's these amount of data stored in somewhere that I cannot see and taking up space in disk .

Highlighted
Level 10
Report Inappropriate Content
Message 7 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

'du' Linux command is your friend to pinpoint directories consuming precious disk space.

From the files that you listed above, check their filesizes.

View solution in original post

Highlighted

Re: McAfee SIEM - ESM Data lost

Jump to solution

Thanks aszotek, seems the problem is not only in the ./usr/local/ess/data folder: I have a lot of files as well in /var/log .

102052  ./usr/lib/locale

104604  ./tmp

112756  ./usr/share

125164  ./usr/lib/perl5

130536  ./var/log/shm/collector/p/rpcclient

142940  ./var/log/shm/collector/p

142960  ./var/log/shm/collector

144252  ./var/log/shm

147508  ./var/www/html/help

153148  ./var/log/data/inline/tmp

153336  ./etc

160080  ./root/update_db_backups

161304  ./root

184204  ./var/www/html

185496  ./var/www

188412  ./usr/java/jre1.6.0_26-i586/lib

189764  ./usr/java/jre1.6.0_26-i586

217924  ./usr/local/bin

280980  ./usr/java

306228  ./var/log/data/inline/thirdparty.logs

516144  ./usr/lib64

881764  ./usr/local/ess/dbbackup

907020  ./usr/lib

1084484 ./var/log/data/autodisc/syslog-syslog/input

1084620 ./var/log/data/autodisc/syslog-syslog

1084700 ./var/log/data/autodisc

1196088 ./usr/local/ess/update/archive

1249924 ./usr/local/ess/update/updates

2446020 ./usr/local/ess/update

3177668 ./var/log/httpd

5074604 ./usr/local/ace/incoming

5157552 ./usr/local/ace

77777600        ./usr/local/ess/data

81431336        ./usr/local/ess

87070864        ./usr/local

89045636        ./usr

157540960       ./var/log/data/inline

158625676       ./var/log/data

162400212       ./var/log

162608528       ./var

252209780       .

McAfee-ENMELM-VM12 / #

I have a lot of log folders storing a huge size. I'll check with McAfee support if I can delete something inside of it .

Highlighted

Re: McAfee SIEM - ESM Data lost

Jump to solution

For our case we can see high disk space utilization in ACE. the path is /var/log/data/inline and the files which is eating up the space are below two types of files (XX is the increamental number) and there are multiple files. Any idea what are these files, what is the purpose ad under which circumstances these files keep on storing in ACE device.

packet1.blob_pXX

event1_1.data_pXXX

You Deserve an Award
Don't forget, when your helpful posts earn a kudos or get accepted as a solution you can unlock perks and badges. Those aren't the only badges, either. How many can you collect? Click here to learn more.

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community