Showing results for 
Show  only  | Search instead for 
Did you mean: 
Former Member
Not applicable
Report Inappropriate Content
Message 1 of 9

McAfee SIEM - ESM Data lost

Jump to solution

Hello guys,

Recently I faced it 2 times with 2 different clients.

The data disappeared after some problems and I'd like to know if you guys are facing it or have this terrible experience before.

Now I have only 5 days of data:
last 60 days, but I only have 5.JPG

and for only that I have 250GB consumed so I ask, what can files are atoring the data lost? Where is it?

df h.JPG

Look my Data folder:

McAfee-ENMELM-VM12 /usr/local/ess/data # ls

ADGroup.blob                        ExtDeviceAttr.blob         TagSevBits.index                              NotificationEMailGroups.index

ADGroup.index                       ExtDeviceAttr.index            TagUpdateException.index                           NotificationMembers.index

ADGroupSM.index                     ExternalDevice.index              Theme.index                                      NotificationUser.index         ThirdPartyConfig.blob

Access.index                        GeoLoc.index                                                   GetRedundantSettings.sql          OS.index                       ThirdPartyConfig.index

Action.index                        Groups.blob                       Obfuscation.blob                                          ThirdPartyType.index

AggException.index                  Groups.index                      Obfuscation.index              Timezone.blob

Alert1.blob_p35                     HCFilters.blob                    PluginData.blob      

Alert1.data_p35                                     Timezone.index

Alert_AlertID_1.index_p35           HCFilters.index                   PluginData.index               TriggeredAlarm.blob


Alert_DstMac_1.index_p35                          Plugins.index                  TriggeredAlarm.index

Alert_DstPort_1.index_p35           Hosts.index                       PortApps.blob        

Alert_GUID1_1.index_p35                                TriggeredCondition.index

Alert_GUID2_1.index_p35             ICMPType.index                    PortApps.index       

Alert_ID_1.index_p35                IPS.blob                                     UCFA2U.index

Alert_SigIDDstIP_1.index_p35                          Ports.index          

Alert_SigIDSrcIP_1.index_p35        IPS.index                         Preprocess.blob                UCFC2U.index

Alert_SigID_1.index_p35             IPSBlob.blob                  

Alert_SrcIP_1.index_p35                         Preprocess.index               UCFN2U.index

Alert_StaticStrings1.bloom_p35      IPSBlob.index                  UCFName.blob                                    PreprocessException.index

Asset.index                         IPSChange.index                   PreprocessGroup.blob           UCFName.index                       

AssetGroup.index                    IPSCheck.index                    PreprocessGroup.index          US.index                                   UpdateBlob.blob

AssetGroupXRef.index                ItemRights.index                  Profile.index                                            Query.blob                     UpdateBlob.index

AssetVendor.index                   Job.index                                       LocaleString.blob                 Query.index                    Usage.index

AssetVulnerability.index                              LocaleString.index                RemoteAction.index             UserField.index

AutoCreateRule.index                LocaleString_StrValue.bloom         Log.blob_p2                       RemoteActionAttr.index         UserFieldUse.index

AutoCreateRuleCriteria.index        Log.data_p2                       RemoteCommandAttr.blob                      Log.index_p2                      ReportComponent.blob           UserFilterList.index

Blacklist.index                                    LogCategory.index                 ReportComponent.index          UserIPSIDJoin.index

BlacklistBuffer.index               MessageTemplate.blob                                 ReportFolder.index             UserLicense.index

CaseEvents.index                    MessageTemplate.index             Reports.blob         

CaseMgt.blob                                            UserViewExclusion.index                        NDDevice.index                    Reports.index        

CaseMgt.index                         Rights.blob                    User_IPS.index

CaseMgt_Name.bloom                  NDDeviceAddresses.index                     Users.blob

CaseMgt_Notes.bloom                   Rights.index         

CaseMgt_Viewed.bloom                NDDeviceInterface.index           Users.index                               RightsAssignment.index

CaseOrg.index                       NDDeviceVLAN.index                Rule.blob                      UsersPW.index                                         Var.blob

CaseStatus.index                    NDEPDevices.index                 Rule.index           

ChangeLog.blob                               RuleParam.blob                 Var.index                      NDEPParams.index                         VarException.blob

ChangeLog.index                            RuleParam.index      

Class.blob                          NDEndPointIP.index                RuleParamChange.blob           VarException.index                           View.blob

Class.index                         NDEndPointIPHistory.index         RuleParamChange.index

Condition.blob                              View.index                      NDEndPoints.index                 RuleUseException.index         ViewComponent.blob


Connection1.blob_p1                 NDEndPointsHistory.index          RuleVA.index                   ViewComponent.index

Connection1.data_p1                 NDFolder.blob                    

Connection_ConnectionID_1.index_p1                     RuleVIN.index                  ViewFolder.index

Connection_DstIPDur_1.index_p1      NDFolder.index                    

Connection_DstPort_1.index_p1               SMXRef.index                   Vulnerability.index

Connection_ID_1.index_p1            NDFolderDevice.index              Scoring.blob         

Connection_LocIDDst_1.index_p1                               WMIType.index

Connection_Prot_1.index_p1          NDIPLoc.index                     Scoring.index        

Connection_SrcIPDur_1.index_p1                  ScoringSource.blob             WatchListValues1.index

Connection_SrcPort_1.index_p1       NDNeighbors.index                    WatchLists.blob

Connection_StaticStrings1.bloom_p1                     ScoringSource.index  

Connection_User16_1.index_p1        NDParams.index                     WatchLists.index

DataEnrichment.blob                      SelectFieldName.index                 NDParamsDetail.index              SendEMail.blob                 Zone.index

DataEnrichment.index                          NDParamsExclusion.index           SendEMail.index                ZoneIPMap.index

DataEnrichmentFields.index                SendSyslog.blob                connect_esm.sql            NDPortControl.index                     finalpartitionlist.sql

DataEnrichmentIPSID.index                     SendSyslog.index     

DeviceFolder.blob                   NDProcess.index                         ngcp.cfd_old                       StringMap1.index               ngcp.cfg

DeviceFolder.index                  NDSearchResults.index             StringMap_Name1.bloom          ngcp.cfg_old            NitroError.Log                    SysSettings.blob               ngcp.cpy

DeviceFolderIPSJoin.index           Notes.blob                             ngcp.cpy_old                               SysSettings.index              ngcp.dfl

DistributedESM.index                Notes.index                      ngcp.dfl1407848751                          Notification.blob                 TPTypeApplication.index        ngcp.dfl1410804574

EMail.index                                             ngcp.dfl_1399392877                     Notification.index                Tag.index                      ngcp.old

EMailGroup.index                             old_sa/         NotificationAction.index          TagAsset.index                 packet1.blob_p1

EMailGroupEMailAddress.index        NotificationActionAttr.blob         packet1.blob_p2

ESMFilters.blob                  TagAssetException.index        packet1.data_p1                     NotificationActionAttr.index             packet1.data_p2

ESMFilters.index                      TagAssetGroup.index            packet1.index_p1

EventForwarding.blob                NotificationCheck.index                    packet1.index_p2         TagRule.index                  partitionlist.sql

EventForwarding.index               NotificationEMailAddresses.index

I noticed many *old files ... but none seems the database lost.

In attachment some logs.

I appreciate your on that, I don't want to face it for the thrid time hehe. Tks

1 Solution

Accepted Solutions
Former Member
Not applicable
Report Inappropriate Content
Message 7 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

'du' Linux command is your friend to pinpoint directories consuming precious disk space.

From the files that you listed above, check their filesizes.

View solution in original post

8 Replies
Former Member
Not applicable
Report Inappropriate Content
Message 2 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

check for detached database partitions:

nquery -d '/usr/local/ess/data/ngcp.dfl|:::1|1111' -q 'show partitions from alert'

Former Member
Not applicable
Report Inappropriate Content
Message 3 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

This is what I get:

McAfee-ENMELM-VM12 ~ # nquery -d '/usr/local/ess/data/ngcp.dfl|:::1|1111' -q 'show partitions from alert'

executing queryToRun: [show partitions from alert]

queryToRun 100% complete, 00:00.000 elapsed


Partition     35 | 02/14/2015 00:00:00.000 to 02/19/2015 23:59:59.999 |attached  |      18,000,259 recs|v193956454654519|mod 02/19/2015 16:58:47|open

Seems everthing is attached?

Former Member
Not applicable
Report Inappropriate Content
Message 4 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

Sorry, now I see. Only partitions from 02/14/2015 to 02/19/2015 are attached. How can I see old partitions and add them?

Former Member
Not applicable
Report Inappropriate Content
Message 5 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

You don't have any detached partitions, I'm afraid.

They may be elsewhere, have you checked backups?

Also, as you mentioned "recreation of data source" in another thread, deleting old data source, wipes all the data received from this data source from ESM database.

Former Member
Not applicable
Report Inappropriate Content
Message 6 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

Unfortunately I don't have any events backup but there's no problem to recover the data since those partitions are not there. My problem now it's these amount of data stored in somewhere that I cannot see and taking up space in disk .

Former Member
Not applicable
Report Inappropriate Content
Message 7 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

'du' Linux command is your friend to pinpoint directories consuming precious disk space.

From the files that you listed above, check their filesizes.

View solution in original post

Former Member
Not applicable
Report Inappropriate Content
Message 8 of 9

Re: McAfee SIEM - ESM Data lost

Jump to solution

Thanks aszotek, seems the problem is not only in the ./usr/local/ess/data folder: I have a lot of files as well in /var/log .

102052  ./usr/lib/locale

104604  ./tmp

112756  ./usr/share

125164  ./usr/lib/perl5

130536  ./var/log/shm/collector/p/rpcclient

142940  ./var/log/shm/collector/p

142960  ./var/log/shm/collector

144252  ./var/log/shm

147508  ./var/www/html/help

153148  ./var/log/data/inline/tmp

153336  ./etc

160080  ./root/update_db_backups

161304  ./root

184204  ./var/www/html

185496  ./var/www

188412  ./usr/java/jre1.6.0_26-i586/lib

189764  ./usr/java/jre1.6.0_26-i586

217924  ./usr/local/bin

280980  ./usr/java

306228  ./var/log/data/inline/thirdparty.logs

516144  ./usr/lib64

881764  ./usr/local/ess/dbbackup

907020  ./usr/lib

1084484 ./var/log/data/autodisc/syslog-syslog/input

1084620 ./var/log/data/autodisc/syslog-syslog

1084700 ./var/log/data/autodisc

1196088 ./usr/local/ess/update/archive

1249924 ./usr/local/ess/update/updates

2446020 ./usr/local/ess/update

3177668 ./var/log/httpd

5074604 ./usr/local/ace/incoming

5157552 ./usr/local/ace

77777600        ./usr/local/ess/data

81431336        ./usr/local/ess

87070864        ./usr/local

89045636        ./usr

157540960       ./var/log/data/inline

158625676       ./var/log/data

162400212       ./var/log

162608528       ./var

252209780       .

McAfee-ENMELM-VM12 / #

I have a lot of log folders storing a huge size. I'll check with McAfee support if I can delete something inside of it .

Re: McAfee SIEM - ESM Data lost

Jump to solution

For our case we can see high disk space utilization in ACE. the path is /var/log/data/inline and the files which is eating up the space are below two types of files (XX is the increamental number) and there are multiple files. Any idea what are these files, what is the purpose ad under which circumstances these files keep on storing in ACE device.



You Deserve an Award
Don't forget, when your helpful posts earn a kudos or get accepted as a solution you can unlock perks and badges. Those aren't the only badges, either. How many can you collect? Click here to learn more.

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community