cancel
Showing results for 
Search instead for 
Did you mean: 
sroering
Level 13
Report Inappropriate Content
Message 11 of 13

Re: Concerns on WebReporter as a plausible tool for a large company

Jump to solution

lpp wrote:

We are using REST only temporarily to test this new approach (collecting small chunks of logs every few minutes instead of processing a huge volume at night), as WebGateway FTP push is always difficult to work (curl error 7) and needs time to set-up.

Most people push logs from the MWG after log rotation. You shouldn't be getting curl errors unless the files are really large. 100MB with compression enabled seems to be a common and good choice.

The overall settings are basically default, except for a few parameters.

I set the maximum concurrent log processing jobs to the value of 2.

Since the CPUs are basically idle, I hoped that this parameter would increase parallelism during log import.

If the system is idle, and there are jobs to be processed, it would be best to see why it is slow. If adding more log parsing jobs keeps it busy, then it would indicate that the log parsers are getting stuck on something.  The only 2 things that come to mind are 1) directories: If you have a directory attached, the log parser does the LDAP lookup for the display name.  2) Use hostname log parsing option is enabled:  If you have a lot of unauthenticated traffic for IP addresses not in DNS (such as DHCP addresses), then Windows will try to resolve them using NBIOS broadcasts. That puts the thread on hold for 30 seconds on every record. 

In those 2 cases, having more log parsing threads could help ride over the wait times, but I would still try to address the real problem instead of using more threads as a work-around.

We always loved page views (along with the detailed web), but again we thought that clearing the processing tab would decrease importing time. I set back page views. I just doubled the value of IP cache size, User cache size and Site request cache size, as the hit ratio was 99.9, 100% and 98.9% respectively.

Page views will always be faster.  Detail data has the most significant impact on the size of the DB, but shouldn't affect log parsing too much. Log records by default are "detailed data".  It is actually more work to condence that data into summary records. That is where the cache management is important and why you don't want to be changing values unless you know what you are doing.  I've documented all the advanced performance options here.

https://kc.mcafee.com/corporate/index?page=content&id=KB71665

Increasing IP/User/Site cache sizes larger isn't likely to make any difference. The number of records to load is relatively low and won't blow out memory.  But it would be rare that the defaults are not enouugh. Don't go above double on the aggregate cache.

Thank you again for your precious help.

Best regards,

LPP

you're welcome

lpp
Level 9
Report Inappropriate Content
Message 12 of 13

Re: Concerns on WebReporter as a plausible tool for a large company

Jump to solution

I am sorry, I didn't understand the hit cache ratio results I got.

Now I am using defaults for  IP cache size, User cache size and Site request cache size, and I doubled Aggregate record cache.

Thank you for your attention.

LPP

sroering
Level 13
Report Inappropriate Content
Message 13 of 13

Re: Concerns on WebReporter as a plausible tool for a large company

Jump to solution

lpp wrote:

I am sorry, I didn't understand the hit cache ratio results I got.

Now I am using defaults for  IP cache size, User cache size and Site request cache size, and I doubled Aggregate record cache.

Thank you for your attention.

LPP

After parsing several hours of logs, all the cache hit rations should be at or near 99% except the aggregate cache.  Aggregate cache will never hit 100% unless you are parsing the same log file in a loop.  Aggregate cache is summary data. We need to load summary data when parsing logs because they are grouped by hour.

https://kc.mcafee.com/corporate/index?page=content&id=KB71665

More McAfee Tools to Help You
  • Subscription Service Notification (SNS)
  • How-to: Endpoint Removal Tool
  • Support: Endpoint Security
  • eSupport: Policy Orchestrator
  • Community Help Hub

      New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

    • Find Forum FAQs
    • Learn How to Earn Badges
    • Ask for Help
    Go to Community Help

    Join the Community

      Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

    • Get helpful solutions from McAfee experts.
    • Stay connected to product conversations that matter to you.
    • Participate in product groups led by McAfee employees.
    Join the Community
    Join the Community