cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
fw_mon
Reliable Contributor
Reliable Contributor
Report Inappropriate Content
Message 1 of 7

MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

the "Bypass Microsoft (Office 365) Services" rules and various whitelists are placed usually before the url filter so the categories and the reputation cannot be checked against the trustedsource DB (local or cloud). In the log cycle such requests have an empty list of categories and the web reputation "0".

How to distinguish if the url was bypassed or just is not categorized yet? I need to accomplish this using available properties only, without using user-defined properties or "marking" any whitelisting rules.

Currently I'm using "*forString(URL)" properties in the logging cycle to get this info after the active part of the transaction is over, but I'm looking for a better way.

Any ideas?

2 Solutions

Accepted Solutions
asabban
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 4 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Hello,

the logging cycle comes immediately after the transaction is finished. But writing the file to disk is "buffered", so the file is not opened, written and closed immediately, but once some data to log has been collected.

Anyway, writing  the user-defined property does not cause any performance impact that would be noticable. All you do is remembering a string and write it down later, compared to other operations that are performed in the transaction this is not an expensive task.

If you do not want to log a user-defined property it might be possible to do it differently. You will need to set a user-defined property anyway to remember yourself if you have executed URL categorization or not (so something that is false by default but becomes true when you entered the URL Categorization rule set).

But in the log handler you could - for example - log a string like "No categories found" IF the user-defined property is true (URL Categorization performed) and URL.Categories is empty on the column you usually log the URL categories.

IF the user-defined property is false (no URL Categorization performend due to bypasses) and URL.Categories is empty you can log a string like "No categorization performed".

If categories were found, just log the categories.

Note: If the logs are parsed and processed by some additional service (Splunk, CSR, etc.) this might change how the reporting looks.

But doing it this way you do not need to log an additional column.

The user-defined property as such is cheap to use, we have customers using hundrets of them for each transaction. It depends on the action you perform with them. In this case we just set it manually to true/false.

Best,
Andre

View solution in original post

asabban
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 6 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Hello,

no action is performed in the logging cycle. Its usage is restricted. Any property that is not filled already will not be fillen when called in the logging cycle. So basically yes, if you do a hash calculation in the logging cycle it will not have an impact on performance, because the hash calculation is not done and the property stays empty 🙂

Apart from that, of course, you can use some large regex list as criteria for matching a rule in the logging cycle, which will have an impact. But it does not make any difference if you do this in reqeust, response or logging cycle - the result is the same.

Writing the log is not bound to the transaction. The transaction ends with the logging cycle and the log line that should go into the log is written into a buffer by calling the Logging event. The transaction finishes. If additional transaction finish or the timer expires, the log data held in memory is written to disk. This happens independently from the original transaction.

I am not aware of "high priority" transactions in MWG. Every transaction is worked on, there is no way (that I know) causing MWG to prefer one transaction over another. 

If you manage to enforce a rule engine error in the logging cycle (which I believe could happen by calling incompatible properties in the criteria) the transaction will end and you don't have a log line. The processing of request and response cycle is finished before the logging cycle is executed, so the user already has received the file he requested, so there should not be an impact (apart from the missing log).

Andre

View solution in original post

6 Replies
asabban
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 2 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Hello,

you can add another user-defined property like "User-Defined.My.WasCategorizedByGTI" as Bool and set it to false. When you go into the URL Filter rule set where the lookup is performed you set it to true.

You can log this propery as an additional value. So you know if your request has entered the URL Filter rule set and categoriation was attempted or not.

Andre

fw_mon
Reliable Contributor
Reliable Contributor
Report Inappropriate Content
Message 3 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Thank you Andre,

that was I did before. But I'm looking for a way without user-defined properties.

Anyway, it works. Does/can the url categorization in the log cycle impact the overall performance? I'm trying to get the web reputation score and url categories for bypassed requests. My understanding the logging is decoupled from the req/resp cycles and is asyncronous, isn't it?

asabban
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 4 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Hello,

the logging cycle comes immediately after the transaction is finished. But writing the file to disk is "buffered", so the file is not opened, written and closed immediately, but once some data to log has been collected.

Anyway, writing  the user-defined property does not cause any performance impact that would be noticable. All you do is remembering a string and write it down later, compared to other operations that are performed in the transaction this is not an expensive task.

If you do not want to log a user-defined property it might be possible to do it differently. You will need to set a user-defined property anyway to remember yourself if you have executed URL categorization or not (so something that is false by default but becomes true when you entered the URL Categorization rule set).

But in the log handler you could - for example - log a string like "No categories found" IF the user-defined property is true (URL Categorization performed) and URL.Categories is empty on the column you usually log the URL categories.

IF the user-defined property is false (no URL Categorization performend due to bypasses) and URL.Categories is empty you can log a string like "No categorization performed".

If categories were found, just log the categories.

Note: If the logs are parsed and processed by some additional service (Splunk, CSR, etc.) this might change how the reporting looks.

But doing it this way you do not need to log an additional column.

The user-defined property as such is cheap to use, we have customers using hundrets of them for each transaction. It depends on the action you perform with them. In this case we just set it manually to true/false.

Best,
Andre

View solution in original post

fw_mon
Reliable Contributor
Reliable Contributor
Report Inappropriate Content
Message 5 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Thank you Andre, I've accepted your answer + kudo.

But just to confirm: any action in the loggig cycle, like performing a url lookup, string manipulations, hash calculations, DNS lookups, etc - all this do not have any impact on a whole transaction? Or such operation should be completed before the timer flushes the buffer (every 30 sec)? Can I attach such rules to high priority transaction where no additional delay is tolerated? Asking differently - can a "rule engine error" in the logging cycle lead to aborting the transaction? 

🤔

asabban
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 6 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

Hello,

no action is performed in the logging cycle. Its usage is restricted. Any property that is not filled already will not be fillen when called in the logging cycle. So basically yes, if you do a hash calculation in the logging cycle it will not have an impact on performance, because the hash calculation is not done and the property stays empty 🙂

Apart from that, of course, you can use some large regex list as criteria for matching a rule in the logging cycle, which will have an impact. But it does not make any difference if you do this in reqeust, response or logging cycle - the result is the same.

Writing the log is not bound to the transaction. The transaction ends with the logging cycle and the log line that should go into the log is written into a buffer by calling the Logging event. The transaction finishes. If additional transaction finish or the timer expires, the log data held in memory is written to disk. This happens independently from the original transaction.

I am not aware of "high priority" transactions in MWG. Every transaction is worked on, there is no way (that I know) causing MWG to prefer one transaction over another. 

If you manage to enforce a rule engine error in the logging cycle (which I believe could happen by calling incompatible properties in the criteria) the transaction will end and you don't have a log line. The processing of request and response cycle is finished before the logging cycle is executed, so the user already has received the file he requested, so there should not be an impact (apart from the missing log).

Andre

View solution in original post

fw_mon
Reliable Contributor
Reliable Contributor
Report Inappropriate Content
Message 7 of 7

Re: MWG: how to distinguish url which bypassed (url filter) and which are not categorized?

Jump to solution

exactly what I need to know, thank you!

 

You Deserve an Award
Don't forget, when your helpful posts earn a kudos or get accepted as a solution you can unlock perks and badges. Those aren't the only badges, either. How many can you collect? Click here to learn more.

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community