6 Replies Latest reply on Aug 25, 2015 4:29 PM by jmickley

    Weighted Dictionaries in Spam

    runcmd

      In recent weeks we contracted McAfee Professional Services for assistance in upgrading our email appliance from IronMail v6.7.2 to MEG 7.6 and also switched to a clustered environment.  In the old system it was possible to provide end users access to inbound messages that were quarantined as the result of compliance dictionary matches by quarantined both spam and compliance to the same user accessible quarantine queue.  Also, end user whitelist submissions could bypass compliance on inbound messages.  In the new system, it does not appear to be possible to allow end users the ability to view, release, and/or whitelist messages that are quarantined as the result of compliance dictionary matches.


      We utilize a custom dictionary that was migrated from the old system and is used to combat phishing schemes.  Ideally, I would like to see this dictionary utilized in the spam category for contribution as "Spam Terms".  I suspect this would solve the problem of users being completely unware of messages destined for them that are blocked as the result of the dictionary being in compliance.  However, this custom dictionary is weighted and it does not appear to be possible to specify a dictionary threshold when adding a dictionary to "Spam Terms".


      It looks like KB83041 might be the solution I'm looking for but I don't see "Add score to spam score (Monitor)" as an option when creating a compliance rule in my environment.  I have already submitted a Product Enhancement Request for the addition of a dictionary threshold feature in spam; however, what options are available as a workaround for this problem?  Our executives are having legitimate messages blocked in compliance without any knowledge of the message that is being withheld from them.  At this time, I've had no choice but to discontinue the use of this valuable custom dictionary.


      Thanks for taking the time to read my post.  Any feedback would be greatly appreciated!

        • 1. Re: Weighted Dictionaries in Spam
          runcmd

          I opened a case with technical support and it is my understanding that there is currently no way to utilize a weighted dictionary in spam or have compliance dictionaries contribute to the spam score.  The information provided in knowledge base article KB83041 is now useless because the feature that allowed compliance to contribute to the spam score was removed in v7.6.  I can implement an MQM device for storage of quarantined messages off of the gateway(s), which I'm told should allow my users to see messages quarantined as the result of the compliance category, but they would still be unable to release or whitelist those messages.  Right now, I'm relying solely upon my feature request (PER) for resolution.  If having the ability to have weighted dictionaries in the spam category is important to anyone else reading this, I'd recommend that you submit a PER too.

          • 2. Re: Weighted Dictionaries in Spam
            jmickley

            Hi runcmd,

             

            The statements about the threshold dictionaries not being available for spam use in version 7.x are correct.  It is one feature that did not get carried over from 6.7.2 unfortunately.  However, although the end users cannot directly release these messages when on-box quarantine is used, they should still be notified in the quarantine digest about these messages.  They can then submit them to be released, however an admin would have to fully release the message.  If they are not seeing these messages at all in their digests, it sounds like the digest template has been modified to not show the messages blocked for compliance.

             

            One other option is to look into using a McAfee Quarantine Manager(MQM).  If you are familiar with 6.7.2, it is the rough equivalent of a CQS.  If you use an MQM to quarantine messages, you can then give end users the ability to directly release messages that were blocked for compliance reasons.  MQM is a free application that resides on a Windows server and you should be able to download it from our downloads page.  Keep in mind though that this unfortunately will still not give the end users the ability to whitelist through compliance, but they will be able to directly release the messages if you allow them to.  Hope this helps.

             

            --Jake

            • 3. Re: Weighted Dictionaries in Spam
              runcmd

              Thanks for the reply, Jake.  I appreciate your insight.  I've been working with Rumed in support (who has been great) on this issue and he said that he might also have an idea on how to address or work around this problem but I'm not getting my hopes up at this point.  I have a meeting scheduled with our account representative and my upper management to discuss this issue later this week because this could be the straw that breaks the camel's back in me recommending that our organization look for another email gateway solution.  In my opinion, weighted dictionaries in spam is basic functionality.  Intel's development team apparently made the decision to provide a valuable feature to "Add score to spam score" for compliance as a viable workaround in this situation, only to remove it later.  I question what the development team's logic was in deciding to remove this feature from 7.6.  (If you can enlighten me, that'd be great.)  Equally frustrating is to be told directly in a support case on this issue, "…at this time we do not have plans to address your specific PER."  (I had another case open because MEG 7.6 isn't compatible with a Microsoft FTP server and the response was relatively the same--pushing the problem to Microsoft and telling me to use a different FTP service.)  How dictionaries are handled and having no clean migration of dictionaries from 6.x to 7.x format is why I held out as long as I could on upgrading.  Sadly, when it came time to bite the bullet and upgrade, it wasn't even that difficult for me to write my own script to convert my dictionaries--weights and all!  Based upon the problems I've encountered since upgrading, I can't say I'm surprised that Intel Security isn't in the "Leaders" quadrant for Secure Email Gateways for 2015 in Gartner's Magic Quadrant.  I've considered installing 7.5 on our appliances just so I can have access to the "Add score to spam score" feature long enough to look for an alternate solution.

               

              Someone reading my comments might be inclined to believe that I am overreacting.  However, I'll take the time to explain why weighted dictionaries in spam is so important (at least to me).  GTI is a great tool in combating spam and phishing but it is a never-ending process of updating reputations in a database that all MEGs look to for determining whether a sender or URL is legitimate.  However, there will always be newly compromised mail servers used to pump spam, and messages will always slip through in that period of time between when a mail server is compromised and when its reputation is updated in the Threat Intelligence database.  Additionally, in a targeted spear phishing attack, my mail server may be the only one receiving hits from a specific mail server that has been newly compromised.  For phishing and scam emails, I must then rely primarily on my dictionaries as a frontline defense.  Yes, I can configure such a weighted dictionary in compliance but then I'm left with two feasible options (or so I've been told) where I must decide which is the lesser of two evils:  (1) Quarantine in a manner that is blind to the user, so that they have no idea that a message was ever sent to them.  This can be REALLY bad in the case of false positives.  (2) Setup an MQM where the end user can see the message but can't release it.  This results in headache on the part of the end user (who sees a message they might want but can't release), our Help Desk (who the users are going to call to open a ticket to get the message) and me (in the manual administrative intervention required to release it).

               

              …although the end users cannot directly release these messages when on-box quarantine is used, they should still be notified in the quarantine digest about these messages.  They can then submit them to be released, however an admin would have to fully release the message.  If they are not seeing these messages at all in their digests, it sounds like the digest template has been modified to not show the messages blocked for compliance.

               

              Our digest was setup by the McAfee professional services person we contracted for the upgrade.  I see that we are not using the default value but I'm not sure what was removed or added (if anything) other than our company branding of the digest.  Can having compliance quarantine visible to users be accomplished with on-box quarantine?  I only see the tokens listed below available for use in the digest and there doesn't appear to be a variable available specifically for a compliance list.  We are using %SPAM_LIST% in our digest.  Should we be using %CONTENT_LIST% or %FULL_CONTENT_LIST% instead?

               

              Email > Quarantine Configuration > Digest Message Content > Edit the digest report : Tokens
              %SPAM_LIST%
              %FULL_SPAM_LIST%
              %CONTENT_LIST%
              %FULL_CONTENT_LIST%
              %WHITE_LIST%
              %BLACK_LIST%
              %SENDER%
              %RECIPIENT%
              %EXP_DELAY%
              %MAX_EXP_DELAY%
              %PRODUCT_NAME%
              %POST_MASTER%
              %DIGEST_DATE%
              %ADD_WHITE_LIST%
              %ADD_BLACK_LIST%
              %SET_EXP_DELAY%

               

              If I can have the digest contain a full list of messages in quarantine for spam and compliance, that might be the easiest way to avoid implementing an MQM.  The MQM isn't really going to give me additional functionality, as far as users being able to release messages anyway.

              • 4. Re: Weighted Dictionaries in Spam
                jmickley

                Hi runcmd,

                 

                I sit right next to Rumed, he is a good guy and he says hi.

                 

                As far as why that functionality was removed, I cannot answer that question.  However, I can help out with the quarantine digest issue.  To notify the end users of messages quarantined due to compliance, you are correct and you would want to use either the %CONTENT_LIST% or %FULL_CONTENT_LIST% tokens.  Without going into an explanation(I can if you want), if you are using the %SPAM_LIST% token, then I recommend using the %CONTENT_LIST% token.  Keep in mind that the content token is not meant to replace the spam token.  The spam token is explicitly for spam messages and the content list is for compliance type blocks.  If pro serv modified this value, you could get the default back by putting the checkmark back in the box to use the default value for the digest report content.  If you don't want to do that, here is the default template from one of our lab boxes so you have an idea of what it looks like:

                 

                Quarantined email messages

                Spam quarantine

                The email messages listed below have been placed in your personal spam quarantine since you last received your spam quarantine summary. They will be deleted after %EXP_DELAY% days.

                %SPAM_LIST%

                Content quarantine

                The email messages listed below have been placed in your personal content quarantine since you last received your content quarantine summary. They will be deleted after %EXP_DELAY% days.

                %CONTENT_LIST%

                Blacklist and whitelist summaries

                The email addresses listed below represent individuals or organizations whom you have blacklisted or whitelisted.

                • Email from senders in your blacklist will be classified as spam.
                • Email from senders in your whitelist will not be filtered for spam.

                Blacklist

                %BLACK_LIST%

                %ADD_BLACK_LIST%

                Whitelist

                %WHITE_LIST%

                Quarantine settings

                Email messages in your personal quarantine area are automatically deleted after %EXP_DELAY% days.

                %SET_EXP_DELAY%

                 

                For the MQM side of things, you would gain a bit of functionality by using it.  As opposed to having the end users only able to submit the messages to be released, you would be able to grant them the ability to fully release the messages blocked for compliance rules.  The two downsides to this is it would require another server (most people run it on a VM), and it would also require a bit of learning on the admin and end user side.  Hope this helps.

                 

                --jake

                • 5. Re: Weighted Dictionaries in Spam
                  runcmd

                  Thanks Jake!  I've started taking a closer look at the MQM option to see if it will meet our needs.  This will definitely be a part of the discussion with our McAfee account manager when the meeting occurs.  If users can release messages quarantined as the result of Compliance matches when an MQM is in place, that's a LOT better than the options we have now--even if they can't whitelist.  Thanks for share this.  This offers some form of light at the end of the tunnel.

                  • 6. Re: Weighted Dictionaries in Spam
                    jmickley

                    No problem sir.  If you would like to talk about it, let Rumed know and he can get me your contact info.

                     

                    --jake