1 2 Previous Next 10 Replies Latest reply: Apr 10, 2012 4:01 PM by ajclements RSS

    Content Analysis & Advanced Content Analysis

    wormnrifle

      I'm looking for a good step by step tutorial on how to create and manage these.  I've been using a Cisco IronPort device for the past three years and finding it hard to make the transition to my IronMail box.

       

      I was also wondering if there was a way, at the CLI, to grep for messages?  I know I can look at the summary log, but it's not detailed enough for what I'm looking for.  Thanks.

        • 1. Re: Content Analysis & Advanced Content Analysis

          show events | grep <parameter>

          • 2. Re: Content Analysis & Advanced Content Analysis
            ijahnke

            Content Analysis can be used for either anti-spam purposes or compliance.

             

            Anti-spam

             

            There are times when spam messages manage to just barely make it through the appliance before TrustedSource has updated message sig/ip. Often times they have some sort of consistency and just need a little help to push them over the edge.

             

            Lets say there is a message coming in with the string "I am spam".

             

            1. You would go to Compliance -> Content Analysis -> Dictionaries
              1. Click on "Add New"
              2. We will name the dictionary "Spam_dictionary"
              3. Contribute Toward SpamProfiler: yes
              4. Search Option for HTML Parts:both
              5. Click "submit"
            2. You are brought back to the page Content Analysis - Manage Dictionaries
              1. Click on the dictionary that was just created called "Spam_dictionary"
                1. Click "Add New" at the bottom of the screen
                2. Content Type: Words/Phrases
                3. Search Type: Word Boundary
                4. Search Text: I am Spam
                5. Weight: 50
                6. Check the include box
                7. Scan Area: Body
                8. Contribution Type: Maximum Contribution
                9. Click "submit"
                10. Back on the Content Analysis - Manage Dictionaries page, make sure that Spam Profiler is checked.
            3. Now that the dictionary is created, we need to go to Anti-Spam -> Spam Profiler -> Configure
              1. Scroll to the very bottom of the page and you should see the dictionary you just created.
              2. Here you combine the scores to creat an aggregate value
                1. Confidence Value is the percentage of points you would like to apply for this dictionary
                2. Threshold Value is the maximum amount of points that can be applied for this dictionary
                3. The forumula will be (Point Score / Threshold Value) * Confidence Value
                4. If we had a message come in that scored 50 points because it found "I am spam" once, the confidence value is set at 50 and the threshold value is 100, the dictionary will score 25 points. (50/100) * 50 = 25
              3.  

            • 3. v6.7.1+ CLI Commands
              runcmd

              wormnrifle wrote:

               

               

              I was also wondering if there was a way, at the CLI, to grep for messages?  I know I can look at the summary log, but it's not detailed enough for what I'm looking for.  Thanks.

               

               

              Yeah, I think they made it exceedingly painful to pull information out of the CLI in v6.7.x.  It used to be that you could just "show log [log name]" and pipe it to a grep.  ...And you could FTP the smtpproxy log off of the appliance automatically as a text file and do traces for however far back you needed to.  I currently have a case open to figure out how to automatically transfer this detailed information off of the appliance on a daily basis.  I've been on v6.7.2 for a little more than a week and I'm less than pleased with how the CLI commands have changed in v6.7.1+.  In searching the good old "Secure Computing" forum ("https://supportcenter.ciphertrust.com/forum/"), I couldn't find much information on using the v6.7.x CLI, either.  Here's what I've been able to figure out since upgrading from 6.5.x...

               

              "show event" is okay, as long as the search you are doing is for the same day.  Here are commands I have found useful so far (but I'm still learning the new CLI commands)...

               

              showevents -s cfile="/conf/scevents.ini" -s ifile="/log/scmail-logs.bin" -g module=smtpproxy -d head | grep "whatever"

              showevents -s cfile="/conf/scevents.ini" -s ifile="/log/scmail-logs.bin.endsyyyymmdd" -g module=smtpproxy -d head | grep "whatever"

               

              Another complaint that I'm just about to raise with support is that "grep -i" to ignore case doesn't seem to work anymore.

               

              Here are the resources I've been able to find on using the CLI...

               

              IronMail 6.7 Administration Guide > "The Command Line" (Pages 599-613)

              Go to the Technical Support ServicePortal ("https://mysupport.mcafee.com/eservice/"); Click "Product Documentation"; Change product to "IronMail" and version to "IronMail 6.7"; Download "IronMail 6.7 Administration Guide"

               

              Secure Mail 6.7.1 Administration Guide > Appendix on "Event Logging Events" (Pages 667-699)

              Search the Knowledge Base for PD21612

               

              KB64659 - Secure Mail/Email Gateway: Using the CLI to view binary logs with the showevents command - 6.7.1 and later

               

              KB64625 - Secure Mail/Email Gateway: Tailing SMTPProxy Log from the CLI - 6.7.1 and later

               

              KB64599 - Secure Mail/Email Gateway: How to Use Show Events - Logging and CLI Changes in 6.7.1 and later

               

              Secure Mail 6.7.1 ShowEvents Utility

              Go to the McAfee / Secure Computing Support Center ("https://supportcenter.ciphertrust.com/index.php") and logon. If you don't have an account, you can request an account on the site but you'll probably have to call the Support Center Team to have the account activated (800-700-8328 x7 for Customer Service); After you are logged on, look under "Downloads"


              I hope that helps.  If anyone else has useful tips and tricks on using the CLI in v6.7.1+, or more accurate information than what I've posted, please share.  Thanks!

              • 4. Re: v6.7.1+ CLI Commands
                wormnrifle

                Thanks, that's very helpful.

                • 5. Re: v6.7.1+ CLI Commands
                  runcmd

                  In working with support, I was able to determine that you can setup the binary file to FTP automatically every night...

                   

                  Reporting > Reporting Advanced > Logs

                   

                  "Internal - Binary Log" is second from the bottom on the list in my GUI.  You can also click on the link to "Download" the file manually through the GUI.  I played with this for a while, yesterday, figuring out the "Secure Mail 6.7.1 ShowEvents Utility".  Don't expect the "showevents_win32.exe" to be any more user friendly than the CLI's "showevents", as it uses relatively the same syntax.

                   

                  The binary file is downloaded from the appliance as "scmail-logs.bin.tgz".  I used 7Zip to extract "scmail-logs.bin.tar" from that archive and then used 7Zip, again, to extract "scmail-logs.bin" and "scmail-logs.idb" from that tar file.  The "scmail-logs.bin" file appears to be the binary log file you need.  I copied both "scmail-logs.bin" and "scmail-logs.idb" to the same folder as the Windows "Secure Mail 6.7.1 ShowEvents Utility" and ran:

                   

                  showevents_win32 -s cfile="scevents.ini" -s ifile="scmail-logs.bin" -g module=smtpproxy -d head>smtpproxy.txt

                   

                  Running the above command floods my screen with "There is no such event id" but it does appear to redirect the valuable information to the file for me.  This gives me a (large) text file named "smtpproxy.txt" where the really useful information is.  You can then use a utility like BareGrep to search that file for sender's address, recipient's address, IP address, message ID, or whatever you want.  Or, if you don't want to use a 3rd party utility like BareGrep, just use Microsoft's "find" or "findstr".

                   

                  Technical Support was also able to confirm that "-i", to ignore case, no longer functions when used with grep in the CLI on v6.7.2.  I submitted a "Feature Request" asking for it back.  (I'm not holding my breath.)

                   

                  Hope that helps!

                  • 6. Re: v6.7.1+ CLI Commands

                    Hello,

                        I'd like to direct you to the "showevents" command, which is not to be confused with "show events"

                     

                    A full command syntax can be found in KB64659

                    http://kb.mcafee.com/agent/index?page=content&id=KB64659

                     

                     

                        In 6.7.x the development team implemented binary logging on the appliances. This means the standard log data is stored in a flat, indexed database file. Basically, it allows us to pull specific data much more quickly, and from one location. Previously, if the data you wanted did not appear in the summary log, you would have to know which file contained the data you were looking for. For example, if an email was delivered to the appliance and never made it out, you may need to read the smtpproxy, ripq, cfq, ccq, superq, vfq, and smtpo logs to find the log data which explained exactly what happened to the message. Now, we can reference all of this data in one binary logging file. Also, if you already know the event ID, connection ID, or message ID that pertains to whatever data you are looking for the showevents command will return the data many, many times faster than with standard log files.

                     

                       Including all this expanded functionality, and increased efficiency in one command requires adding some command arguments, which may seem a little complicated at first. I have outlined a simple explanation of "how it works" for the showevents command from the CLI.

                     

                     

                    ---

                     

                    All commands will start with this (you can copy/paste it for anything you need to do, this will return all log data):

                    showevents -s cfile=/conf/scevents.ini -s ifile=/log/scmail-logs.bin -d head

                     

                    Or This (for logs other than today):

                    showevents -s cfile=/conf/scevents.ini -s ifile=/log/scmail-logs.bin.ends20100427 -d head

                     

                     

                     

                    You can add a -g modifier (most common modifiers, many other modifiers are listed in the above KB article):

                     

                    -g module=smtpproxy

                    -g module=superq

                    -g module=smtpo

                    -g msgid=xxxxxx

                    -g eid=xxxxxx

                     

                    OR just grep the output for what you are looking for (You can use -g commands and still include |grep <whatever> at the end, just like any other unix/linux command line)

                     

                    |grep user@domain

                     

                     

                    Every line of the log file will have a connection ID or message ID as well as an event ID. Once you figure out which one belongs to your message, you should use the -g msgid=<msg or connection ID> option to get all the other info about the connection or message or the "-g eid=<event ID>" to pull out all occurrences of a specific type of event.

                     

                    Connection ID (and event ID):

                    20080901:00:19:09|22590300344740|9308|Message information <Source IP:Port:MessageID>|10.0.0.36:58228:4904983|

                    20080901:00:19:09|<CONNECTION ID NUMBER>|<EVENT ID NUMBER>|Message information <Source IP:Port:MessageID>|10.0.0.36:58228:4904983|

                     

                    Message ID (and event ID):

                    20090113:00:09:54|137|9233|Processing started.|

                    20090113:00:09:54|<MESSAGE ID NUMBER>|<EVENT ID NUMBER>|Processing started.|

                    Specific examples of finding specific data are included in the above KB article.
                    ---

                     

                     

                       In short: if you are looking for a little bit of data, show events |grep "whatever" is probably the easiest command to return what you are looking for. However, administrators who either create reports based on log data, or need to formulate complicated commands will find "showevents" is a much more efficient way of getting exact data from the appliance.

                     

                     

                    *Note: grep arguments were removed in 6.7.2 for security reasons. Support has already opened a feature modification request to have grep -i, grep -v, and grep -c added back to the CLI.

                     

                     

                    Message was edited by: Kevin Floda on 6/23/10 3:37:18 PM CDT
                    • 7. Re: v6.7.1+ CLI Commands
                      runcmd

                       

                      *Note: grep arguments were removed in 6.7.2 for security reasons. Support has already opened a feature modification request to have grep -i, grep -v, and grep -c added back to the CLI.

                       

                       

                      Interestingly, although a grep through the CLI is case sensitive, a search filter through the GUI does not appear to be case sensitive.  For example, if I issue the following two commands through the CLI...

                       

                      tail events | grep "Connection accepted"

                      tail events | grep "Connection Accepted"

                       

                      ...The first will yield results but the second will not because of case.  However, if I setup a filter through the GUI...

                       

                      Reporting > Reporting Advanced > Logs > View Log : Internal - Binary Log

                      Search Filter = Connection Accepted

                       

                      ...I still get results--even though the capital "A" of "Accepted" in the Search Filter is being comparted to a lower case "a" in my logs.

                       

                      The only down side is that the view through the GUI appears to only perform tailing and not a search of past events in the binary log.  For searching past events and ignoring case, you'd still need to download the binary and refer back to the "showevents_win32" utility mentioned in my previous post.  But if you are just tailing the logs and need to ignore case while searching for something, the GUI Search Filter is a nice workaround.

                      • 8. Re: v6.7.1+ CLI Commands
                        ijahnke

                        There are a couple options here actually:

                         

                        The first is:

                        McAfee]: show events | grep [A-a]ccepted

                        20100908:05:01:45|22851189970310|9236|Connection accepted.||
                        20100908:05:01:47|14590220|4139|-|Reply:  '250 Accepted'|

                         

                         

                         

                        Another option you would have for looking for case insensitive would be to use the "."

                         

                        In Unix, the period is considered a wildcard that matches any character.

                         

                        Here would be an example looking for the word "[C-c]onnected"

                         

                         

                        [McAfee]: tail events | grep .onnection

                        20100908:09:29:09|22851255687540|9236|Connection accepted.||
                        20100908:09:29:09|22851255675240|9312|Socket communication failed with client. Connection dropped||
                        20100908:09:29:09|22851255687541|9236|Connection accepted.||
                        20100908:09:29:09|22851255675237|9312|Socket communication failed with client. Connection dropped||
                        20100908:09:29:09|14782315|9492|Connection Status <status> -|1|
                        20100908:09:29:09|14782319|9492|Connection Status <status> -|1|
                        20100908:09:29:09|22851255683441|9312|Socket communication failed with client. Connection dropped||
                        20100908:09:29:10|22851255683442|9312|Socket communication failed with client. Connection dropped||
                        20100908:09:29:10|22851255691638|9236|Connection accepted.||
                        20100908:09:29:10|22851255675239|9312|Socket communication failed with client. Connection dropped||

                         

                         

                        so if youre looking for either "Accepted" or "accepted" you could run this command:

                         

                        [McAfee]: show events | grep .ccepted

                        20100908:05:01:45|22851189970310|9236|Connection accepted.||
                        20100908:05:01:47|14590220|4139|-|Reply: '250 Accepted'|

                         

                         

                         

                        Hope this helps

                         

                        --Ivan

                         

                         

                        Message was edited by: Ivan Jahnke on 9/8/10 8:40:59 AM CDT

                         

                         

                        Message was edited by: Ivan Jahnke on 9/8/10 8:41:23 AM CDT
                        • 9. Re: Content Analysis & Advanced Content Analysis
                          wormnrifle

                          I just recently upgrade to 6.7.2 HF6 and it would appear the command kfloda posted earlier no longer works?  At least when I try it, I get an Invalid Command reply.  Has this changed now? 

                           

                          Thanks.

                          1 2 Previous Next