Skip navigation
McAfee Secure sites help keep you safe from identity theft, credit card fraud, spyware, spam, viruses and online scams
6268 Views 10 Replies Latest reply: Apr 10, 2012 4:01 PM by ajclements RSS 1 2 Previous Next
wormnrifle Newcomer 8 posts since
Feb 5, 2010
Currently Being Moderated

Feb 5, 2010 2:25 PM

Content Analysis & Advanced Content Analysis

I'm looking for a good step by step tutorial on how to create and manage these.  I've been using a Cisco IronPort device for the past three years and finding it hard to make the transition to my IronMail box.

 

I was also wondering if there was a way, at the CLI, to grep for messages?  I know I can look at the summary log, but it's not detailed enough for what I'm looking for.  Thanks.

  • jp74 Newcomer 1 posts since
    Apr 25, 2010
    Currently Being Moderated
    1. Apr 25, 2010 1:48 PM (in response to wormnrifle)
    Re: Content Analysis & Advanced Content Analysis

    show events | grep <parameter>

  • ijahnke McAfee Employee 118 posts since
    May 12, 2010
    Currently Being Moderated
    2. May 13, 2010 12:20 AM (in response to wormnrifle)
    Re: Content Analysis & Advanced Content Analysis

    Content Analysis can be used for either anti-spam purposes or compliance.

     

    Anti-spam

     

    There are times when spam messages manage to just barely make it through the appliance before TrustedSource has updated message sig/ip. Often times they have some sort of consistency and just need a little help to push them over the edge.

     

    Lets say there is a message coming in with the string "I am spam".

     

    1. You would go to Compliance -> Content Analysis -> Dictionaries
      1. Click on "Add New"
      2. We will name the dictionary "Spam_dictionary"
      3. Contribute Toward SpamProfiler: yes
      4. Search Option for HTML Parts:both
      5. Click "submit"
    2. You are brought back to the page Content Analysis - Manage Dictionaries
      1. Click on the dictionary that was just created called "Spam_dictionary"
        1. Click "Add New" at the bottom of the screen
        2. Content Type: Words/Phrases
        3. Search Type: Word Boundary
        4. Search Text: I am Spam
        5. Weight: 50
        6. Check the include box
        7. Scan Area: Body
        8. Contribution Type: Maximum Contribution
        9. Click "submit"
        10. Back on the Content Analysis - Manage Dictionaries page, make sure that Spam Profiler is checked.
    3. Now that the dictionary is created, we need to go to Anti-Spam -> Spam Profiler -> Configure
      1. Scroll to the very bottom of the page and you should see the dictionary you just created.
      2. Here you combine the scores to creat an aggregate value
        1. Confidence Value is the percentage of points you would like to apply for this dictionary
        2. Threshold Value is the maximum amount of points that can be applied for this dictionary
        3. The forumula will be (Point Score / Threshold Value) * Confidence Value
        4. If we had a message come in that scored 50 points because it found "I am spam" once, the confidence value is set at 50 and the threshold value is 100, the dictionary will score 25 points. (50/100) * 50 = 25
      3.  

  • runcmd Apprentice 221 posts since
    Feb 22, 2006
    Currently Being Moderated
    3. May 14, 2010 2:59 PM (in response to wormnrifle)
    v6.7.1+ CLI Commands

    wormnrifle wrote:

     

     

    I was also wondering if there was a way, at the CLI, to grep for messages?  I know I can look at the summary log, but it's not detailed enough for what I'm looking for.  Thanks.

     

     

    Yeah, I think they made it exceedingly painful to pull information out of the CLI in v6.7.x.  It used to be that you could just "show log [log name]" and pipe it to a grep.  ...And you could FTP the smtpproxy log off of the appliance automatically as a text file and do traces for however far back you needed to.  I currently have a case open to figure out how to automatically transfer this detailed information off of the appliance on a daily basis.  I've been on v6.7.2 for a little more than a week and I'm less than pleased with how the CLI commands have changed in v6.7.1+.  In searching the good old "Secure Computing" forum ("https://supportcenter.ciphertrust.com/forum/"), I couldn't find much information on using the v6.7.x CLI, either.  Here's what I've been able to figure out since upgrading from 6.5.x...

     

    "show event" is okay, as long as the search you are doing is for the same day.  Here are commands I have found useful so far (but I'm still learning the new CLI commands)...

     

    showevents -s cfile="/conf/scevents.ini" -s ifile="/log/scmail-logs.bin" -g module=smtpproxy -d head | grep "whatever"

    showevents -s cfile="/conf/scevents.ini" -s ifile="/log/scmail-logs.bin.endsyyyymmdd" -g module=smtpproxy -d head | grep "whatever"

     

    Another complaint that I'm just about to raise with support is that "grep -i" to ignore case doesn't seem to work anymore.

     

    Here are the resources I've been able to find on using the CLI...

     

    IronMail 6.7 Administration Guide > "The Command Line" (Pages 599-613)

    Go to the Technical Support ServicePortal ("https://mysupport.mcafee.com/eservice/"); Click "Product Documentation"; Change product to "IronMail" and version to "IronMail 6.7"; Download "IronMail 6.7 Administration Guide"

     

    Secure Mail 6.7.1 Administration Guide > Appendix on "Event Logging Events" (Pages 667-699)

    Search the Knowledge Base for PD21612

     

    KB64659 - Secure Mail/Email Gateway: Using the CLI to view binary logs with the showevents command - 6.7.1 and later

     

    KB64625 - Secure Mail/Email Gateway: Tailing SMTPProxy Log from the CLI - 6.7.1 and later

     

    KB64599 - Secure Mail/Email Gateway: How to Use Show Events - Logging and CLI Changes in 6.7.1 and later

     

    Secure Mail 6.7.1 ShowEvents Utility

    Go to the McAfee / Secure Computing Support Center ("https://supportcenter.ciphertrust.com/index.php") and logon. If you don't have an account, you can request an account on the site but you'll probably have to call the Support Center Team to have the account activated (800-700-8328 x7 for Customer Service); After you are logged on, look under "Downloads"


    I hope that helps.  If anyone else has useful tips and tricks on using the CLI in v6.7.1+, or more accurate information than what I've posted, please share.  Thanks!


    ---
    Start/RunCMD...
    C:\>
    ePO v4.5 / MA v4.6.0 / VSE v8.8, P1 / Engine v5400
    MEG (IronMail) v7.6-2810
  • runcmd Apprentice 221 posts since
    Feb 22, 2006
    Currently Being Moderated
    5. May 20, 2010 1:51 PM (in response to wormnrifle)
    Re: v6.7.1+ CLI Commands

    In working with support, I was able to determine that you can setup the binary file to FTP automatically every night...

     

    Reporting > Reporting Advanced > Logs

     

    "Internal - Binary Log" is second from the bottom on the list in my GUI.  You can also click on the link to "Download" the file manually through the GUI.  I played with this for a while, yesterday, figuring out the "Secure Mail 6.7.1 ShowEvents Utility".  Don't expect the "showevents_win32.exe" to be any more user friendly than the CLI's "showevents", as it uses relatively the same syntax.

     

    The binary file is downloaded from the appliance as "scmail-logs.bin.tgz".  I used 7Zip to extract "scmail-logs.bin.tar" from that archive and then used 7Zip, again, to extract "scmail-logs.bin" and "scmail-logs.idb" from that tar file.  The "scmail-logs.bin" file appears to be the binary log file you need.  I copied both "scmail-logs.bin" and "scmail-logs.idb" to the same folder as the Windows "Secure Mail 6.7.1 ShowEvents Utility" and ran:

     

    showevents_win32 -s cfile="scevents.ini" -s ifile="scmail-logs.bin" -g module=smtpproxy -d head>smtpproxy.txt

     

    Running the above command floods my screen with "There is no such event id" but it does appear to redirect the valuable information to the file for me.  This gives me a (large) text file named "smtpproxy.txt" where the really useful information is.  You can then use a utility like BareGrep to search that file for sender's address, recipient's address, IP address, message ID, or whatever you want.  Or, if you don't want to use a 3rd party utility like BareGrep, just use Microsoft's "find" or "findstr".

     

    Technical Support was also able to confirm that "-i", to ignore case, no longer functions when used with grep in the CLI on v6.7.2.  I submitted a "Feature Request" asking for it back.  (I'm not holding my breath.)

     

    Hope that helps!


    ---
    Start/RunCMD...
    C:\>
    ePO v4.5 / MA v4.6.0 / VSE v8.8, P1 / Engine v5400
    MEG (IronMail) v7.6-2810
  • Newcomer 1 posts since
    Jun 23, 2010
    Currently Being Moderated
    6. Jun 23, 2010 3:37 PM (in response to runcmd)
    Re: v6.7.1+ CLI Commands

    Hello,

        I'd like to direct you to the "showevents" command, which is not to be confused with "show events"

     

    A full command syntax can be found in KB64659

    http://kb.mcafee.com/agent/index?page=content&id=KB64659

     

     

        In 6.7.x the development team implemented binary logging on the appliances. This means the standard log data is stored in a flat, indexed database file. Basically, it allows us to pull specific data much more quickly, and from one location. Previously, if the data you wanted did not appear in the summary log, you would have to know which file contained the data you were looking for. For example, if an email was delivered to the appliance and never made it out, you may need to read the smtpproxy, ripq, cfq, ccq, superq, vfq, and smtpo logs to find the log data which explained exactly what happened to the message. Now, we can reference all of this data in one binary logging file. Also, if you already know the event ID, connection ID, or message ID that pertains to whatever data you are looking for the showevents command will return the data many, many times faster than with standard log files.

     

       Including all this expanded functionality, and increased efficiency in one command requires adding some command arguments, which may seem a little complicated at first. I have outlined a simple explanation of "how it works" for the showevents command from the CLI.

     

     

    ---

     

    All commands will start with this (you can copy/paste it for anything you need to do, this will return all log data):

    showevents -s cfile=/conf/scevents.ini -s ifile=/log/scmail-logs.bin -d head

     

    Or This (for logs other than today):

    showevents -s cfile=/conf/scevents.ini -s ifile=/log/scmail-logs.bin.ends20100427 -d head

     

     

     

    You can add a -g modifier (most common modifiers, many other modifiers are listed in the above KB article):

     

    -g module=smtpproxy

    -g module=superq

    -g module=smtpo

    -g msgid=xxxxxx

    -g eid=xxxxxx

     

    OR just grep the output for what you are looking for (You can use -g commands and still include |grep <whatever> at the end, just like any other unix/linux command line)

     

    |grep user@domain

     

     

    Every line of the log file will have a connection ID or message ID as well as an event ID. Once you figure out which one belongs to your message, you should use the -g msgid=<msg or connection ID> option to get all the other info about the connection or message or the "-g eid=<event ID>" to pull out all occurrences of a specific type of event.

     

    Connection ID (and event ID):

    20080901:00:19:09|22590300344740|9308|Message information <Source IP:Port:MessageID>|10.0.0.36:58228:4904983|

    20080901:00:19:09|<CONNECTION ID NUMBER>|<EVENT ID NUMBER>|Message information <Source IP:Port:MessageID>|10.0.0.36:58228:4904983|

     

    Message ID (and event ID):

    20090113:00:09:54|137|9233|Processing started.|

    20090113:00:09:54|<MESSAGE ID NUMBER>|<EVENT ID NUMBER>|Processing started.|

    Specific examples of finding specific data are included in the above KB article.
    ---

     

     

       In short: if you are looking for a little bit of data, show events |grep "whatever" is probably the easiest command to return what you are looking for. However, administrators who either create reports based on log data, or need to formulate complicated commands will find "showevents" is a much more efficient way of getting exact data from the appliance.

     

     

    *Note: grep arguments were removed in 6.7.2 for security reasons. Support has already opened a feature modification request to have grep -i, grep -v, and grep -c added back to the CLI.

     

     

    Message was edited by: Kevin Floda on 6/23/10 3:37:18 PM CDT
  • runcmd Apprentice 221 posts since
    Feb 22, 2006
    Currently Being Moderated
    7. Sep 8, 2010 8:22 AM (in response to kfloda)
    Re: v6.7.1+ CLI Commands

     

    *Note: grep arguments were removed in 6.7.2 for security reasons. Support has already opened a feature modification request to have grep -i, grep -v, and grep -c added back to the CLI.

     

     

    Interestingly, although a grep through the CLI is case sensitive, a search filter through the GUI does not appear to be case sensitive.  For example, if I issue the following two commands through the CLI...

     

    tail events | grep "Connection accepted"

    tail events | grep "Connection Accepted"

     

    ...The first will yield results but the second will not because of case.  However, if I setup a filter through the GUI...

     

    Reporting > Reporting Advanced > Logs > View Log : Internal - Binary Log

    Search Filter = Connection Accepted

     

    ...I still get results--even though the capital "A" of "Accepted" in the Search Filter is being comparted to a lower case "a" in my logs.

     

    The only down side is that the view through the GUI appears to only perform tailing and not a search of past events in the binary log.  For searching past events and ignoring case, you'd still need to download the binary and refer back to the "showevents_win32" utility mentioned in my previous post.  But if you are just tailing the logs and need to ignore case while searching for something, the GUI Search Filter is a nice workaround.


    ---
    Start/RunCMD...
    C:\>
    ePO v4.5 / MA v4.6.0 / VSE v8.8, P1 / Engine v5400
    MEG (IronMail) v7.6-2810
  • ijahnke McAfee Employee 118 posts since
    May 12, 2010
    Currently Being Moderated
    8. Sep 8, 2010 8:41 AM (in response to runcmd)
    Re: v6.7.1+ CLI Commands

    There are a couple options here actually:

     

    The first is:

    McAfee]: show events | grep [A-a]ccepted

    20100908:05:01:45|22851189970310|9236|Connection accepted.||
    20100908:05:01:47|14590220|4139|-|Reply:  '250 Accepted'|

     

     

     

    Another option you would have for looking for case insensitive would be to use the "."

     

    In Unix, the period is considered a wildcard that matches any character.

     

    Here would be an example looking for the word "[C-c]onnected"

     

     

    [McAfee]: tail events | grep .onnection

    20100908:09:29:09|22851255687540|9236|Connection accepted.||
    20100908:09:29:09|22851255675240|9312|Socket communication failed with client. Connection dropped||
    20100908:09:29:09|22851255687541|9236|Connection accepted.||
    20100908:09:29:09|22851255675237|9312|Socket communication failed with client. Connection dropped||
    20100908:09:29:09|14782315|9492|Connection Status <status> -|1|
    20100908:09:29:09|14782319|9492|Connection Status <status> -|1|
    20100908:09:29:09|22851255683441|9312|Socket communication failed with client. Connection dropped||
    20100908:09:29:10|22851255683442|9312|Socket communication failed with client. Connection dropped||
    20100908:09:29:10|22851255691638|9236|Connection accepted.||
    20100908:09:29:10|22851255675239|9312|Socket communication failed with client. Connection dropped||

     

     

    so if youre looking for either "Accepted" or "accepted" you could run this command:

     

    [McAfee]: show events | grep .ccepted

    20100908:05:01:45|22851189970310|9236|Connection accepted.||
    20100908:05:01:47|14590220|4139|-|Reply: '250 Accepted'|

     

     

     

    Hope this helps

     

    --Ivan

     

     

    Message was edited by: Ivan Jahnke on 9/8/10 8:40:59 AM CDT

     

     

    Message was edited by: Ivan Jahnke on 9/8/10 8:41:23 AM CDT
1 2 Previous Next

More Like This

  • Retrieving data ...

Bookmarked By (1)

Legend

  • Correct Answers - 5 points
  • Helpful Answers - 3 points