6 Replies Latest reply on Sep 7, 2017 3:36 AM by d_aloy

    Looking for a way to get full packet capture (pcap) data from NSM Database

    sonofliberty

      Hello Everyone,

       

      We have developed a custom script to deal with .pcap files and do some actions with them on a regular basis. The question was how to get .pcap files from McAfee NSM database.

      Having looked at the API, we have found that there's no way to export alert capture, only sensor packet capture which is a different thing.


      At the same time, we were not able to get .pcaps straight from McAfee NSM database too. Query we used is:

      SELECT iv_alert.SensorAlertUUID, iv_packetlog.packetData FROM iv_alert LEFT JOIN iv_packetlog ON iv_alert.packetLogId = iv_packetlog.packetLogId

       

      The problem is that there is layer7data only presented in iv_packetlog.packetData field, not the whole capture (Ports, IP Addresses, MAC addresses etc)

       

      But, as we can see from McAfee NSM Interface, if you go to Analysis tab -> Threat Explorer -> View Attacks, find the attack there and click on "Export" link, you will get the whole .pcap will all the needed data. How is this thing working then?


      Is there any way to export full packet capture (.pcap, not just layer7 data) with one query, like we do this by clicking on a "Export capture" link in NSM Web interface?

       

       

       

      Thank you.

        • 1. Re: Looking for a way to get full packet capture (pcap) data from NSM Database
          peter.mason

          Hi Sonofliberty,

           

          The packetData field is a longblob type field, you need to figure out how to write the data back out of it.

           

          Regards

           

          Peter

          1 of 1 people found this helpful
          • 2. Re: Looking for a way to get full packet capture (pcap) data from NSM Database
            d_aloy

            And to add what Peter said, on top of being on a blob format, I believe it is encrypted as well, so you will need to figure out how to get something you can actually read.

             

            This is doable though, as McAfee ESM (SIEM) will show you the pcap details on the ESM interface.

             

            We have asked for a PER on this, so that the evidence report pcap can be pulled 'on demand' from 3rd party SIEMs.

             

            Regards

            David

            1 of 1 people found this helpful
            • 3. Re: Looking for a way to get full packet capture (pcap) data from NSM Database
              sonofliberty

              Guys,

              Thank you for your kind answers.

               

              As there's no easy way to achieve this, I've written script that pulls new packet capture information from the database and then retrieves PCAPs one-by-one via HTTP interface. Hope it will be helpful.

               

               

              #!/usr/bin/python
               # Script to retrieve PCAP files from McAfee database and save them to a folder with %alert_id name
               # import modules
              import MySQLdb
              import time
              import requests
              
              
               # define main function
              def getPCAP():
                  # define main variables
                  folder = "C:\PCAPS"  # folder to which we save files
                  db_name = "nsmdb_02"  # NSM database name
                  db_user = "root"  # NSM Database user
                  db_passwd = "pass"  # NSM Database password
                  nsm_host = "nsm_http"  # NSM Hostname
                  nsm_login = "nsm_login"  # Login to access NSM HTTP Interface
                  nsm_password = "nsm_pass"  # Password to access NSM HTTP Interface
                  while True:
                      connection = MySQLdb.connect(host="localhost", user=db_user, passwd=db_passwd,
                                                   db=db_name)  # Connecting to database
                      cursor = connection.cursor()
                      cursor.execute("SELECT MAX(creationTime) from iv_packetlog;")  # getting last packet creation time
                      lasttime = cursor.fetchone()  # Fetching last packet creation time
                      # execute the SQL query using execute() method.
                      try:
                          executedBefore  # Check if loop has been executed before
                      except NameError:
                          print "This is a new run, so I will load PCAPs for 1 minute ago only. Executing query..."
                          query = "SELECT iv_alert.sensorId, iv_alert.sensorAlertUUID, iv_packetlog.packetData \
               FROM iv_alert LEFT JOIN iv_packetlog ON iv_alert.packetLogId = iv_packetlog.packetLogId WHERE iv_packetlog.creationTime > DATE_SUB(NOW() , INTERVAL 1 MINUTE) AND iv_packetlog.creationTime <= %s"
                          args = (lasttime)
                          cursor.execute(query, args)
                          lastquerytime = lasttime
                      else:
                          print "Found previous execution time. Executing query starting from %s" % lastquerytime
                          query = ("SELECT iv_alert.sensorId, iv_alert.sensorAlertUUID, iv_packetlog.packetData FROM iv_alert \
               LEFT JOIN iv_packetlog ON iv_alert.packetLogId = iv_packetlog.packetLogId WHERE iv_packetlog.creationTime > %s AND iv_packetlog.creationTime <= %s")
                          args = (lastquerytime + lasttime)
                          cursor.execute(query, args)
                          lastquerytime = lasttime
                      executedBefore = 1
                      # fetch all of the rows from the query
                      data = cursor.fetchall()
                      print "Connecting to NSM to obtain PCAPs..."
                      s = requests.Session()  # Establishing HTTPS Session to grab PCAPs
                      http_data = {"iaction": "login", "node": "", "bwVer": "999", "Login%20ID": nsm_login, "password": nsm_password}
                      url = "https://" + nsm_host + "/intruvert/jsp/module/Login.jsp"
                      headers = {"Host": nsm_host,
                                 "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0",
                                 "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                                 "Accept-Language": "en-US,en;q=0.5", "Content-Type": "application/x-www-form-urlencoded",
                                 "Upgrade-Insecure-Requests": "1"}
                      request = s.post(url, headers=headers, data=http_data, cookies=s.cookies, verify=False)
                      # print the rows to file with name as AlertID(uuid) which is unique to each message
                      for row in data:
                          SensorID = row[0]
                          AlertID = row[1]
                          PCAP_URL = "https://" + nsm_host + "/intruvert/action/AlertLogAction?userAction=getPacketCapture&sensorId={0}&sensorAlertUUID={1}&topMenuName=INVESTIGATIONX&topMenuName=INVESTIGATIONX&secondMenuName=Threat%20Explorer&thirdMenuName=Threat%20Explorer&description=Threat%20Explorer&helpId=GUID-46AF9550-083C-4331-ABE0-4634416213BD&resourceName=%2FMy%20Company%3A0%2FInvestigation%3A0&shortResourceName=%2FInvestigation%3A0&domainName=%2FMy%20Company%3A0&currentDomainName=%2FMy%20Company%3A0&domain=false&vidsId=0&sensorName=&accessRight=fullaccess&breadcrumb=%2FMy%20Company%20%3E%20Threat%20Explorer&moduleId=13&isRootDomain=true&selectedDomain=%2FMy%20Company%3A0&OWASP_CSRFTOKEN=OACG-08MX-UJOT-64AO-SNOJ-L6WR-9NQ7-EAOA&OWASP_CSRFTOKEN=OACG-08MX-UJOT-64AO-SNOJ-L6WR-9NQ7-EAOA&nsmVersion=8.3.7.52&nsmVersion=8.3.7.52&Module_id=13&csrftokenname=OWASP_CSRFTOKEN&includeChildDomains=false&extjsDebugEnable=false&localeLnStr=".format(
                              SensorID, AlertID)
                          PCAP = s.get(PCAP_URL, cookies=s.cookies)
                          with open(folder + "\{0}.pcap".format(AlertID), "wb") as pcap_file:
                              pcap_file.write(PCAP.content)
                              # close the cursor object
                      cursor.close()
                      print "\nLast Query time set to %s" % lastquerytime
                      print "\n Waiting for 10 seconds ... \n"
                      time.sleep(10)
              
              
               getPCAP()
              
              • 4. Re: Looking for a way to get full packet capture (pcap) data from NSM Database
                d_aloy

                Hi all

                 

                I need to correct myself on this thread....

                 

                The blob packetdata on iv_packetlog is not encrypted....

                So it can be pulled from the db, but I found out that the packetdata field does not contain L2 to L4 headers, just L7 data.

                 

                I've logged a support case and will share the full query or procedure once I have it.

                 

                Regards,

                David

                • 5. Re: Looking for a way to get full packet capture (pcap) data from NSM Database
                  d_aloy

                  Guys...

                   

                  Guess what? Anyone checked the docs for this? (not me...definitely!)

                   

                  The Integaration Guide explains how to get the pcap from the db in detail:

                   

                  https://kc.mcafee.com/resources/sites/MCAFEE/content/live/PRODUCT_DOCUMENTATION/ 26000/PD26349/en_US/NSP_83_Integration_G…

                  Page 255 of the PDF or search for "Create PCAP format packet logs"

                   

                  Regards

                  David

                  • 6. Re: Looking for a way to get full packet capture (pcap) data from NSM Database
                    d_aloy

                    Still the issue with the L2 to L4 headers remains... I am able to pull the packetdata but that seems to be L7 only, and I don't think 'creating' headers would be accepted by any law enforcement agencies...

                     

                    I've asked about this and will update the thread as soon as I have more info.

                     

                    BTW, the query I've been working on is this, just FYI:

                    # mysql -h -u -p --database lf -e "SELECT packetData FROM iv_packetlog WHERE packetLogId =6454434134835140992;"| od -Ax -t x1z -v

                     

                    This is from a 'remote' linux box, so you need to make sure the user and password and systems have the correct grant privileges on the mysql db.

                    Regards

                    David