3 Replies Latest reply on Feb 4, 2015 8:07 AM by jasonhull

    Dynamic Bad IP Watchlist Automation with Powershell and SIEM


      The following sites display a text like list of known Bad IP's, which is easy to automate, parse and import into SIEM as a Dynamic Watchlist.











      I have built a powershell script that goes to a saved file of the sites above, and scrapes those sites, for the IP's and saves them to a file location.


      After the powershell script executes daily and it saves an individual file per site above. I then created a dynamic watchlist in SIEM that updates daily based on the new files written.


      Below is a generic script I have built for everyone to use, every line has been commented at the end with a "###" to display what that line is doing.


      $sitelist=Get-Content 'c:\locationoffile\filename.txt' #Change Location of the File Containing Sites providing 1 single column of Bad IP's

      $web=New-Object Net.WebClient

      $ipregex="(\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[ 0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[ 0-9][0-9]?)\b)"###RegEx for IP Matching to Filter out characters###

      $folderpath='c:\destinationfolderlocation\' #Change location to the folder path you wish to store the IP Files

      Remove-Item 'c:\destinationfolderlocation\*' -Force ##Removes old files to build new ones##

      foreach ($site in $sitelist){ #imports Sitelist

        $filename=$site.Split("///") #building filename based on domain name of site

        $filename=$filename -replace('\.','-') #building filename based on domain name of site

        $filename=$filename[2] #building filename based on domain name of site

        $filename=$filename+"-raw.txt" #adding "raw" to the end of the filename for the initial Site download

        $fullfilepath=$folderpath+$filename #full file path of the first "raw" output

        $web.DownloadFile($site,$fullfilepath) #downloads site, outputs file to the full path declared on line 11

        $cleanoutputfile=$fullfilepath -replace("-raw","") #declares the new file name it will output to, containing the clean IP information

        Get-Content -Path $fullfilepath | #imports the raw file

        Select-String -Pattern $ipregex | #will select strings only containing IP address, will filter out commented out sections, and other non-IP information

        Set-Content $cleanoutputfile #outputs the data to the new clean file, this will be the file you will import into the SIEM dynamic watchlist

        Remove-Item $fullfilepath



      All lines above highlighted in GREEN should be changed to reflect your environment:


                     Line 1: Update ‘c:\locationoffile\filename.txt’ to the location they have imported the text file of the Sites in item number 1. Example: c:\users\username\desktop\SIEM\cleanIPsitelist.txt

                     Line 4: Update ‘c:\destinationfolderlocation\’ to the folder as to which you want your files to export to once complete

                     Line 5: Update ‘c:\destinationfolderlocation\*’ to the same as line 4 (Keep the * at the end, as this is a cleanup function)


      I recommend after editing the parameters to test run this, look in the destination folder to make sure everything has worked properly.


      There will be a file per site with the format of “domainname.txt” Example: “rules-emergingthreats-net.txt”


      When choosing the output folder location, it’s key to remember the SIEM needs to have access to that share in order to actively get the updated information, whatever schedule is set:


      screen 1.jpg


      In order to run this script you need to change your powershell settings Execution Policy. If this is locked down in your environment, you will need to sign this script with a certificate of your own in order to execute. If you have access to change the Execution Policy, you can change it by entering into your powershell profile “Set-ExecutionPolicy RemoteSigned”, keep in mind, this is not a completely secure manner to allow scripts to run, but if this acceptable in your environment, this is how you will get it to execute.

      More information on Execution Policies can be read about here:



      I have also automated this via task scheduler built into Windows.


      Once scheduled and the files are output to the preferred directory, you can build the integration with SIEM.


      Once logged in to your SIEM appliance, click on System Properties>Watchlists>Add


      I recommend creating a watchlist per site that contains the Bad IP information, with the Name to the site providing the information like below:


      Choose "Dynamic", and a Time you wish it to update, and occurrence.


      screen 2.jpg

      In the script provided it saves to a CIFS share called SIEM (\\servername\SIEM)


      Enter the Valid IP, Share Name, Path Name will be the file name of the text file containing IP Information, and a Valid User ID and Password

      screen 3.jpg


      In my implementation I chose the "Type:" as IP Address, so I could apply this whitelist in my dashboards as a source or destination IP filter.


      After you choose the "Type:", you can click "Run Now" and it should populate a list similar to the one below.


      screen 4.jpg


      You can now apply this to a dashboard, and it can show you alerts for all activity going from a specific device or all devices going to those malicious IP's.