Web Gateway Cloud Service: Cloud Log Puller for Windows (Powershell Script)

[#1]

• Introduction
• Troubleshooting

Introduction

If you need to pull logs out of the Web Gateway Cloud Service here is a script which can be used to do just that.

# Collect logs from last 4 hours

.\CloudLogPullerLatest.ps1 -CustomerID XXXXXXXXXX -User EPO_CLOUD_USER -Hours 4

# Collect logs from last 7 days in chunks of 5 minutes (300 seconds) and Ignore Certificate Warnings

.\CloudLogPullerLatest.ps1 -CustomerID XXXXXXXXXX -User EPO_CLOUD_USER -Days 7 -Interval 300 -IgnoreCerts

Features

• Interval based collection (useful for when there is a large amount of data over a certain time period)
• Collect over last X days, hours, minutes
• Error handling
• Logging
• Checks for updates to script

Link to the zip file with the script:

https://community.mcafee.com/nysyc36988/attachments/nysyc36988/business-documents/1767/1/CloudLogPul...

Screenshot of success:

Screenshot of failure:

Troubleshooting

The script generates a log file which can be used to understand the problem. If you have any issues start a Community thread.

NOTE: This tool is not supported by McAfee Technical Support in any way. Do not contact them for help with problems.

[tarunmann]

What do you think about these errors ​

[04/Jan/18 16:16:25 -0700] MFE-CheckForUpdates: ERROR: Not sure what happened here...

1. System.Management.Automation.CommandNotFoundException: The term 'Invoke-WebRequest' is not recognized as the name of a c

mdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify tha

t the path is correct and try again.

[04/Jan/18 16:16:25 -0700] MFE-DownloadLogs: ERROR: Not sure what happened here...

1. System.Management.Automation.CommandNotFoundException: The term 'Invoke-RestMethod' is not recognized as the name of a c

mdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify tha

t the path is correct and try again.

[hiwai]

I cannot download the powershell script  from the link... Could you fix it?

[feickholt]

Unfortunately the script doesn't work for me. Here is the extract from the logfile

[05.Mrz.18 13:36:38 +0100] MFE-Startup: INFO: Starting up Cloud Log Puller version 0.6
[05.Mrz.18 13:36:38 +0100] MFE-GetAccountCredentials: INFO: Customer ID: 9999999
[05.Mrz.18 13:36:38 +0100] MFE-GetAccountCredentials: INFO: Username: xxxxxxxxxxxxx
[05.Mrz.18 13:36:38 +0100] MFE-GetAccountCredentials: INFO: Duration: 14400
[05.Mrz.18 13:36:38 +0100] MFE-GetAccountCredentials: INFO: Interval: 600
[05.Mrz.18 13:36:38 +0100] MFE-CheckForUpdates: INFO: Checking for updates at ftp://custftp2.nai.com...
[05.Mrz.18 13:36:39 +0100] MFE-CheckForUpdates: INFO: You are up to date (Your Version: 0.6)
[05.Mrz.18 13:36:39 +0100] MFE-IgnoreCertWarnings: Ignoring certificate warnings
[05.Mrz.18 13:36:39 +0100] MFE-DownloadLogs: WARN: Start collect -0000 to -0000
[05.Mrz.18 13:36:39 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:41 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:41 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:42 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:42 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:43 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:43 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:44 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:44 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:45 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:45 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:46 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:46 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:47 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:47 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:48 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:48 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:49 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:49 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:50 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:50 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:51 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:51 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:51 +0100] MFE-DownloadLogs: INFO: Collecting logs -0000 to -0000
[05.Mrz.18 13:36:52 +0100] MFE-DownloadLogs: WARN: Finished collecting logs
[05.Mrz.18 13:36:52 +0100] MFE-DeleteEmptyFiles: WARN: Deleting Empty Log Files

[feickholt]

I think I found the error:

in MFE-GetEpoch the time must be devided by 100000 otherwise since the function returns ms instead of sec.

Function MFE-GetEpoch
{
$ED=[Math]::Floor([decimal](Get-Date(Get-Date).ToUniversalTime()-uformat "%s") / 100000)
return $ED
}

[jebeling]

Well I understand the link isn't working for some and I can't do an attachment here so..... I don't know whether or not it needs the fix mentioned by @feickholt above

Param (
[Parameter(Mandatory=$false)][string]$CustomerID,
[Parameter(Mandatory=$false)][string]$Username,
[Parameter(Mandatory=$false)][int]$Interval,
[Parameter(Mandatory=$false)][int]$Minutes,
[Parameter(Mandatory=$false)][int]$Hours,
[Parameter(Mandatory=$false)][int]$Days,
[Parameter(Mandatory=$false)][switch]$IgnoreCerts,
[Parameter(Mandatory=$false)][switch]$Help
)
<#
Parameters MUST be at the top of the powershell script.
#>

#$global:Password = 'PasswordHereIfYoureLazy'

# In what time increments should we collect logs?
$global:Interval = 600;
# How far back should we collect log logs?
$global:Duration = 3600;

#############################################
# Web Gateway Cloud Service Log Puller Script v0.5
#
# Collect logs from Web Gateway Cloud Service (last 4 hours):
# .\CloudLogPullerLatest.ps1 -CustomerID XXXXXXXXXX -User EPO_CLOUD_USER -Hours 4 -IgnoreCerts
#
# Collect logs from Web Gateway Cloud Service (last 7 days in chunks of 5 minutes):
# .\CloudLogPullerLatest.ps1 -CustomerID XXXXXXXXXX -User EPO_CLOUD_USER -Days 7 -Interval 300 -IgnoreCerts
#
#############################################
$global:ScriptVersion = '0.6';

$global:CustomerID = $CustomerID;
$global:Username = $Username;
$global:UserInterval = $Interval;
$global:Duration = $Duration;
$global:Minutes = $Minutes;
$global:Hours = $Hours;
$global:Days = $Days;
$global:IgnoreCerts = $IgnoreCerts;
$global:Help = $Help;
Clear-Variable -name CustomerID
Clear-Variable -name Username
Clear-Variable -name Interval
Clear-Variable -name Duration
Clear-Variable -name Minutes
Clear-Variable -name Hours
Clear-Variable -name Days
Clear-Variable -name IgnoreCerts
Clear-Variable -name Help

Function MFE-Startup
{
clear

If ($global:Help.IsPresent)
{
MFE-ShowHelp;
}

$global:CloudApiVersion = '3';
$global:FatalError = $false;
$global:Server = 'msg.mcafeesaas.com';
$global:ScriptPath = $PSScriptRoot;
$global:MfeLogFile = -Join($global:ScriptPath, '\CloudLogPuller.log');
$global:DownloadDirectory = -Join($global:ScriptPath, '\Logs');
If (!(Test-Path $global:DownloadDirectory))
{
$r = New-Item -ItemType Directory -Force -Path $global:DownloadDirectory
}

$global:VersionCheckUrl = "https://technicalmarketing.mcafee.com/scripts/CloudLogPullerVersion.txt"
$global:DownloadUrl = "https://technicalmarketing.mcafee.com/scripts/CloudLogPullerLatest.ps1"

Write-Host @"

*************************************************
* *
* Cloud Log Puller $global:ScriptVersion *
* *
* (This script will download Web Gateway *
* Cloud Service Logs) *
* *
*************************************************

"@
MFE-GetAccountCredentials;
MFE-CheckForUpdates;
If ($global:IgnoreCerts.IsPresent)
{
MFE-IgnoreCertWarnings;
}
MFE-DownloadLogs;
MFE-DeleteEmptyFiles;
MFE-Cleanup;
}

Function MFE-GetAccountCredentials
{
# Prompt for credentials if they are not set
If($global:CustomerID -eq $null -Or $global:CustomerID -eq '')
{
$global:CustomerID = Read-Host -Prompt 'Customer ID'
}
Else
{
echo "Customer ID: $global:CustomerID"
}

If($global:Username -eq $null -Or $global:Username -eq '')
{
$global:Username = Read-Host -Prompt 'Username'
}
Else
{
echo "Username: $global:Username"
}

If($global:Password -eq $null -Or $global:Password -eq '')
{
$SecurePassword = Read-Host -Prompt "Password" -AsSecureString
$SecString = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($SecurePassword)
$global:Password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($SecString)
}
Else
{
echo "Password: [Using Password from Script]"
}

If ($global:Days -ne $null -And $global:Days -ne '')
{
$global:Duration = [int]$global:Days * 86400;
echo "Collection Duration: $global:Days days ($global:Duration seconds)"
}
ElseIf ($global:Hours -ne $null -And $global:Hours -ne '')
{
$global:Duration = [int]$global:Hours * 3600;
echo "Collection Duration: $global:Hours hours ($global:Duration seconds)"
}
ElseIf ($global:Minutes -ne $null -And $global:Minutes -ne '')
{
$global:Duration = [int]$global:Minutes * 60;
echo "Collection Duration: $global:Minutes minutes ($global:Duration seconds)"
}

If ($global:UserInterval -ne $null -And $global:UserInterval -ne '')
{
$global:Interval = $global:UserInterval;
}
echo "Collection Interval: $global:Interval seconds"
echo "`n";

MFE-Logger -Message "INFO: Starting up Cloud Log Puller version $global:ScriptVersion" -Func 'MFE-Startup'
MFE-Logger -Message "INFO: Customer ID: $($global:CustomerID)" -Func $MyInvocation.MyCommand
MFE-Logger -Message "INFO: Username: $($global:Username)" -Func $MyInvocation.MyCommand
MFE-Logger -Message "INFO: Duration: $($global:Duration)" -Func $MyInvocation.MyCommand
MFE-Logger -Message "INFO: Interval: $($global:Interval)" -Func $MyInvocation.MyCommand


$global:base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $global:Username,$global:Password)))
Clear-Variable -name Password -Scope Global
Clear-Variable -name Username -Scope Global
}

Function MFE-DownloadLogs
{
<#
Download Logs from Web Gateway Cloud Service
#>
$TimeTo = MFE-GetEpoch;
$TimeTo = $TimeTo - ($TimeTo % 1800)
$TimeFrom = $TimeTo - $global:Duration;
$CycleFrom = $TimeFrom;
$CycleTo = $CycleFrom + $global:Interval;

MFE-Logger -Message "WARN: Start collect $(EpochToTimestamp -UnixTime $TimeFrom) to $(EpochToTimestamp -UnixTime $TimeTo)" -Func $MyInvocation.MyCommand -Level 'warn'

$HttpHeaders = @{
'Authorization' = ("Basic {0}" -f $global:base64AuthInfo)
'Accept' = "text/csv"
'User-Agent' = "Cloud Log Puller v$($global:ScriptVersion)"
'X-MFE-DownloadFileName' = "Forensic Report.csv"
'X-MFE-CSVRowXPath' = "//request"
'X-MWG-API-Version' = "$global:CloudApiVersion"
}

Do
{
MFE-Logger -Message "INFO: Collecting logs $(EpochToTimestamp -UnixTime $CycleFrom) to $(EpochToTimestamp -UnixTime $CycleTo)" -Func $MyInvocation.MyCommand -Level 'info'
$uri = "https://$($global:Server):443/mwg/api/reporting/forensic/$($global:CustomerID)?filter.requestTimestampFrom=$($CycleFrom)&filter.requestTimestampTo=$($CycleTo)&order.0.requestTimestamp=asc";
$response = try {
Invoke-RestMethod -Headers $HttpHeaders -TimeoutSec 60 -Uri $uri -OutFile "$($global:DownloadDirectory)\CloudLog_$($CycleFrom).csv"
}
catch {
$StatusCode = $_.Exception.Response.StatusCode.value__ ;
$StatusDescription = $_.Exception.Response.StatusDescription
$WebException = $_.Exception
MFE-CheckForHttpError -StatusCode $StatusCode -WebException $WebException -Func $MyInvocation.MyCommand -UrlHost $global:Server
}

$CycleFrom = $CycleTo + 1;
$CycleTo = $CycleTo + $global:Interval;
} While (!$global:FatalError -And $TimeTo -gt $CycleTo)

If (!$global:FatalError)
{
MFE-Logger -Message "INFO: Collecting logs $(EpochToTimestamp -UnixTime $CycleFrom) to $(EpochToTimestamp -UnixTime $TimeTo)" -Func $MyInvocation.MyCommand -Level 'info'
$uri = "https://$($global:Server):443/mwg/api/reporting/forensic/$($global:CustomerID)?filter.requestTimestampFrom=$($CycleFrom)&filter.requestTimestampTo=$($TimeTo)&order.0.requestTimestamp=asc";
$response = try {
Invoke-RestMethod -Headers $HttpHeaders -TimeoutSec 60 -Uri $uri -OutFile "$($global:DownloadDirectory)\CloudLog_$($CycleFrom).csv"
}
catch {
$StatusCode = $_.Exception.Response.StatusCode.value__ ;
$StatusDescription = $_.Exception.Response.StatusDescription
$WebException = $_.Exception
MFE-CheckForHttpError -StatusCode $StatusCode -WebException $WebException -Func $MyInvocation.MyCommand -UrlHost $global:Server
}
}

MFE-Logger -Message "WARN: Finished collecting logs" -Func $MyInvocation.MyCommand -Level 'warn'
}

Function EpochToTimestamp ([long]$UnixTime)
{
$epoch = New-Object System.DateTime (1970, 1, 1, 0, 0, 0, [System.DateTimeKind]::Utc);
$time = $epoch.AddSeconds($UnixTime);
#$timestamp = "[{0:dd/MMM/yy} {0:HH:mm:ss} -0000]" -f ($time)
$timestamp = "{0:dd/MMM/yy} {0:HH:mm:ss} -0000" -f ($time)
return $timestamp;
}

Function MFE-DeleteEmptyFiles
{
MFE-Logger -Message "WARN: Deleting Empty Log Files" -Func $MyInvocation.MyCommand -Level 'warn'
#Get-ChildItem $global:DownloadDirectory -Filter CloudLog_*.csv -recurse |?{$_.PSIsContainer -eq $false -and $_.length -lt 5}|?{Remove-Item $_.fullname -WhatIf}
Get-ChildItem $global:DownloadDirectory -Filter CloudLog_*.csv -recurse |?{$_.PSIsContainer -eq $false -and $_.length -lt 5}|?{Remove-Item $_.fullname}
}

Function MFE-CallApi
{
Param ( [string]$uri, [string]$Func, [string]$Method, [string]$Body, [string]$ContentType )

$response = try {
If($Method -ne $null -And $Method -ne '')
{
Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $global:base64AuthInfo)} -TimeoutSec 5 -Uri $uri -Method $Method -Body $Body -ContentType $ContentType
}
Else
{
Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $global:base64AuthInfo)} -TimeoutSec 5 -Uri $uri
}
}
catch {
$StatusCode = $_.Exception.Response.StatusCode.value__ ;
$StatusDescription = $_.Exception.Response.StatusDescription
$WebException = $_.Exception
MFE-CheckForHttpError -StatusCode $StatusCode -WebException $WebException -Func $Func -UrlHost $global:Server
}
}

Function MFE-CheckForHttpError
{
Param ( [string]$StatusCode, [string]$WebException, [string]$Func, [string]$UrlHost )
#Write-Host $StatusCode;
#Write-Host $WebException;

If ($StatusCode -eq '401')
{
MFE-Logger -Message "ERROR: Unauthorized, check username/password (Status Code: 401)" -Func $Func -Level 'error'
$global:FatalError = $true;
}
ElseIf ($StatusCode -eq '403')
{
MFE-Logger -Message "ERROR: Request Blocked (Status Code: 403)" -Func $Func -Level 'error'
}
ElseIf ($StatusCode -like '40x')
{
MFE-Logger -Message "ERROR: Other failure (Status Code: $($StatusCode))" -Func $Func -Level 'error'
}
ElseIf ($WebException -like "*File unavailable*")
{
MFE-Logger -Message "ERROR: File not found" -Func $Func -Level 'error'
}
ElseIf ($WebException -like "*The remote name could not be resolved*")
{
MFE-Logger -Message "ERROR: DNS Failure ($UrlHost)" -Func $Func -Level 'error'
}
ElseIf ($WebException -like "*The handshake failed due to an unexpected packet format*")
{
MFE-Logger -Message "ERROR: Handshake Failure" -Func $Func -Level 'error'
}
ElseIf ($WebException -like "*Unable to connect to the remote server*" -Or $WebException -like "*The operation has timed out*")
{
MFE-Logger -Message "ERROR: Gave up waiting, server not reachable (SYN, SYN, SYN)" -Func $Func -Level 'error'
}
Else
{
MFE-Logger -Message "ERROR: Not sure what happened here..." -Func $Func -Level 'error'
Write-Host $WebException -ForegroundColor "Red"
}
}

Function MFE-IgnoreCertWarnings
{
MFE-Logger -Message "Ignoring certificate warnings" -Func $MyInvocation.MyCommand

Add-Type @"
using System;
using System.Net;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
public class ServerCertificateValidationCallback
{
public static void Ignore()
{
ServicePointManager.ServerCertificateValidationCallback +=
delegate
(
Object obj,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors errors
)
{
return true;
};
}
}
"@

# Only use TLS1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

[ServerCertificateValidationCallback]::Ignore();
}

Function MFE-CheckForUpdates
{
MFE-Logger -Message "INFO: Checking for updates at ftp://custftp2.nai.com..." -Func $MyInvocation.MyCommand
$TmpFile = "$($global:ScriptPath)\Update.txt"

$response = try {
Invoke-WebRequest -TimeoutSec 5 -Uri $global:VersionCheckUrl -OutFile $TmpFile
}
catch {
$StatusCode = $_.Exception.Response.StatusCode.value__ ;
$StatusDescription = $_.Exception.Response.StatusDescription
$WebException = $_.Exception
MFE-CheckForHttpError -StatusCode $StatusCode -WebException $WebException -Func $MyInvocation.MyCommand -UrlHost 'custftp2.nai.com'
}

If ([System.IO.File]::Exists($TmpFile))
{
$ServerVersion = Get-Content $TmpFile;
[int]$cver = ([Decimal]$global:ScriptVersion) * 100;
[int]$sver = ([Decimal]$ServerVersion) * 100;
If ($ServerVersion -ne '' -And $ServerVersion -gt $global:ScriptVersion)
{
MFE-Logger -Message "INFO: Update Available (Your Version: $global:ScriptVersion vs Available Version: $ServerVersion)" -Func $MyInvocation.MyCommand -Level 'info'
MFE-Logger -Message "Download: $global:DownloadUrl" -Func $MyInvocation.MyCommand -Level 'info'
}
If ($global:ScriptVersion -eq $ServerVersion)
{
MFE-Logger -Message "INFO: You are up to date (Your Version: $global:ScriptVersion)" -Func $MyInvocation.MyCommand -Level 'info'
}
Remove-Item $TmpFile;
}

}

Function MFE-Logger
{
Param ( [string]$Message, [string]$Func, [string]$Level )
# Timestamp format = [31/Jul/17 12:00:00 -0500]
$timezone = "{0:zzz}" -f (Get-Date)
$timezone = ($timezone -replace ":", '')
$timestamp = "[{0:dd/MMM/yy} {0:HH:mm:ss} $timezone]" -f (Get-Date)
Add-Content $global:MfeLogFile -Value "$($timestamp) $($Func): $($Message)"

If ($Level.ToLower() -eq 'error')
{
Write-Host "$($timestamp) $($Func): $($Message)" -ForegroundColor "Red"
}
ElseIf ($Level.ToLower() -eq 'warn')
{
Write-Host "$($timestamp) $($Func): $($Message)" -ForegroundColor "Yellow"
}
ElseIf ($Level.ToLower() -eq 'info')
{
Write-Host "$($timestamp) $($Func): $($Message)" -ForegroundColor "Cyan"
}
ElseIf ($Level.ToLower() -eq 'success' -Or $Level.ToLower() -eq 'ok')
{
Write-Host "$($timestamp) $($Func): $($Message)" -ForegroundColor "Green"
}
Else
{
Write-Host "$($timestamp) $($Func): $($Message)"
}
}

Function MFE-GetEpoch
{
return $ED=[Math]::Floor([decimal](Get-Date(Get-Date).ToUniversalTime()-uformat "%s"))
}

Function MFE-Cleanup
{
# Cleanup Variables
Clear-Variable -name base64AuthInfo -Scope Global
}

MFE-Startup;