I'm getting this error message:
curl: (35) error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small
Date: Tue Jun 23 10:00:13 -03 2020 Error retrieving Scheduled log from WGCS. Curl return code:35 Log file not created
Could you help me please?
Version 5.3 Attached.
Many changes and improvements. Most importantly if a partial file is returned it will be correctly processed and the data will not be pulled a second time. Also improved error handling. From the notes in the script includes all improvements from 5.2:
#version 5.2.1 Tries smaller queries if time taken exceeds query timeout and tries reducing time before reducing record count limit
# Also identifies error based on query timeout (version not released)
#version 5.2.2 Adds x-mfe-timeout header to curl and uses non blank last line to detect error
# rather than query timeout (version not released)
#version 5.2.3 Tries to recover data if partial data returned(version not released)
#version 5.2.4 Reorder code file returned check at beginning of file processing
# Consolidate code for processing returned file (version not released)
#version 5.2.5 Fix bugs and adds 10 sec to request vs created to to account for clock skew. (version not released)
#version 5.3 Rewrote merging to eliminate redundancies and make more logical
#version 5.3.1 Changed update of last success epoch to be based on record count rather than log file exists
# Removed header x-mfe-timeout added in 5.2.2
# Improved efficiency by ensuring for each query request created from not more than a day prior to request start and
# Coded so request to never gt created to
# Changed logging / reporting so that if request to is not specified current epoch - dbBackset (default 2 mins) is used
Limited testing has been performed. However it is recommended that you verify changes in your environment before putting into use.
We just recently discovered that pulling logs from msg.mcafeesaas.com or msg.mcafee-cloud.com does NOT download logs from the EU and CSR has the same issue.
It turns out that the EU has its own endpoint for pulling logs. eu.msg.mcafeesaas.com. However, when we point this script at this endpoint, we were getting 500 errors. The culprit was this:
--header "x-mfe-timeout: $queryTimeout"
Once we removed that it worked.
So now, we have 2 copies of this script running on the same server. And the one for the EU has the problematic header removed.
Just an FYI if you have users in the EU and elsewhere. And if someone feels like updating the script to include multiple locations, that would be helpful.
Thanks for the post and heads up. That header is optional, and actually as it turns out, not really effective in addressing what it was intended to address so it can be stripped regardless of whether you are pulling from EU or NA log repositories. There are other repositories currently available as well, log location and server to pull from depends on where you choose to log in your UCE or Cloud ePO configuration.
CSR 2.8 you would use multiple log sources to pull from EU and NA
As for modifying the script to pull from multiple locations, yes it could be done but I don't think I can get to it any time soon, because it would be a non-trivial effort. Each source needs separate tracking which is done currently in the single conf file. Probably better to just run two instances that run independently in separate folders with separate conf files as it sounds like you've already done. Good work.