cancel
Showing results for 
Search instead for 
Did you mean: 
Nick_B
Level 11
Report Inappropriate Content
Message 11 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

As luck would have it, Richard is busy interviewing but he did send through the logs from ePO, no luck with the SQL logs as yet though. A snip from the Orion log is below (seems to be repeating this over and over).

2019-06-20 14:48:12,443 ERROR [core-ClusterService-thread-1] queue.TaskQueueEngine - failed to re-enqueue a task queue items. will attempt again after waiting for a bit
java.sql.SQLException: com.mcafee.orion.core.db.base.DatabaseConnectivityException: Failed to get a connection: Network error IOException: null. Navigate to https://localhost:8443/core/config and verify database connection settings
at com.mcafee.orion.core.db.base.MfsDatabase.getUnderlyingConnection(MfsDatabase.java:330)
at com.mcafee.orion.core.db.base.MfsDatabase.getContextualConnection(MfsDatabase.java:305)
at com.mcafee.orion.core.db.base.MfsDatabase.getDelegatedConnection(MfsDatabase.java:270)
at com.mcafee.orion.core.db.base.MfsDatabase.getConnection(MfsDatabase.java:234)
at com.mcafee.orion.task.queue.TaskQueueEngine.reenqueueForRetry(TaskQueueEngine.java:1522)
at com.mcafee.orion.task.queue.TaskQueueEngine.stop(TaskQueueEngine.java:444)
at com.mcafee.orion.task.queue.TaskQueueEngine.beforeDisconnect(TaskQueueEngine.java:1761)
at com.mcafee.orion.core.cluster.ClusterServiceImpl$Connected.disconnect(ClusterServiceImpl.java:449)
at com.mcafee.orion.core.cluster.ClusterServiceImpl.disconnect(ClusterServiceImpl.java:152)
at com.mcafee.orion.core.cluster.ClusterServiceImpl$HeartBeat.forceReconnect(ClusterServiceImpl.java:894)
at com.mcafee.orion.core.cluster.ClusterServiceImpl$HeartBeat.run(ClusterServiceImpl.java:869)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mcafee.orion.core.db.base.DatabaseConnectivityException: Failed to get a connection: Network error IOException: null. Navigate to https://localhost:8443/core/config and verify database connection settings
... 18 more
Caused by: java.sql.SQLException: Network error IOException: null
at net.sourceforge.jtds.jdbc.JtdsConnection.(JtdsConnection.java:460)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:185)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.mcafee.orion.core.db.sqlserver.JtdsJdbcDriver.getConnection(JtdsJdbcDriver.java:107)
at com.mcafee.orion.core.db.base.DriverRegistry.getConnection(DriverRegistry.java:323)
at com.mcafee.orion.core.db.base.DriverRegistryConnectionFactory.createConnection(DriverRegistryConnectionFactory.java:34)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at com.mcafee.orion.core.db.base.DbConnectionPool$LoggingConnectionFactory.makeObject(DbConnectionPool.java:232)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at com.mcafee.orion.core.db.base.DbConnectionPool$CountingGenericObjectPool.borrowObject(DbConnectionPool.java:183)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at com.mcafee.orion.core.db.base.DbConnectionPool.getConnection(DbConnectionPool.java:63)
at com.mcafee.orion.core.db.base.Database.getConnectionInternal(Database.java:610)
at com.mcafee.orion.core.db.base.Database.getConnection(Database.java:598)
at com.mcafee.orion.core.db.base.MfsDatabase.getUnderlyingConnection(MfsDatabase.java:324)
... 17 more
Caused by: java.io.IOException
at net.sourceforge.jtds.ssl.TdsTlsInputStream.readFully(TdsTlsInputStream.java:137)
at net.sourceforge.jtds.ssl.TdsTlsInputStream.primeBuffer(TdsTlsInputStream.java:100)
at net.sourceforge.jtds.ssl.TdsTlsInputStream.read(TdsTlsInputStream.java:78)
at com.rsa.sslj.x.ap.c(Unknown Source)
at com.rsa.sslj.x.ap.a(Unknown Source)
at com.rsa.sslj.x.ap.a(Unknown Source)
at com.rsa.sslj.x.ap.j(Unknown Source)
at com.rsa.sslj.x.ap.i(Unknown Source)
at com.rsa.sslj.x.ap.h(Unknown Source)
at com.rsa.sslj.x.aT.startHandshake(Unknown Source)
at net.sourceforge.jtds.ssl.SocketFactories$TdsTlsSocketFactory.createSocket(SocketFactories.java:137)
at net.sourceforge.jtds.jdbc.SharedSocket.enableEncryption(SharedSocket.java:329)
at net.sourceforge.jtds.jdbc.TdsCore.negotiateSSL(TdsCore.java:578)
at net.sourceforge.jtds.jdbc.JtdsConnection.(JtdsConnection.java:387)
... 32 more

McAfee Employee cdinet
McAfee Employee
Report Inappropriate Content
Message 12 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Looks like some possible tls issues.  Get an nmap output on the sql ports on sql server (both nodes of cluster) and port 8443 on epo server - see kb91115

 

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

Highlighted
Nick_B
Level 11
Report Inappropriate Content
Message 13 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Thanks Caryn - will do.

Nick_B
Level 11
Report Inappropriate Content
Message 14 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Hi Caryn,

Just wanted to furnish you with a bit of background info on this DB Server Key Check failure error. Some of it you will probably already be aware of.

The customer's ePO server was upgraded from 5.3.2 to 5.9.1 on 23 May.

The backend SQL Server is a HA Cluster consisting of two nodes - VC027A and VC027E each running Windows SQL Server 2016.

Prior to the upgrade the DB was taken out of the Availability Group (this was one of the failures when we ran the ePIP tool on the ePO server).

Another of the checks which failed was the RSA Compatibility Check. To rectify this we ran the IISCrypto tool on one of the nodes (VC027A) but as there were 2 other DBs (Trend Micro) that resided there, they were failed over to VC027E so we could reboot it without it impacting on the Trend DBs.

The plan was that once the Trend DBs were failed back to VC027A, the IISCrypto tool would be run on that server (VC027E) too.

I recall Richard saying he had not had a chance to run IISCrypto on the second node when I was last onsite and I'm not 100% certain he has had a chance to run it yet.

In any event, we'll arrange for the nmap tool - full version recommended? - to be installed on a separate device (so not the ePO server!) and run the nmap command pointing at the SQL port, 1433 as well as ePO' on 8443?

McAfee Employee cdinet
McAfee Employee
Report Inappropriate Content
Message 15 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Not running iiscrypto on the other node in cluster could very well affect it and yes, you can run nmap remotely.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

Nick_B
Level 11
Report Inappropriate Content
Message 16 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Indeed, I would have thought so!

I just ran the nmap tool using the command at the bottom of kb91115 against my Lab SQL Server and my Lab ePO server and the results are below.

Is this typical of what you would see?

So what we are looking to see is the two results from the SQL nodes in the Cluster to match exactly?

nmap-tool-results-test-lab-sql-server.PNGnmap results - SQL Server (Test Lab)

There's a lot more going on with regard to ciphers on my test ePO server...

nmap-tool-results-test-lab-epo-server.PNGnmap results - ePO Server (Test Lab)

 

 

McAfee Employee cdinet
McAfee Employee
Report Inappropriate Content
Message 17 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

That looks like a 5.3.2 server - if that is the case, you would fail an upgrade to 5.9 or 5.10. 

What you are looking for in those higher versions of epo is basically the tls 1.2 protocol enabled on both servers with the following ciphers (at least one of the tls_rsa ones and tls_ecdhe ones)

kb91296

  • TLS_RSA_WITH_AES_128_CBC_SHA256
  • TLS_RSA_WITH_AES_256_CBC_SHA256
  • TLS_RSA_WITH_AES_128_CBC_SHA
  • TLS_RSA_WITH_AES_256_CBC_SHA

kb91304

  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

McAfee Employee cdinet
McAfee Employee
Report Inappropriate Content
Message 18 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

No, they don't have to match exactly, just contain any of those ciphers (both servers would need the same ones).  There could be other non-related ciphers enabled, which is fine - we just need to have at least one of those each present - same ones on both servers.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

Nick_B
Level 11
Report Inappropriate Content
Message 19 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Ah OK that makes sense.

Sorry, I think there's a bit of confusion with the name of the ePO server in the above nmap output; although its hostname is eposrv532 it does have v5.9.1 installed - it was upgraded from v5.3.2 (and this is in my test Lab).

So the nmap results listed above are only from the test lab machines, the ePO and SQL Servers.

What I find odd here is that even though the IISCrypto tool has been run on this SQL Server in my lab, it still reports as having the TLS 1.0 Cipher suite only, enabled. This is after applying the Best Practices on it and rebooting.

Looking at the nmap results of the ePO server on the other hand, it apparently has the TLS 1.1 and TLS 1.2 Cipher Suites.

Despite this apparent mismatch, there are no DB connectivity issues between the test lab ePO and SQL Servers.

Perhaps I am missing something - but on the face of it this ePO server should not be able to talk to the backend DB! Also in theory it should not have been possible to upgrade it?

Nick_B
Level 11
Report Inappropriate Content
Message 20 of 29

Re: ePO Database Connection Issue (DB Server Key Check Failed)

Hi,

Having reviewed KB91304, perhaps the reason I am able to connect to my ePO and SQL backend in my test lab is because the cipher suites below are enabled on the SQL Server:

Cipher Suites - TLS1.0 - enabled on SQLSRV.PNGCipher suites on SQL Server (Test Lab)

So, even though it is not the TLS1.1 or 1.2 ones, the ePO version is 5.9.1 rather than 5.10 which is "less fussy"?

However, when I run IISCrypto on the SQL Server it makes no difference to the Cipher suites installed.

I'm still waiting to hear back from the customer around whether IISCrypto was installed on both nodes of their SQL backend and for the results of the nmap commands to discover which cipher suites are enabled on the ePO and SQL nodes.

This is getting quite complicated!

Thanks for all your help, Caryn, you have been amazing.

More McAfee Tools to Help You

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community