Yes, I have done this with an internally signed cert.
The cert is stored here (obviously your path may be different)
Location of OpenSSL: C:\Program Files (x86)\McAfee\Network Security Manager\App\Apache\bin
Location of OpenSSL Conf File: C:\Program Files (x86)\McAfee\Network Security Manager\App\Apache\conf\openssl.cnf
I honestly can't remember if I generated a new CSR or used the one that was already there, but either may work. Just call the new key my-server.key.
In case you need it, here is the openssl command to generate a csr:
openssl req -new -nodes -keyout C:\tmp\ipsmgr.key -out C:\tmp\req.csr -newkey rsa:2048 -config "C:\Program Files\McAfee\Network Security Manager\App\Apache\conf\openssl.cnf"
Once your key has been issued by your SSL admin you can download the base 64 encoded version and put the key on the NSM. You can verify the key with:
openssl x509 -in test.cer -text -noout