Hey rtk, Im also having this issue. An internet search said that it's due to internet explorer's Pop-Up Blocker. If you add your ePO site to the exceptions list, it should work..
However, didn't work for me, so give that a shot and if it doesn't work, I'll be waiting with you for a better answer =)
You have to install "DLPManagementTools" manually with administrator right on that client and check whether it helps or not.
Your answer is in your original post. Firefox is not supported for opening the DLP policy console on 9.3. The console uses an activeX plugin that only functions with IE10 and lower. IE11 might work but is technically not supported.
If using IE does not work then you may need to reinstall the DLPManagementTools as DavidP64 suggested.