This may be caused by the fact that VSE 8.7 is not supported under ePO 5.1 and that you cannot manage its policies.
You first need to upgrade VSE to 8.8, after this you probably won't see these issues again
try try importing your old server security key into your new epo server and creating a DNS aliases for your old server name pointing to the IP address of you new server.
when your endpoints try to check in using DNS name they will resolve to your new server, provide the security key you have imported, and get the server comm setting updates to point to the new server.
also check in the vse8.7 extensions and you can manage them from your new server, or create a VSE 8.8 deployment task.
Best of luck,
I had the same problem a few months ago with one of my customers. The VSE 8.7 protect the computer from the possibility to install another agent since it is already managed by one.
The solution if you don't have access to the old security key(like Richard said) will be to manually disable the Access Protection on the PC and install the agent of your new server.
Maybe you can script it. My client decided to do it manually for his 200 PCs with the problem.
Good luck !
I got this sorted after a bit of head scratching.
I created an uninstall script and set it to run in GPO on computer shutdown and then on startup the new AV was pushed.
Cheers for your help