This is exactly the process we have gone through in the past when migrating from ePO4x to 5x.
You can deploy a newer agent "over" an existing agent providing you enable/tick the force installation over existing agent in the Agent Deployment.
Before you do this though check you have recreated your management product policies on the new server or have a migration plan up to newer endpoint products such as from VSE 8.7 to 8.8.
If you have any specific questions about migration let us know.
McAfee Certified Product Specaliat - ePO
Just to clarify: deploying an agent from the new server is definitely the best approach, but you do *not* need to have the Force Install option selected. This should only be used for machines that have problems, or of you need to change the install path, for example.
Agent installers currently work like this:
If the version being installed is newer than the currently installed version, the whole agent is upgraded.
If the versions are the same, the installer only replaces the sitelist and key files - effectively moving the client to the new ePO server.
I have a couple of considerations you should acknowledge before deciding on the path to take.
1.) Are you concerned with saving any of your event data from the old ePO instance?
2.) Is this ePO instance managing McAfee Disk Encryption?
If the answer is yes to either of those questions, you may be better off taking a different approach than was recommended prior; as unless you make some concessions, you could lose data.
All of that said, there are a couple of general ways you can approach this.
First things first, back up EVERYTHING ePO.
Take a look at KB66616 (McAfee KnowledgeBase - ePO server backup and disaster recovery procedure) for a list of the things you should back up. If this is done properly; worst case scenario, you could restore functionality to the point in time the backups were taken.
Getting on with it:
Retain MDE recovery keys and Event Data:
You could update ePO until you get to your target version and then perform KB66616 again and perform the restore steps on the new OS and/or server. Maybe even utilize ePO's DR snapshot capability? I have not done this yet, so I cannot comment on it. Unfortunately, it would need to happen in stages. Update ePO to 4.6 and then update the OS to 2008. Then update to ePO 5.1.1 and finally upgrade the OS to 2012. We haven't really talked about SQL versions so, use KB51569 (https://kc.mcafee.com/corporate/index?page=content&id=KB51569) to get you where you need to be to support the various versions.
Start fresh without the need to deploy anything.
Alternately, if not managing MDE or/and the event data is of no concern; you can install ePO on the new server, copy over the Agent security keys and alter DNS records to point systems trying to reach the old one to the new one. To ensure your point products do not get messed-up you may want to duplicate the system tree, client tasks and policies and their assignments from the old environment.
Hope this gets you started.