Easiest would be to use old IP: configure the new ePO server with the IP from the old XEN machine.
Hope this helps.
I forgot to mention.
I used the same IP and hostname.
The only difference is that now it's running on VMWare and that I used Windows 2008 R2 server instead of Windows 2003 server.
Check the DNS on an endpoint: check name resolution for the ePO server and also check that all the needed ports are opened on this new server.
If everything is ok from networking point of view, check the ePO server settings regarding Security Keys.
I can open the manager console in my browser, so I guess the network is ok.
What do you mean with the Security Keys?
The process to transfer to a new server:
OId Agent and servers use PKI to communicate. The server IP, name and keys are included in the framepkg.exe. If you don't have a backup of the old server to restore the keys, then your best bet is to repush a new agent with logon scripts, ad sync, rogue system detection or with lists of devices.