ePO 5+ requires minimum win2k8r2 64 bit and SQL 2008 R2 64 bit..
Your best off building a new server/VM, setting up ePO 5 and SQL and then migrating from old > new as you won't be able to do a "in-place upgrade".
Basically you could at a high level:
1. build your new ePO
2. add the new ePO as a registered server in the old, share the security keys
3. export from old your policies/tags/tree/queries/tasks/etc and import it into the new
4. migrate machines from old > new
5. configure the rest as needed
I know we can transfer the systems but i don't want to transfer the systems.
Is there any other way other than that ?
Unless you want to deploy the agent from the new ePO on-top of the old, and make sure you delete them out of the old, i don't think there is really any other way.. but maybe someone else can chime in regarding this.
You can also create a automatic response to transfer the systems from the old > new, but that is the same thing as doing it yourself.
You could go through this if you'd like, but seems like a pain: