Go in Programe and features and look if the setup wizard has installed another sql instance?
See in your default directiory any other sql install folder?
I tghink previous setup wizard has installed another SQL DB and is trying to connect that.
You had not run compatibility tool if you are upgrading on the same machine.
But in your case I think you have run this tool, now remove .zip file created by this tool as well.
Simply run the 5.0 setup if machine is 64 bit. Do not run compatibility tool.
Just wanted to post an update to this issue. I was able to get past the error by restarting the SQLSERVER Service on the server and am now able to continue the installation again.