An ePO upgrade is usually an easy thing if you meet all requirements, it's almost just running the installer and "next" until it finishes.
Just remember that your ePO 4.6 must be installed on a Windows 2008R2, 2008 64 bits or Windoss 2012 (because of the 64 bits OS needed) and that if you use an SQL Express 2005 you'll need at least an SQL express 2008 (I suppose it's automatically migrated during the setup but not sure as I haven't tried it yet)
To upgrade the sql from 2005 to 2008, I have created the complete documentations with step by step procedure.
Please follow the attached document with screenshots to follow the steps.
Note: Since, you are doing In-place upgrade and if the sql 2005 on same server, then don't try to do the sql 2008 upgrade, do the new install of sql 2008.
Since, SQL multiple versions can be installed on same server.
>> More over, attached one more PDF for complete information for EPO 5.0 Upgrade /Migrate details, I hope this may help you.
NOTE: If the isue resolved, after following these steps, please select the correct or helpful answer, which helps others to follow the same steps.
Once, you install sql 2008 as new install.
You have to stop the sql services for the old instance and then copy the MDF and LDF file to the new location path of the sql 2008 and then fromt he management studio, try to connect the new sql 2008 with instance name and then expand the Databases and right click and select attach database and import the MDF and LDF file to add in new instance.
Then, you need to start the OLD instance sql service and login in to EPO as usual and then change the URl to https://localhost:8443/core/config and add the new instance name port for connection and the credentails tor each the new SQL 2008 with EPO DB.
Note: backup the db.properties file before any changes int he core/config page.
For more explanation, Please go through the previous attached document shared with screenshots for these steps to follow.
As you're on a 64bit OS, you can upgrade in place: but you'll need to move the database to a supported SQL version first. Assuming you want to keep the SQL instance on the same machine, then in a nutshell:
1) Install a supported version of SQL - for example 2008 R2 Express - on the same machine
2) Stop the 4.6 services
3) Move the ePO database to the 2008 instance, either by detaching it from 2005 and attaching it to 2008, or by backing up the 2005 DB and restoring it on the 2008 instance
4) Start the 4.6 services again
5) Log in to the config-auth page and point the server to the 2008 instance - make sure to use the test connection function first. Assuming it passes, click OK and you'll be prompted to restart the services: please restart them
You now have a 4.6 server on a 64bit OS with its DB on a supported SQL version. Log in to ePO and confirm that everything is OK - machines can communicate, reports still work and so on. (There's no reason why anything should be affected simply by moving the DB, though.) Assuming all is OK, you can now run the upgrade. I would strongly recommend backing up the relevant files and database in case of a problem, though - KB 66616 describes this in detail.