check whether SQL browser services are running.
P.S: To be moved to ePO for better support.
Moved to ePO - Moderator
In the database server field type <hostname>\<instance name>
In the database name field type the name which you have created manually.
for 1434 error
1.Goto SQL server configuration manager | SQL network configuration | protocols for <ePO server name>| TCP\IP
3.In the IP address column goto IP ALL type 1434 in TCP port
4.Restart the services of SQL
5.Re run the setup again.
It is recommended to keep Firewall off during the installation.
Did you check it?