Hi ArnsteinLangnes, there's no problem installing without domain, the only problem you're having here is that SQLExpress won't instal because the SQL package used by ePO 4.5 installer is not suported on Windows 2008 R2 (because it's 64bits).
1- Once you have manually installed SQL Express 2008 on the server you need to enable TCP/IP, Named Pipes and Shared memory. You can enable them with SQL Configuration manager, under SQL Server 2008 Network Configuration -> Protocols for EPOSERVER (or your instance's name). You'll be asked to restart SQL services and then the ePO installer should see the SQL engine.
2- You can work with Device Control without domain. The only difference is that your rules won't be user-based but you will enable device control policies under your system tree