I've noted that for SQL Server 2016, support was introduced in ePO 5.3.3, and presumably present in 5.9.
Does ePO 5.9 support being installed to the recently released SQL Server 2017 ?
I'm currently planning a new ePO & SQL Server to (initially) sit along side our existing, and want to install ePO 5.9 on Server 2016, and then a separate Server for the database, again Windows 2016, and SQL Server 2017. I'd then use the "transfer systems" functionality to move agents from old (5.3.2) server to new (5.9) server.
IMO, you can install with SQL 2017. Please be sure to run the pre-installation tool. This KB is updated frequently. I like your strategy of setting up the new server and gradually transferring everything over to the new server. It's lengthy but transparent.