I would know if it is possible install the agent handler for ePO 5 in a windows server 2008 R2 x64 domain controller or it isn't recommendable for some reason.
The only requirement for the installation of a remote agent handler is a server-class OS.
Certainly something to remember is that an Agent Handler needs an excellent and persistent connection to the ePO Sql Instance or it will cause issue for the hosting ePO server.
Also, the right IP ports need to be accessible. for that you can refer to support article:
KB66797 - Ports needed by ePO 4.x and ePO 5.0 for communication through a firewall
It will function. I would probably recommend against that because if something happens with one of those products it will impact the other. With virtualization so easy these days I really prefer not to combine functions.