To manage Host IPS 8.0 clients with ePO 5.1.3 or 5.3.1 (or later), you must install the Host IPS Extension Patch 6 (220.127.116.118) first.
Host IPS 8.0 Extension Patch 6 (18.104.22.1688) must installed before upgrading ePO.
For release note details, see PD25972:
Also see updates to the "Supported Products" KB articles for ePO 5.1.3 and 5.3.1.
KB79169 - ePolicy Orchestrator 5.1 supported products
KB82836 - ePolicy Orchestrator 5.3 supported products
Host Intrusion Prevention System (Windows and non-Windows)
NOTE: Host IPS 8.0 Extensions prior to build 22.214.171.1248 cannot be installed and do not support ePO 5.1.3 or later and ePO 5.3.1 or later. See KB84853.