I believe you will assing ''AD user group" in ePO permission set than users. Correct.
To troubleshoot the issue,
1. Are the users part of user group in AD when you included in ePO permission set?
2. At ePO login screen. Under user name, it will be domain\<user name>. If login fails then may I know the error mesage it fails with?