Bear with me as I'm a little still new to ePO too.
Not sure if you have looked at the hardware sizing guide but it will probably answere all of your questions. It looks like unless you are taling about upwards of 60,000 managed systems, you dont necesarilly need an agent handler at all. The repositories arent a bad plan though if you plan on pushing software and updates regularly and so I don't see why you wouldn't double purpose these systems as an agent handler/repository. The guide doesn't speak to that. Anyway, look it over and I'm sure you'll see what I mean.
What version of ePO are you running ?
if you have updated the ePO at some stage, then the Agent handlers have to be upgraded to the same version, otherwise they won't talk and will be disabled.
Also check the services on these machines to see if they are running.
I have 5 Agent Handlers, and 150 repositories around different sites.
My ePO server updates the agent handlers, and these in turn replicate to the various repositories. I find them useful in our setup where we have multiple customers and restrictions on where they can update from . It reduces the load/bandwidth on my ePO