So I just did a quick test and a simple agent install (without the forceinstall option) does not overwrite the existing custom properties.
You are correct their is no way to specify an custom install option (other than the install directory for the agent) when pushing an agent from ePO. I can think of a rather cumbersome workaround. You can pass commandline arguments when you upgrade the agent via a deployment task so you could push an older version of the agent from the evaluation branch and then use a deployment task with the custom properties option to supply the custom properties.
Specifying commandline options during an agent push would actually be a very useful feature so I would encourage you to submit an FMR on this. You can submit your suggestion here:
Don't forget, when your helpful posts earn a kudos or get accepted as a solution you can unlock perks and badges. Those aren't the only badges, either. How many can you collect? Click here to learn more.
Community Help Hub
New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.