Without knowing your environment it's hard to say. I haven't heard any legitimate horror stories, so I don't see much of a point in holding back.
Also, NO MATTER WHAT, make a copy of the DB and EPO using article
that way in case something does happen you can roll back.
In my case i would prefer to upgrade to ePO 4.5P4 because I usually never uprade to products without patch/service pack. One patch 1 for ePO 4.6 is RTW then you can see if it solves some bugs you could be interested in and then upgrade if you want.