you could check if the Yahoo Messenger sends a specific User-Agent. If it sends a user-agent that is different from what the browser sends, you could create a rule such as "URL equals login.yahoo.com" and "Header.Request.Get(User-Agent) matches *Yahoo*" then Block. Unfortunately I do not have a Yahoo Messenger installed, so I cannot tell if it sends a customer User-Agent header.
You could check the access.logs and compare a request to login.yahoo.com that you made from your browser and one that has been send by the Yahoo messenger to find a difference. This may be the simplest solution (depending on the messenger sending something you could easily block).
You are correct! YM does have a unique user-agent when accessing login.yahoo.com and its called "net_http_transaction_impl_manager/0.1"
Upon creating the following rules: (image below)
The above rule set stops authentication to the following website
If I don't do this, it will keep returning a 407 error to the user.
The next rule set will then check if the request is accessing login.yahoo.com and with a User-Agent of net_http_transaction_impl_manager/0.1
If yes, it will now do an authentication and then check if the user belongs to ymusers group.
If the user does not belong to the said group and it's trying to access login.yahoo.com, it will now block the usage of Yahoo Messenger.
I hope this will help others as well.
Thank you again asabban!