It sounds as though you are running version 8.
If so, you can set the timeout for any service using the generic defense option you are already looking at.
In version 7, the timeout value was associated with the service entry. Now, when you create a custom 'application' you are only required to specify the protocol type and port number. Everything else concerning this service is then handled by the generic application defense.
You can either create a brand new generic defense, or duplicate an existing defense (minimal proxy if you want it to behave like a proxy, or connection settings for a packet filter), scroll down the list of settings and change either the TCP or UDP idle value as required.
Having done that you then create a new application defense group entry - specifying your newly created "Generic" defense and apply this application defense group to the access control rule.
Hope that helps.