To answer the question about what to disabled - On-Access Scanner: YES Access Protection: No
Generally speaking, this will reduce the time to install an application because the scanner won't be scanning on read/write. That said, typically OAS will not cause noticeable negative impact/performance degradation. If you are noticing performance issues, check your OAS max scan time, it may be set too high.
You can set a password in the 'General Options' policy to disallow tampering w/VSE. I don't believe you can change the time threshold to re-enable, at least not that I am aware of.
You may have the Agent configured to re-enable Access Protection and OAS on policy enforcement. You can increase the policy enforcement interval to reduce the frequency with which this occurs.
Depending on how long you want to keep AP/OAS disabled, it may be a good idea to create a secondary, non-baseline policy for this. Since the need should be infrequent, you could apply the policy, wake-up the agent and do your thing - rather than impact all nodes by a policy enforcement interval change. Albeit, if you are wanting to go from 5 min to 10 the impact is much different than 5 to 60.
Most of the time software developers use generic rules of thumb like 'Disable all AV software' as a catchall excuse when software fails to install. Most other AV software may cause conflicts and the developer simply doesn't want to test with every AV package that is available.
I have found VSE and Access Protection to be extremely reliable and have Not 'had' to disable AP/OAS, ever. This includes recent and unintentional, customer initiated Windows 10 upgrades (with compatible VSE version and patch already in place).
Once, I had a problem with a program installing correctly, nearly 10 years ago. The issue had to do with MDAC and .Net Framework Services. The (uninformed) support technician with the (unspecified) company who makes the software took the obvious position blaming everything but their own software and installer. To placate the technician, I had to completely remove VSE and all remnants, etc. The installer still failed. After many other issues were identified with their software, a newer installer with a modified MDAC and .Net Framework Services was needed to complete the process. (This became the next version distributed by this vendor.) Since then, no problems. VSE has not interfered with subsequent updates since that time.
Though this is just an anecdotal example, my experience has been that one of VSE's strengths is AP/OAS. Look elsewhere for problems. In my humble opinion: Leave AP alone even during 'New software installation.' This has been my mode of operation for my Customer base, for years, without issues. (I use basic or default AP and OAS settings as much as possible. If Maximum Security settings are used, your mileage may vary.)
So, my Best Practice would be: Leave AP enabled even during New Software Installation.
As Alex.Hawke stated, performance may be impacted during the installation with AP running, but leaving AP on leaves security enabled. To me this is a good trade-off.
If you have an OAS policy which does not allow programs to be executed from the .temp directory, this can cause the problem. Lots of programs run from the .temp directory
If your policy refresh time is set to something large enough to allow you to complete the installation by the time the policy refreshes, disabling OAS and installing the program will work. If you finish the installation earlier or reboot the system, OAS will restart automatically.
*OAS enabled, AP shouldn't have any bearing unless blocks are being triggered.
Agreed. Good points.
Thus my edits to my original post. Usual blocks caused by setting %TEMP% to block execution (rather than report only) and Maximum Security settings which may tighten things so much that no installation would succeed. Again, I state that the Default or minimally changed defaults, for my customers, seem to work well.
And as far as developers go: I never execute anything from %TEMP% for this very reason. Unfortunately, most developers get lazy with this setting.
There is no One right answer here, as each client site will have different requirements and security issues. Blocking exposures such as Ransomware, may force locking down the system tighter than my recommendations will allow. But that is a Choice, which the Network/Security Administrators should make with all associated issues considered. In this case, the user base should probably be denied access to install software in the first place, leaving it to the Administrator.
For my customers, the default settings with only minor adjustments, have worked well.
I agree with all the other previous posts. In the interest of a complete discussion I would like to point out Knowledge Base article KB52624 which pertains to issues performing Windows updates if you have the "Prevent Windows Process spoofing" enabled (which by default it is NOT). We did have issues with Windows patching and updates triggering this rule, however with some well crafted process exclusions you can overcome these issues so that you do not need to disable Access Protection.