9 Replies Latest reply on Jun 23, 2015 1:42 PM by wwarren

    Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks

    mhlavinka

      If you install VSE 8.8 Patch 5 on a Hyper-V host that contains VMs with pass through disks (physical disks), you will not be able to start those VMs after rebooting the host.  I have experienced this on two hyper-V hosts so I have removed Patch 5 from them and my other Hyper-V hosts (which I have yet to reboot).  One of my hosts is Windows Server 2012 R2 and the other is Hyper-V Server 2012 R2.  The systems hosted by virtual drives will start, but not pass through systems.  You'll get errors during start up such as:

       

      'MyVMName': Attachment '\\?\scsi#disk&ven_lsi&prod_megasr__#5&266e821e&0&020200#{53f56307-b6bf-11d0-94 f2-00a0c91efb8b}' failed to open because of error: 'One or more arguments are invalid' (7864368). (Virtual machine ID 0D7719F8-07B3-4156-9A59-3F5B8983FC63)

       

      At first I thought it was a Windows problem because it occurred at the same time as some Windows Updates were applied.  However, it is not.  McAfee is analyzing the problem.

        • 1. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
          wwarren

          I heard this was connected somehow to our disk driver, mfedisk.sys. That without its presence the issue does not manifest.

          This is a new driver to Patch 5. It is one of the reasons I indicate in my blogs that it should be considered a high risk change to an environment: Patching VSE - risk level

          Patches should be planned for and tested, if at all possible: Patching VSE - testing, testing, is this thing on?

           

          I hope you're able to work with our Support team to see if we can root cause why this occurs, whether it's a code issue on our part or a compatibility issue on the part of the Hyper-V, or other presence that doesn't like our driver. Not a very simple issue to investigate because of the nature of the code we're talking about - disk driver code.

           

          And before anyone has half a mind of removing that driver, DO NOT. You will render your system non-bootable. One cannot simply remove a disk driver - when Windows boots it looks for such drivers and when an expected driver is not present, Windows will not boot.

          It can be disabled though - in an unsupported fashion, such that you'd only ever do it for testing purposes. In the registry at this location:

          HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e967-e325-11ce-bfc1-08002be10318}

          Edit the UpperFilters value to remove only the Mfedisk entry. Then you can reboot without fear of it failing and having to go to Safe mode etc.

          The supported way to remove this driver is to uninstall VirusScan.

          • 2. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
            mhlavinka

            Yes, I have a support case open and it is related to mfedisk.sys.  Removing it from loading using the registry allows the VM guest to boot, but we couldn't get the AV software to run without it.  Thus I went back to update 4.

            • 3. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
              dean.salter

              Can you tell me what the new MFEDISK.SYS driver does? What effect does removing it from the UpperFilters have in regards to the operation of VSE? Is it involved in On Access Protection? Which component(s) of VSE is it tied to?

               

              One reason for asking this is that we have run into a bug with this driver and Server 2003. Anytime a new disk is added to a Server 2003 server there is an Access Denied error message when Windows is trying to install the necessary drivers. I traced it to the MFEDISK.SYS driver. With the driver left in the UpperFilters, Windows cannot install some of the drivers for the disk. One of those is the Microsoft CRCDISK.SYS driver. We connected a USB flash drive to a Server 2003 server and got the Access Denied message upon Windows installing drivers for it and when I checked the disk, the CRCDISK.SYS driver was not installed for the USB disk. I then removed MFEDISK.SYS from UpperFilters and rebooted. During the reboot, Windows was able to complete the Plug and Play installation of the remaining driver for that USB disk (CRCDISK.SYS). I then reinstated the MFEDISK.SYS in UpperFilters and rebooted. The server is fine and it had no further effect on that USB disk, now that Windows had already installed all of the necessary drivers.

               

              I also had this issue for a Server 2003 VMware VM when adding a disk to the VM. Windows returned an Access Denied when it tried to install the Plug and Play drivers for it. I did the same remediation step as for the USB disk above and that resolved the issue.

               

              Finally, we migrated SAN on a Server 2003 from Symmetrix to EMC Invista, which resulted in a Plug and Play discovery of the new LUNs and the Access Denied message. At that time, I had not found your article on the UpperFilters method so I uninstalled VSE and rebooted. Windows then finished the Plug and Play disk driver install. I then reinstalled VSE and rebooted.

               

              As you can imagine, this is not a practical way of managing our corporate Server 2003 servers, every time we need to add disks. That is why I am asking about the MFEDISK.SYS driver being removed from the UpperFilters. I am looking to doing that on all of our Server 2003 servers to avoid this issue. Note that it is not a problem on OSes newer than Server 2003. I suspect it has to do with the design of filter drivers in Server 2003 which was redesigned for newer OSes. I am not to hopeful that McAfee could produce a hotfix for their MFEDISK.SYS driver to overcome the bug. If you think McAfee could/would then please let me know.

               

              Until then, I am wanting to know the consequences of removing MFSDISK.SYS from UpperFilters.

               

              Looking forward to your response and appreciate your help!

              • 4. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
                wwarren

                Hi Dean,

                 

                For now that driver provides no additional functionality. It is installed that it might be leveraged in the future or as needed but for now it's simply readying itself so-to-speak.

                As a disk filter driver, it will be capable of manipulating disk devices it's attached to, and for our purposes that would be associated with bootsector+MBR scanning and cleaning. The product can already deal with that type of threat, but with a disk filter driver it'll be easier and may avoid reboots.

                Your plan to remove this from 2003 servers due to the interop issue you've noted seems sound. What I would watch for in future is what happens when adding later patch revisions (patch 6, for example) because the installer is going to want to do "something" with that driver...

                Being 2003 server we would be on our own for investigating and solving this behavior if we needed input from Microsoft, which could preclude ever finding root cause. Even if we did invest time/resources and find root cause we could still discover that our driver exposes a bug in PnP routines, which wouldn't be something we can fix - we encountered this type of outcome when our mfetdik.sys driver exposed issues with Microsoft's TDI.sys legacy driver.

                Indeed the driver architecture of the OS is very different in later versions, and therein one could also infer appropriate fixes had been added by Microsoft to accommodate 3rd parties. I suspect we'll never know the truth of it all.

                • 5. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
                  dean.salter

                  Thanks for your good answer William. Always appreciate your responses and posts.

                   

                  The Access Denied message when updating disk drivers makes me think that McAfee Access Protection (self-preservation) is blocking the other disk drivers from installing, now that MFEDISK has already been installed. But we get this error even when Access Protection is totally disabled. Is it possible that the McAfee module responsible for access protection wasn't programmed for this new McAfee driver and so it is stuck in a deny/block mode for the MFEDISK driver? Or is the Access Denied message generated by Windows and no change to any McAfee module is going to correct that?

                   

                  While I am glad that we can implement the registry change to avoid future instances of this issue (being careful as you said if we upgrade VSE), the pain point is having to reboot each affected server and take the outage. This could be 100s of servers, many of them production. Is there a way to accomplish this without a reboot? That would be great! Any ideas?

                  • 6. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
                    wwarren

                    Hi Dean,

                     

                    In lieu of knowing root-cause, your guess is as good as mine but we could benefit from peer review of such guesses.

                    In the scenario you describe, where perhaps the Access Protection code is blocking other disk drivers from installing, we would expect to see that same behavior on other OS's if that were true... unless of course, you're also inferring that this aspect of the operating system itself is what Microsoft might have changed in later releases, which allows install to succeed despite that alleged restriction?  Well, it's plausible given the assumptions.  VSE doesn't have OS-specific AP rules, so the difference in behavior would have to be accounted for in OS changes.

                     

                    If there is a denial occurring from our code, it'll be visible from trace logs (requires a tool from Support).

                    If the denial is occurring on registry or file objects, it will be visible in Process Monitor - a very helpful tool for getting clues and another data point for investigation purposes.

                     

                    The observation that it occurs when AP is disabled greatly weakens the idea of there being any connection to the Access Protection feature.  But, not all restrictions in our product are handled by Access Protection... maybe it's something else. Hmm. An idea comes to mind if you're willing to test it out... I'll relay it to the Product Specialist handling your reported incident, because sharing the idea here could lead to abuse. And, it might not pan out to anything.

                     

                    Regarding reboot, I do not believe there's any way to avoid a reboot when dealing with a boot-start disk driver.

                    • 7. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
                      dean.salter

                      Thanks William. I have asked my McAfee case worker (Andrew Dawes) to contact you to get that information you mention to give to me. I am hopeful McAfee will have a solution that is better than your registry edit and reboot (outage to service), that allows us to keep the functionality McAfee intended with the MFEDISK.SYS driver. There are 100s of affected servers. I expect there will also be other McAfee customers who experience the same issue on their Server 2003 systems and need a solution.

                      • 8. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
                        ingmar79

                        Hello,

                         

                        The problem of the boot sequence change of mfedisk.sys in patch 5 brought, is not only affecting Windows 2003 servers. VSE 8.8 patch 5 also affected our Windows 2012 R2 servers, rendering our backup recovery software (uses isci disks) unable to load the proper disks. We tracked it down to the upperfilters in the exact same location that was posted here.

                         

                        So I hope VSE 8.8 patch 6 won't cause the same problem that patch 5 caused.

                         

                        I tried to solve this, by editing the sequence of the upperfilters setting, but this is changed by mcafee on restart of the VM.. I could write a Group Policy setting, to solve this. But that's a dirty solution.

                         

                        Kind regards

                        • 9. Re: Do NOT install VSE 8.8 Patch 5 on Hyper-V host if you have pass through disks
                          wwarren

                          We are investigating a couple of issues associated associated with MFEDISK.sys, which like yours, do not have a direct relationship to the driver - like an error that faults it - but has an indirect relationship to the driver, and that removing it avoids the symptom.

                           

                          You could bank on us eventually solving the existing reports and wait for the results, to then see if the resolution works for you, or you could work with Support to allow us to investigate and collect data about your specific scenario - maybe throw test code your way to help prove working theories etc (we have none right now, I'm just saying it might come up).