1 2 3 Previous Next 24 Replies Latest reply: Mar 22, 2012 7:21 AM by Minkus Go to original post RSS
      • 10. Re: 8.8 patch 1 killed my Microsoft cluster...
        Minkus

        Received this from McAfee just now. I shall be contacting Microsoft to get the fix shortly - kudos to McAfee if this wasn't their fault!

         

        After our development team analyzed all the data and weighted the options these are the conclusions of the investigation:

         

        Solution

        Microsoft has confirmed that a Cluster (Windows 2008 R2) will put a Cluster Server Volume (CSV) in redirected mode if a filter’s Altitude is not an integer.

         

        Contact Microsoft to obtain a fix to address this issue.

         

        Workaround

        Install a version of VSE that has not adopted the new drivers, for example:

         

        VSE 8.7 Patch 4 or below

        VSE 8.8

         

        This has been documented in our knowledge base under KB73596.

         

        Please let me know if you have any questions or if you can confirm we can now close this support case.

        • 11. Re: 8.8 patch 1 killed my Microsoft cluster...
          samd

          Looks to be a bug in Microsoft however Mcafee shoulda known better #1 as Microsoft's filter driver spreadsheet http://msdn.microsoft.com/en-us/windows/hardware/gg462961.aspx they have available on their website doesn't have any filter driver with decimal places in it. And it certainly doesn't excuse Mcafee from properly testing this patch especially in so obvious an environment like a Microsoft Cluster using CSVs. Not like that's an oddball setup. Come on. Mcafee is owned by Intel. They should have done better QA testing. This is a major bug in my opinion. Anything that brings down a whole cluster is a major bug.

           

          I'm wondering if this is fixed in R2 SP1 as I haven't applied that yet. Would save me from opening a ticket with Microsoft.

           

          Message was edited by: samd on 12/6/11 9:26:43 AM EST

           

          Message was edited by: samd on 12/6/11 9:27:38 AM EST
          • 12. Re: 8.8 patch 1 killed my Microsoft cluster...
            Minkus

            Have contacted Microsoft and opened a case - they are aware of the issue & are working on a fix to resolve it.

             

            Only suggestion at the moment is to roll back to the previous patch level & wait for the fix to be released.

             

            Will post here when I hear from them again. Got the impression it might be a while due to Microsoft's usual extensive testing regime etc... but it will happen.

            • 13. Re: 8.8 patch 1 killed my Microsoft cluster...
              Minkus

              P.S. No, it's not fixed in R2 SP1 - have been running this for ages.

              • 14. Re: 8.8 patch 1 killed my Microsoft cluster...
                samd

                I planned on rolling this back on my cluster when I get a chance. Thanks for opening the case with Microsoft. I bet Mcafee will address this in a later patch or release as well. I can only hope.

                • 15. Re: 8.8 patch 1 killed my Microsoft cluster...
                  samd

                  Well I removed 8.8 patch 1 and EPO reapplied it quickly as I watched since policy enforcement every 5 minutes was happening. Luckily all VMs and cluster were down at time. I changed policy to only install previous branch for Virusscan. Thought I was all set. Then like 3:30 am Sunday morning cluster comes crashing down. Couldn't see what version of Mcafee was on there (if somehow patch 1 got reapplied for instance) as my first thing was get the cluster back up so I removed Mcafee altogether. But the crash occurred on all nodes and I have this in event logs at 3:06 am. I have nothing scheduled at this time that I can see as my 8.8 install I didn't schedule a full scan. Maybe a DAT update happened?

                   

                  The description for Event ID 259 from source McLogEvent cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

                   

                  If the event originated on another computer, the display information had to be saved with the event.

                   

                  The following information was included with the event:

                   

                  The scan found detections. Scan engine version 5400.1158 DAT version 6555.

                   

                  I see no evidence that an install of 8.8 patch 1 was attempted. I am really confused now and can only guess a remnant of 8.8 patch 1 was around and killed things. Since this time I have totally removed Mcafee including agents from my nodes. I'm tired of this. In the future I may install 8.7 again with patch 4 which I believe won't have issues. I know patch 5 will. 8.7 is not in my epo repository so it won't cause problems. I will do a test of one node and move all VMs and CSVs off beforehand and do a test for one day.

                  • 16. Re: 8.8 patch 1 killed my Microsoft cluster...
                    marcphilips

                    Hi Minkus

                    Can you give me your MS case ID so I can have our TAM attach ours and some others customers so we can so more psuh on getting the solution implemetented.

                    Thanks

                    • 17. Re: 8.8 patch 1 killed my Microsoft cluster...
                      Minkus

                      Hi,


                      My case ID is 111120644043954

                       

                      Kind regards,

                      Chris

                      • 18. Re: 8.8 patch 1 killed my Microsoft cluster...
                        Minkus

                        Hi,

                         

                        Found this Microsoft blog post today. The part at the end sounds promising...

                         

                        There is an issue with Cluster Shared Volumes and McAfee VirusScan Enterprise that I wanted to pass along.  When installing McAfee VSE 8.7 Patch 5 or 8.8 Patch 1, the CSV drives will go into redirected mode and will not go out of it.

                        The reason for this is that the McAfee filter driver (mfehidk.sys) is using decimal points in the altitude to help in identifying upgrade scenarios for their product.  The Cluster CSV filter only accepts whole numbers and puts the drives in redirected access mode when it sees this decimal value.

                        When seeing this, if you run FLTMC from an administrative command prompt, you may see something similar too:

                        C:\> fltmc

                        Filter Name    Num Instances      Altitude    Frame
                        ------------------------------------------------------
                        CSVFilter            2            404900        0
                        mfehidk                           329998.99   <Legacy>
                        mfehidk              2            321300.00     0

                        If you were to generate a Cluster Log, you would see the below identifying that it cannot read the altitude value properly.

                        INFO [DCM] FsFilterCanUseDirectIO is called for \\?\Volume{188c44f1-9cd0-11df-926b-a4ca2baf36ff}\
                        ERR  mscs::FilterSnooper::CanUseDirectIO: BadFormat(5917)' because of 'non-digit found'
                        INFO [DCM] PostOnline. CanUseDirectIO for C2V1 => false

                        McAfee has released the following document giving a temporary workaround.

                        Cluster Shared Volumes (CSV) status becomes Online (Redirected access)
                        https://kc.mcafee.com/corporate/index?page=content&id=KB73596

                        Microsoft is aware of the problem and currently working on a fix.  When this fix is available, this will be updated and a new KB Article will be created with the fix.

                        John Marlin
                        Senior Support Escalation Engineer
                        Microsoft Enterprise Platforms Support

                        • 19. Re: 8.8 patch 1 killed my Microsoft cluster...
                          samd

                          I still don't have any Mcafee VirusScan on my Cluster nodes because of this because I couldn't stop my EPO from updating to SP1.  I'll be watching this forum in hopes that some people here have a cluster test environment. I don't so I can't take a chance of installing this until I know it won't cause a problem again.