1 2 3 Previous Next 24 Replies Latest reply: Mar 22, 2012 7:21 AM by Minkus RSS

    8.8 patch 1 killed my Microsoft cluster...

    samd

      I manage everything via EPO 4.6.1 and I approved 8.8 patch 1 yesterday. Therefore all machines, including my nodes, got an installation. I didn't do an uninstall first.  My 3 node cluster that runs Hyper-V VMs was already running 8.8 and I've already long ago followed Microsoft recommendations and Mcafee article for what to exclude when in a cluster. So last night my cluster comes crashing down not too long after the install along with alot of VMs. Got it back going again but two out of the 3 nodes I run will only connect to my CSVs in redirected access. From the research I've done this can happen with an incompatible filter driver. My immediate thought was, Mcafee. Wondering if anyone has seen this. My one node appears ok. I am thinking of getting rid of Mcafee on my nodes altogether.

       

      Message was edited by: samd on 11/23/11 12:40:29 PM EST
        • 1. Re: 8.8 patch 1 killed my Microsoft cluster...
          Minkus

          Hi,

           

          Am experiencing the same problems on my failover cluster with VirusScan Enterprise 8.7i Patch 5 and/or VirusScan Enterprise 8.8 Patch 1.

           

          All CSVs 'hosted' on a node with these products installed end up in Redirected Access mode.

           

          Have raised an SR with McAfee (3-1733698104 if you're interested) and it is currently with Tier 3. Seems to be a known issue as they said others have also reported this. Here's the email I originally sent:

           

          Here is the Microsoft article describing the problem and possible resolutions:
          http://blogs.technet.com/b/askcore/archive/2010/12/16/troubleshooting-redirected -access-on-a-cluster-shared-volume-csv.aspx

          Here is a forum post describing the error I am getting:
          http://social.technet.microsoft.com/Forums/en-US/winserverClustering/thread/0a34 3fb7-d55e-4c82-be7d-35c50320d887/
          Both my CSV's report as online redirected.  Trying to turn off redirected access fails with the following error:
          [Window Title]
          Error
          [Main Instruction]
          The operation has failed.
          [Content]
          The action 'Turn off redirected access for this Cluster Shared Volume' did not complete.
          [^] Hide details  [OK]
          [Expanded Information]
          Error Code: 0x800713b8
          The cluster request is not valid for this object

          I tried setting various exclusions etc as described in the forum post, but nothing helped. In the end I tried switching off the On-Access Scanner and Access Protection completely via ePO, but this did not help either.

          The only solution I have found is to uninstall VSE 8.7i with Patch 5 and reinstall with Patch 4 only.

           

          Here is the error that comes up in the event log when I try to disable redirected access again. I also should mention that I have tried the vssadmin commands mentioned in the Microsoft blog post and they didn’t help.

          Log Name:      Microsoft-Windows-FailoverClustering-Manager/Admin
          Source:        Microsoft-Windows-FailoverClustering-Manager
          Date:          18/10/2011 17:49:40
          Event ID:      4683
          Task Category: Failover Clusters Manager MMC Snapin
          Level:         Error
          Keywords:     
          User:          <redacted>

          Computer:      <redacted>

          Description:
          Failover Cluster Manager failed while managing one or more cluster. The error was 'An error occurred while disabling redirected access for Cluster Shared Volume '\\?\Volume{75c1e7d0-9abf-452e-901b-99e0ccdf25ed}\'.'. For more information see the Failover Cluster Manager Diagnostic channel.

           

          I would advise raising a case so that they know that it is a widespread issue.

           

          Message was edited by: Minkus on 25/11/11 06:42:27 CST
          • 2. Re: 8.8 patch 1 killed my Microsoft cluster...
            sbenedix

            This is currently being investigated by both parties, McAfee and Microsoft. Please raise a Service Request with MS and McAfee so it can be worked from both sides.

            • 3. Re: 8.8 patch 1 killed my Microsoft cluster...
              samd

              Minkus,

               

              Thanks for the links but actually I already ran across both of those in my search for a solution. It definitely happened right after the 8.8 patch 1 was applied automatically by EPO after I checked it in. I have long ago gone through the process of excluding directories like Clusterstorage, devicevolume, etc as articles from MS and Mcafee detail. I also wasn't doing a backups of volumes so I know that's not the issue for me. I used the utility to look at the filter drivers as mentioned in the article and I see two Mcafee filter drivers both of them being mfehidk. One a legacy and one not. I've included the results below. I have two drives in my nodes so therefore I guess that's why I have 4 instances of it. No idea why no instances of the legacy one.

               

              Z:\>fltmc.exe

               

               

              Filter Name                     Num Instances    Altitude    Frame

              ------------------------------  -------------  ------------  -----

              CSVFilter                               3       404900         1

              mfehidk                                         329998.99   <Legacy>

              mfehidk                                 4       321300.00      0

              luafv                                   1       135000         0

               

               

              Z:\>fltmc.exe instances

              Filter                Volume Name                              Altitude        Instance Name      Frame  VlStatus

              --------------------  -------------------------------------  ------------  ---------------------  -----  --------

              CSVFilter             \Device\Mup                             404900       CSVFilter Instance       1

              CSVFilter             C:                                      404900       CSVFilter Instance       1

              CSVFilter                                                     404900       CSVFilter Instance       1

              mfehidk               C:                                      321300.00    mfehidk                  0

              mfehidk                                                       321300.00    mfehidk                  0

              mfehidk               G:                                      321300.00    mfehidk                  0

              mfehidk                                                       321300.00    mfehidk                  0

              luafv                 C:                                      135000       luafv                    0

               

               

              For some reason one of my nodes is fine even though it has 8.8 patch 1 on it. I was planning on rebooting all my nodes to see if they just needed a reboot. But now I think my plan is going to be one of the following:

               

              1. Exclude patch 1 from being applied to my nodes.

              2. Remove patch 1 from EPO altogether.

              3. Remove Mcafee altogether from my nodes. Not like they are fileservers, email servers, etc. Therefore very unlikely they need virus protection.

               

               

              My nodes are too important to risk Mcafee screwing them up like this again and it's not like I have a cluster test bed where I can test out a future patch that might have a problem in only a cluster environment.

               

              Message was edited by: samd on 11/28/11 11:43:33 AM EST
              • 4. Re: 8.8 patch 1 killed my Microsoft cluster...
                U4iA

                Hi Minkus & Smad,

                 

                I have the same problem on a Windows 2008 R2 Hyper-V CVS Cluster with 2 nodes. Wish I read this earlier so I didn't release patch 1 on these nodes. The CSV is in redirected mode and will not come out of it. It seems here it also happend after installing VSE88P1. I'm pretty worries about the amount of problems I'm facing lately with VSE on many type of installations (Hyper-V, CSV Clusters, Terminal Servers, Citrix, etc.). Did removing patch 1 resolve the issue? Please post here if you have a solution. I will make a case later today.

                • 5. Re: 8.8 patch 1 killed my Microsoft cluster...
                  Minkus

                  Samd,

                   

                  Your fltmc.exe output look almost identical to mine:

                   

                  C:\Windows\system32>fltmc

                  Filter Name                     Num Instances    Altitude    Frame
                  ------------------------------  -------------  ------------  -----
                  CSVFilter                               2       404900         1
                  mfehidk                                         329998.99   <Legacy>
                  mfehidk                                 2       321300.00      0
                  luafv                                   1       135000         0

                   

                  Your fltmc instances command is a little different, but I only have 1 drive on my node:

                   

                  C:\Windows\system32>fltmc instances

                  Filter                Volume Name                              Altitude        Instance Name      Frame  VlStatus

                  --------------------  -------------------------------------  ------------  ---------------------  -----  --------

                  CSVFilter             \Device\Mup                             404900       CSVFilter Instance       1

                  CSVFilter             C:                                      404900       CSVFilter Instance       1

                  mfehidk               C:                                      321300.00    mfehidk                  0

                  mfehidk                                                       321300.00    mfehidk                  0

                  luafv                 C:                                      135000       luafv                    0

                   

                  The best way to fix the problem is just to remove VSE 8.8 Patch 1 completely & reinstall with the original VSE release.

                   

                  This seems to resolve the problem until McAfee / Microsoft get their act together & release (another) patch.

                  • 6. Re: 8.8 patch 1 killed my Microsoft cluster...
                    samd

                    I think I'm going to uninstall Mcafee 8.8 patch 1completely. Guessing this will remove all of Mcafee and not just patch 1. I will leave my nodes without virus protection while I investigate whether I can put on Microsoft's own Security Essentials virus protection in a server environment. Not sure. One time of Mcafee screwing up my cluster is one too many for me. They are way too important. The other workstations and servers Mcafee I'm ok with.

                     

                    Edit: Apparently Security Essentials is for small businesses with less than 10 PCs. Not sure if I can install on just 4 servers and therefore it's ok.  Therefore now may have no choice but no A/V or Mcafee bugs.

                     

                    I've had flaky problems every week or so where a path to a CSV gets lost, no harm but I get cluster errors, and just before this happened I started wondering if maybe this was due to maybe a Mcafee filter driver causing intermittent problems.  Now because of this I'm betting it is. I think even 8.8 and 8.7 had problems. Mcafee just hasn't worked out all the quirks/bugs.

                     

                    I have done everything recommended by MS and Mcafee to configure things properly and I get problems anyway. So time to get rid of Mcafee.

                     

                    Message was edited by: samd on 11/29/11 8:49:26 AM EST

                     

                    Message was edited by: samd on 11/29/11 8:54:33 AM EST

                     

                    Message was edited by: samd on 11/29/11 8:56:39 AM EST
                    • 7. Re: 8.8 patch 1 killed my Microsoft cluster...
                      samd

                      Since it seems that others have raised this issue with McAfee and Microsoft should I raise this issue with McAfee as well so we get a resolution ASAP. This isn't some small little problem affecting some obscure product. This is affecting Microsoft's flagship cluster product that people are using to run Microsoft's flagship virtualization software.

                      • 8. Re: 8.8 patch 1 killed my Microsoft cluster...
                        Minkus

                        Hi samd,

                         

                        Yes, I would raise the issue as well. If more people report it, hopefully more resources will get assigned to it, and it will be resolved quicker.

                         

                        If anyone's got time (and money!) to raise it with Microsoft too, that might help.

                         

                        Chris

                         

                        Message was edited by: Minkus on 01/12/11 05:19:16 CST
                        • 9. Re: 8.8 patch 1 killed my Microsoft cluster...
                          samd

                          I just opened a ticket with McAfee

                          1 2 3 Previous Next