9 Replies Latest reply on Dec 5, 2016 10:15 AM by tao

    Random SuperAgent Repository Replications keep failing on a daily basis

    robertjdrake

      I have 50 SuperAgents across the globe acting as repositories for our various sites and they are set to replicate at various times of the day as to spread the load on ePO.

       

      Most days I am getting between 2 - 5 failed "Distributed Repository Replication failures" they seem to be pretty random which is puzzling, some days no failures whatsoever.

       

      There are a few repeat offending machines, however there's seems no patterns whatsoever as to which ones fail.

       

      When i try and view the packages in ePO>Distributed Repositories for the failed SA's i get "Site Catalog not found" but if i reboot the machine and manually kick off the replication task it completes, if i don't reboot the SA sometimes the next day it replicates okay without it being rebooted.

       

      I have checked that Lazy caching is disabled, and i can't find anything anywhere with a definitive fix as to what is causing this problem.

       

      I'm running ePO 5.3, with a mixture of Agent versions 4.8 & 5.0.1 & 5.0.2, it doesn't seem to matter which version of the Agent is installed.

        • 1. Re: Random SuperAgent Repository Replications keep failing on a daily basis
          meforum

          Hi,

           

          how the remote sites are connected? I had such things with connections over "DSL" that is disconnected once a day by the provider. When this happened during replication - the replication failed (and I also had no access to the files/catalog on that repository - but could do a fresh replication manually). So maybe your connection is somehow unstable?

           

          And keep in mind that dist. repositories not always "better" than direct synch with ePO (e.g. regarding bandwidth) - it depends e.g on how many clients do you have on the remote site. And e.g. you could only use the remote repositories for e.g. software update/install (new product version) but pull daily DATs directly from ePO-

          • 2. Re: Random SuperAgent Repository Replications keep failing on a daily basis
            robertjdrake

            Hi - All are connected on our WAN links, some sites are 24/7 sites so that would rule out ISP disconnections.

            We have alot of clients across the network 5000+ so SA's are definitely the way to go, our SA's are there to replicate all applicable packages.

            • 4. Re: Random SuperAgent Repository Replications keep failing on a daily basis
              andrew.robinson

              I have similar issues. Some of my SA repos work, some don't. Even after a full 'reset' some repos  work but there are still bad repos where I cant see the packages from ePO, although replication claims it has worked (I can see some of the files locally). In addition, I don't think my clients are doing a very good job with 'ping time' and I cant even find the right log for that action these days. I thought the McAfee Agent Common Service service running as a Local Service might have been an issue, but even using Local System still fails on the bad ones. Also, I have some DRs and some of those are barely being used, which makes no sense as they are on bigger sites with no SA repos. Also, I can see clients connected to the bad SAs on port 8190 - I know connectivity works. I have a ticket open with McAfee. I await their attention.............

              • 5. Re: Random SuperAgent Repository Replications keep failing on a daily basis
                tao

                This message may mean that the SA cannot connect to source repository due to; network throttling or network congestion - I'm wonder if this may be due to "to replicate at various times".  Meaning, I manage over 30,000 nodes, with 80+ SA's with  5 specific scheduled replication task set for very specific times of the day (non-business hours or low network traffic hours) - all 5 task run within a half hour of each other, back to back. 

                 

                As for the confirming if the Agent is accessing the correct SA - Under McAfee Agent > Repository; I have configured the client policy: "Use this repository list" & "Subnet distance"< this has to be dialed in; play with the maximum number of hops>.

                 

                If you have moved to Agent 5.0.X; Agent Activity log has been removed from the Web Access Agent Activity Monitor: ePO Agent Log: 5.0.3.316 vs 4.8.0.1995

                 

                I really hope that the removal of the "Enable Agent Activity log" was just an oversight and that it will be reintroduced in Agent 5.0.5 ... Any Intel employees PLEASE chime in; would love to hear from you:

                McAfee Corporate KB - McAfee Agent 4.x Logging policy does not migrate to 5.x remote Logging policy KB82896

                • 6. Re: Random SuperAgent Repository Replications keep failing on a daily basis
                  andrew.robinson

                  Thanks Tao. I have McAfee engaged now. We are at the usual stage of upping logging levels etc. Meanwhile I can re-iterate that I see many client connections on my failing SAs. Its as though the connections do not close and the service gets hung. When I restart macmnsvc.exe it kills the connections of course and then I can see the packages again from ePO and maybe the repo works for a while. I have to check that bit. I set one repo to auto-restart the macmnsvc.exe service every two hours yesterday. I came back to look today and the service is in 'stopping' state.

                   

                  Repos that were good yesterday are no longer good today. Same issue. However, some repos stay good.

                   

                  The McAfee guy says this is how to find the ping times in the logs, but I cant find them. We can use subnet hops as our network team are random with this!


                  Agent logs and McScript.

                  Masvc for agent log  for MA5.0.x

                  Look for something similar to below

                  ping   ICMP::Ping - Pinging “repo” 

                  ping   ICMP::Ping - Avg ping time is “0”

                  The best approach is to debug first. The issue might not just be related to a ping problem.      

                  • 7. Re: Random SuperAgent Repository Replications keep failing on a daily basis
                    tao

                    Service gets hung; is it being scanned?  You may already know about McAfee Profiler; if not, it's a very useful tool in troubleshooting:

                     

                    McAfee Profiler captures top processes and files that are accessed by the VirusScan Enterprise (VSE) On-Access Scanner (OAS). Based on the data collected, an administrator can choose files or processes to exclude from scanning to lessen the impact on the system.

                     

                    https://support.mcafee.com/profiler

                    • 8. Re: Random SuperAgent Repository Replications keep failing on a daily basis
                      andrew.robinson

                      Hi Tao - I have use profiles in the past but never got much out of it. I will look again. However, I don think this is anything to do with scanning. Its all about the client connections sitting on the SAgent repo and not closing I think. Could you maybe check your repos to make sure you can see the packages for all of them from ePO? Or run the repo report for the last 24 hours to make sure all are working. I would be interested to hear if you have the same issue or not.....its an easy thing to miss and just assume they are all OK.

                      • 9. Re: Random SuperAgent Repository Replications keep failing on a daily basis
                        tao

                        Checked and did a spot check; SA's are replicating and packages are available ... client connections; I'm using a client task with " time for the client update task ( along with "Run missed task").