1 2 Previous Next 10 Replies Latest reply on Jul 5, 2011 3:32 PM by Katalyst

    FTP replication upload failure

      Replications to my FTP server keep randomly failing. If I manually copy over the "Current" folder from the ePO master repository to my FTP repository the replications don't fail for several days, sometimes weeks. But eventually the FTP replication fails again.

       

      Here is the message from Server Task Log under Global Update Replication:

      2010-07-15 15:39:44.587 INFO FTP Server Beginning replication to site FTP Server.
      2010-07-15 15:39:56.55 WARN FTP Server Failed to upload file Current\EPOAGENT3000\Install\0409\FramePkg.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. ), will retry
      2010-07-15 15:40:01.387 WARN FTP Server Failed to upload file Current\EPOAGENT3000\Install\0409\FramePkg.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. ), will retry
      2010-07-15 15:40:06.37 WARN FTP Server Failed to upload file Current\EPOAGENT3000\Install\0409\FramePkg.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. ), will retry
      2010-07-15 15:40:11.253 WARN FTP Server Failed to upload file Current\EPOAGENT3000\Install\0409\FramePkg.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. ), will retry
      2010-07-15 15:40:16.317 WARN FTP Server Failed to upload file Current\EPOAGENT3000\Install\0409\FramePkg.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. ), will retry
      2010-07-15 15:40:21.347 ERROR FTP Server Failed to upload file Current\EPOAGENT3000\Install\0409\FramePkg.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. )

       

       

      Here is a copy of what it looks like in EpoApSvr.log:

       

      20100719090641    I    #34064    SIM_InetMgr    Uploading file FramePkg.exe from session 1, LocalDir=C:/PROGRA~1/McAfee/EPOLIC~1/DB\Software\Current\EPOAGENT3000\Install\0 409, RemoteDir=Current\EPOAGENT3000\Install\0409
      20100719090641    I    #34064    NaInet      FTP session 7 Uploading file: /VirusScanDats/Current/EPOAGENT3000/Install/0409/FramePkg.exe to FTP Server: ftp1.dcim.com
      20100719090645    E    #34064    naInet      FTP session 7 socket 6876 SendTimeout() failed on send, WSAGetLastError() = 10054
      20100719090645    E    #34064    naInet      FTP session 7 send on socket 6876 failed, WSAGetLastError = 2
      20100719090645    I    #34064    SIM_InetMgr    Upload file FramePkg.exe failed in session 1, nainet ret=0
      20100719090646    I    #34064    SiteMgr     ReplicationUploadFile: Failed to upload file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, hr=-2147467259, retry limit remaining: 4
      20100719090646    I    #34064    SiteMgr     ReplicationUploadFile: Uploading file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, retry limit remaining: 4
      20100719090648    I    #34064    SIM_InetMgr    Uploading file FramePkg.exe from session 1, LocalDir=C:/PROGRA~1/McAfee/EPOLIC~1/DB\Software\Current\EPOAGENT3000\Install\0 409, RemoteDir=Current\EPOAGENT3000\Install\0409
      20100719090648    I    #34064    NaInet      FTP session 7 Uploading file: /VirusScanDats/Current/EPOAGENT3000/Install/0409/FramePkg.exe to FTP Server: ftp1.dcim.com
      20100719090648    I    #34064    naInet      FTP session 7 Reply data socket not empty before CMD
      20100719090648    I    #34064    naInet      FTP session 7 Flushing Receive CMD buffer, read: 226 Transfer complete.

       

      It doesn't look like its a timeout or network performance issue, because 95% of the time the replication fails on FramkePkg.exe. The replication also fails over the weekend when nobody is using the FTP server.

       

      Should I look at having the FTP repository not update during the incremental global update replication and have a task that does a full reploication instead?

       

      Thanks,

      Tom

        • 1. Re: FTP replication upload failure

          I have ePO 4.5.0 (Build 937)

          • 2. Re: FTP replication upload failure
            jstanley

            So the error in the log is essentially "WSAGetLastError() = 10054" which translates to "connection reset by peer". Here are some possibilities that could lead to that error:

            * You have specified the wrong URL for your FTP site when you defined the distributed repository. To test this edit the distributed repository and on the "server" tab you should see the URL. Plug this into IE or explorer and see if you can browse the site

            * The credentials you have entered for the replication are not valid. To test this logon to the server with whatever account you specified for the replication credentials or use the "run as" command and try to access the FTP site through IE or Explorer. Then try to create and delete a file to confirm you have the required permissions for a repository replication.

             

            As it looks specifically like its failing to replicate the agent package it could be a problem with the repository itself. Try deleting the contents of the repository and re-replicating (this will in effect do a full replication). If your distributed repository had become corrupt this should correct the problem.

            • 3. Re: FTP replication upload failure

              Jeremy,

               

              Thank you for replying.

               

              The replication does start and other files get transferred, so I dont think the credentials are the cause of the issue.

               

              I havn't tried deleting all of the contents of the repository, but I have copied all of the contents from the master repository to the FTP repository and this solves the issue for a random amount of time. I think you are on to something in that it is something with the repository getting corrupted, or maybe the file getting corrupted but I don't understand how. I do notice that FramkePkg.exe is always a small random file slize when the replication fails. So it starts transferring the file but doesn't complete the transfer.

               

              I have 2 othre repositories (non FTP) and they don't have any replication issues. I have even added in an exclusion the directory with the repositry doesn't get scanned, just in case the scanning of the files as they are getting transferred was causing a problem.

               

              Just seems like a strange issue. And with it being a random problem and not happening every day, it makes it hard trying to figure out what is causing the problem.

               

              Do you think it would help if I remove my FTP server from the global incremental replication and create a task to do a full replication each day?

               

              Thanks,

              Tom

              • 4. Re: FTP replication upload failure

                Ok, so this morning I checked in AntiSpyware Enterprise Module 8.7 into ePO and when my global replicaion raen a couple hours later it failed to my FTP server but not to my other repositories. I then deleted all of the contents in my FTP repository and did ran a Full Replication to my FTP repository though ePO.

                 

                Amazingly enough the full replication failed on FramkePkg.exe, even when the repository was empty before the replication started.

                 

                 

                20100812114503    I    #35880    SiteMgr     ReplicationUploadFile: Uploading file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, retry limit remaining: 2
                20100812114505    I    #35880    SIM_InetMgr    Uploading file FramePkg.exe from session 1, LocalDir=C:/PROGRA~1/McAfee/EPOLIC~1/DB\Software\Current\EPOAGENT3000\Install\0 409, RemoteDir=Current\EPOAGENT3000\Install\0409
                20100812114505    I    #35880    NaInet      FTP session 71 Uploading file: /VirusScanDats/Current/EPOAGENT3000/Install/0409/FramePkg.exe to FTP Server: ftp1.dcim.com
                20100812114505    I    #35880    naInet      FTP session 71 Reply data socket not empty before CMD
                20100812114505    I    #35880    naInet      FTP session 71 Flushing Receive CMD buffer, read: 226 Transfer complete.

                 

                20100812114508    E    #35880    naInet      FTP session 71 socket 7736 SendTimeout() failed on send, WSAGetLastError() = 10054
                20100812114508    E    #35880    naInet      FTP session 71 send on socket 7736 failed, WSAGetLastError = 2
                20100812114508    I    #35880    SIM_InetMgr    Upload file FramePkg.exe failed in session 1, nainet ret=0
                20100812114508    I    #35880    SiteMgr     ReplicationUploadFile: Failed to upload file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, hr=-2147467259, retry limit remaining: 1
                20100812114508    I    #35880    SiteMgr     ReplicationUploadFile: Uploading file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, retry limit remaining: 1
                20100812114510    I    #35880    SIM_InetMgr    Uploading file FramePkg.exe from session 1, LocalDir=C:/PROGRA~1/McAfee/EPOLIC~1/DB\Software\Current\EPOAGENT3000\Install\0 409, RemoteDir=Current\EPOAGENT3000\Install\0409
                20100812114510    I    #35880    NaInet      FTP session 71 Uploading file: /VirusScanDats/Current/EPOAGENT3000/Install/0409/FramePkg.exe to FTP Server: ftp1.dcim.com
                20100812114510    I    #35880    naInet      FTP session 71 Reply data socket not empty before CMD
                20100812114510    I    #35880    naInet      FTP session 71 Flushing Receive CMD buffer, read: 226 Transfer complete.

                 

                20100812114513    E    #35880    naInet      FTP session 71 socket 7068 SendTimeout() failed on send, WSAGetLastError() = 10054
                20100812114513    E    #35880    naInet      FTP session 71 send on socket 7068 failed, WSAGetLastError = 2
                20100812114513    I    #35880    SIM_InetMgr    Upload file FramePkg.exe failed in session 1, nainet ret=0
                20100812114513    I    #35880    SiteMgr     ReplicationUploadFile: Failed to upload file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, hr=-2147467259, retry limit remaining: 0
                20100812114513    I    #35880    SiteMgr     ReplicationUploadFile: Uploading file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, retry limit remaining: 0
                20100812114515    I    #35880    SIM_InetMgr    Uploading file FramePkg.exe from session 1, LocalDir=C:/PROGRA~1/McAfee/EPOLIC~1/DB\Software\Current\EPOAGENT3000\Install\0 409, RemoteDir=Current\EPOAGENT3000\Install\0409
                20100812114515    I    #35880    NaInet      FTP session 71 Uploading file: /VirusScanDats/Current/EPOAGENT3000/Install/0409/FramePkg.exe to FTP Server: ftp1.dcim.com
                20100812114515    I    #35880    naInet      FTP session 71 Reply data socket not empty before CMD
                20100812114515    I    #35880    naInet      FTP session 71 Flushing Receive CMD buffer, read: 226 Transfer complete.

                 

                20100812114518    E    #35880    naInet      FTP session 71 socket 6952 SendTimeout() failed on send, WSAGetLastError() = 10054
                20100812114518    E    #35880    naInet      FTP session 71 send on socket 6952 failed, WSAGetLastError = 2
                20100812114518    I    #35880    SIM_InetMgr    Upload file FramePkg.exe failed in session 1, nainet ret=0
                20100812114518    e    #35880    SiteMgr     ReplicationUploadFile: Failed to upload file FramePkg.exe to site FTP Server::Current\EPOAGENT3000\Install\0409, hr=-2147467259, retry limit reached
                20100812114518    e    #39868    SiteMgr     WorkQueueUploadFile: Failed to upload file FramePkg.exe to site FTP Server
                20100812114518    e    #39868    SiteMgr     ReplicateSite: Failed to upload directory tree to site FTP Server
                20100812114518    I    #39868    SrvEvtInf    Generating Event
                20100812114518    I    #39868    naInet      FTP session 71 Reply data socket not empty before CMD
                20100812114518    I    #39868    naInet      FTP session 71 Flushing Receive CMD buffer, read: 226 Transfer complete.

                 

                 

                I then manually copied over all of the contents from my master repository to the FTP repository and ran a Full Replication and the replication failed again on FramkePkg.exe! I then copied over just the folder Current\EPOAGENT3000 and ran a incremental replication and the replication succeeded. I then ran a full replication again just to see what would happen and the full replication failed.

                 

                So basically it looks like the replication is failing anytime it has to replace FramkePkg.exe.

                 

                I have also had the replications fail to my FTP repository over the past couple days after I have checked in new packages.

                 

                Current\VIRUSCAN8700\HotFix\539488\0000\VSE87HF539488.exe to site ftp1.dcim.com:21, error code 2 ( The system cannot find the file specified. )    [HotFix for VSE 8.7]

                Failed to upload file Current\VSEMAS870000\Install\0000\VSE87MAS.exe to  site ftp1.dcim.com:21, error code 2 ( The system cannot find the file  specified.  ) [AntiSpyware Enterprise 8.7i]

                 

                So there looks to be a larger issue with the repository replications failing when certian files need to be replaced or pushed to the FTP repository for the first time.

                • 5. Re: FTP replication upload failure

                  I am seeing the same sorts of results. It all appears to have stemmed from installing ePO 4.5 Patch 3, which was done on the server around Oct 9th. Since then, we have seen this issue on several Super Agents (repositories). I have the same Super Agents that fail on every task and then the sitestat.xml fails to upload to random Super Agents.

                   

                  I have tried:

                  - Created New incremental replication tasks - same results

                  - Remove data from Current Branch in the Super Agents (incremental then performs a full to those sites (McAfee's recommendation)) - Same results

                  - Created a FULL replication to the same 2 repositories that have the largest amount of warnings - task failed in 6 minutes for a full replication to 2 SAs

                   

                  I have been told that 10054 indicates a network issue, but nothing has changed in the network, except installing Patch 3 on the ePO.

                   

                  This is now affecting clients across the world, as they are now looking across WAN links to find their updates.

                   

                  2010-10-22 08:59:15.357
                  INFO
                  Beginning replication to site xxxxxxxx_xxxxxxxx.
                  2010-10-22 08:59:17.94
                  WARN
                  Failed to upload file SiteStat.xml to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 08:59:19.973
                  WARN
                  Failed to upload file SiteStat.xml to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 08:59:22.02
                  WARN
                  Failed to upload file SiteStat.xml to site xxxx, error code 5 ( Access is denied. ), will retry
                  2010-10-22 08:59:24.043
                  WARN
                  Failed to upload file SiteStat.xml to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 08:59:26.077
                  WARN
                  Failed to upload file SiteStat.xml to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 08:59:58.28
                  WARN
                  Failed to upload file catalog.z to site xxxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:00.58
                  WARN
                  Failed to upload file catalog.z to site xxxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:04.867
                  WARN
                  Failed to upload file catalog.z to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:06.337
                  WARN
                  Failed to upload file catalog.z to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:24.447
                  WARN
                  Failed to upload file Current\BOCVSE__1000\BocDet_VSE.McS to site xxxx , error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:43.52
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\BocInstall.mcs to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:45.663
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\BocInstall.mcs to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:00:47.757
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\BocInstall.mcs to site xxxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:01:05.147
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\PkgCatalog.z to site xxxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:01:07.357
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\PkgCatalog.z to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:01:09.49
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\PkgCatalog.z to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:01:11.63
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\PkgCatalog.z to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. ), will retry
                  2010-10-22 09:01:13.643
                  WARN
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\PkgCatalog.z to site xxxx , error code 10054 ( An existing connection was forcibly closed by the remote host. )will retry
                  2010-10-22 09:01:15.667
                  ERROR
                  Failed to upload file Current\BOCVSE__1000\DAT\0000\PkgCatalog.z to site xxxx, error code 10054 ( An existing connection was forcibly closed by the remote host. )will retry

                   

                  We uploaded a MER to the SR this morning, level 8 capture. Hopefully this will shed some light

                   

                   

                  Any advice is appreciated

                  • 6. Re: FTP replication upload failure

                    All I got from McAfee was that is was our network or or firewall creating the issue despite the fact that if I copied our master repository to our FTP repository myself using the FTP protocol I had no issues. But anytime EPO did a full or incrimental repository replication automatically it would fail to our FTP repository.

                     

                    I ended up switching our FTP repositry to a UNC repository and no longer have any issues, keep in mind McAfee said it was my network that was the problem. The problem is obviously with the way the EPO server is replicating to FTP repositories. Its sad that after months of problems and contacting support over a dozen times I got nowhere and all they did was say it had something to do with our network. I got sick of being jerked around and gave up trying to troubleshoot the problem. I also uploaded multiple MER's to McAfee with the same results. Hopefully yours will go better!

                     

                    If you manually copy the repositry files to your FTP repositories does tha copy fail? Log into your EPO server and FTP to your FTP repository and copy your master repository over . If that fails, then I would say you have a bigger issue. If not, then the only time the replication is failing is when its running through EPO's replication process, which would point the problem at EPO and not your network.

                    Make sure you try copy and replacing the files, and also deleting all of the files in the FTP repository and then copy everything over. Be sure to FTP into your FTP repositry using the same credentials the EPO server uses when it does its replications. Also check to see if the replications to your Super Agents fail on the same file or same file type. My issue was with .exe files, since FramePkg.exe was the first .exe to be copied over it would fail on that one first, but I realized after some extra troubleshooting that it was actually all .exe files not just 1 file.

                     

                    Hope this helps some.

                    Tom

                    • 7. Re: FTP replication upload failure
                      jkussow

                      Have you checked available disk space whereever your FTP repository resides?  We had something similar happen and it was because the disk was full.  I've yet to understand why but our FTP repositories are all 15 GB in size even though all our SuperAgent and UNC repositories are all only 800 MB. I've deleted and recreated them and nothing changed. In any case we added disk to the failing FTP repository and all our issues disappeared.

                      • 8. Re: FTP replication upload failure

                        Actually I forgot to mention. We are not using FTP repositories. We are using Super Agents, that are our SMS servers, that we have added the Super Agent information to. They have been working great and have plenty of storage on them.


                        Everything was working great until Patch 3 was applied. I am not sure what changed in the configuration, but it appears that something has changed.

                        • 9. Re: FTP replication upload failure

                          I am also getting a similar kind of issue if I am using a FTP repository.

                           

                          Failed to upload file SiteStat.xml to site221.122.113.208:11121, error code 10054 ( An existing connection was forciblyclosed by the remote host. ), will retry

                           

                          I am able to connect to the same FTP repository from any location with the credentials provided and able to write and read from the same.

                           

                          When I click on the test credentials it takes around 5 minutes and then gives me a reply the credentials are incorrect.(server to repository). If I enter the same credentials under agent to repository it is giving credentials valid.

                           

                          Not sure, what is happening and McAfee wants the MER log for this.

                          1 2 Previous Next