9 Replies Latest reply on Aug 31, 2012 7:57 AM by diwi

    EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll

    bretzeli

      Hello,

       

      We have a customer with EPO problems:

       

      Error: EventID 1000, Apache.EXE Crash.

      How do i see: Last Contact of systems shows the date of the Event AND the DLP Events (In Monitoring) also show that date. If i force a Framework contact on a client or server i get an error at that moment.

       

      Server 2008R2, EPO4.6, Agent 4.6 on all clients and Servers, Around 500+ clients, English OS, SQL Express 2005 SP2, Version 9.00.4035.00.

       

       

      1 The system was migrated since EPO 4.0/4.5 and now 4.6. We had this problem before running under Server 2003R2 and EPO 4.5/4.6

      2 We then migrated the full EPO to Server 2008R2 and EPO 4.6 including the existing DB on SQL Express. Hoping to get rid of the apache.exe because mcafee could not solve it...

       

      We had Tier2/3 recommandation of running the MS Crash Analyzer tracking the apache.exe but they could not say what it was.

       

      Since we did a fresh install of Server 2008R2 and the EPO 4.6 (Apache/Orion) they only thing which is still the same is the SQL Database and our settings.

      I did read some storys about reindexing the SQL DB but we did not do that at the moment. Currently we are running SQL 2005 SP2, Version 9.00.4035.00.

       

      Newly, and that was the reason to migrate to 2008R2 we have DLP 9.1.1 installed. I am not sure if we have the crash of the apache.exe because

      of those components more often now but it seems so. The DLP in full seems to be running fine if the Apache underneath is running fine. I don't see

      any special problem there. It just the amount of load maybe that makes tha apache.exe crash more often.

       

      The Servlog.txt shows the "Busy connection 245" error but i am not sure if this the source or a follow up problem of my first problem.

       

      Thank you for any help or hints in any direction.

      Mike

       

      Logfile: C:\Program Files (x86)\McAfee\ePolicy Orchestrator\DB\Logs\serverlog.txt

       

      20110913105605 E #09112 mod_epo  Server is too busy (245 connections) to process request

      20110913105605 X #09112 mod_epo  epo request processed, rc=503, session ID=691695, session time=0ms

      20110913105605 X #09112 mod_epo  Current open sessions=244

       

       

      Event 1000:

       

      Faulting application name: Apache.exe, version: 2.2.9.0, time stamp: 0x48d60715

      Faulting module name: MSVCR80.dll, version: 8.0.50727.4940, time stamp: 0x4ca2b271

      Exception code: 0xc0000005

      Fault offset: 0x0001500a

      Faulting process id: 0xb74

      Faulting application start time: 0x01cc6e04e09ca88d

      Faulting application path: C:\Program Files (x86)\McAfee\ePolicy Orchestrator\Apache2\bin\Apache.exe

      Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d 08cc06a442b34fc\MSVCR80.dll

      Report Id: 9289ee96-da4d-11e0-adec-00505681000c

        • 1. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
          JoeBidgood

          Do you by any chance have any machines that are used by a large number of users, for example a Citrix server or something similar?

           

          Thanks -

           

          Joe

          • 2. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
            bretzeli

            Hello Joe,

             

            Not not at that customer, we have other Citrix Customer running same EPO 4.6 constaltion without the apache.exe error. That customer still has multiply users "Per machine" since it's healthcare and they have up to 20+ users sharing the same machine.

             

            Di you think about the 254+ connections? They may be a problemn with realy large terminal servers i guess.

            • 3. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
              JoeBidgood

              The symptoms you describe are an exact match for an issue we have at the moment where Apache crashes if it is sent too large a list of users - usually, this means things like Citrix servers, but not exclusively.

              This is fixed in ePO 4.5 Patch 5, which just when on managed release, and in ePO 4.6 Patch 1, which is due to go on managed release next week.

              Given this, I would strongly recommend that you open a case with Support and request to take part in the managed release phase of Patch 1.

               

              Regards -

               

              Joe

              • 4. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
                bretzeli

                Thank you: Ticket, 3-1686673611, Sumbited by Mcafee ELITE Partner

                • 5. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
                  bretzeli

                  This has not been the solution after 4 days it crashed again, we are wating for EPO 4.6 SP1 urgently.

                   

                   

                  Hello,

                   

                  It's been running for 2 days now without any crash.

                   

                  We are not sure but we have maybe solved the problem or let's say found the source for the Apache crash. We did two things at once (bad) so you can choose what you want to do ;-)

                   

                  a) Joe, mentioned that the have problems with high amount of users like on Citrix Server which may cause the problem. The "Rogue Sensor" is also sending large amounts of data BACK to the EPO. We have deinstalled a  Rogue Sensor (This may could go in direction of: Server is too busy (245 connections) to process request). Only the coder of the multi layer constelation with Tomcat/Apache, NT-Services can say what and how.

                   

                  b) We had a rather large SQL DB running under SQL Express. It clearly was under the 4.0 and 3.5 GB but there where poeple REBUILDING the index of the SQL DB. The Apache.exe crash problem often happens to people

                  who migrated from EPO 4.0 to 4.5 and then to 4.6. This are often people with large amounts of data. With large i mean anything around 2-3.5GB+ SQL DB-data.

                   

                  Now people say why don't they use SQL or run in on a consiltaed SQL Cluster. The Remote SQL is a security Risk for ma. If a virus floods the Net and the EPO can't reach the SQL you have problems and the EPO is kind of useless in such a breakout.

                   

                  Most even large customers want pay an Extra CHF 2000.- / EUR 1500.- for a full SQL on the EPO so they stick with SQL Express.

                   

                  Here some info i collected during the years, your SQL DBA may be a better source ;-)

                   

                  http://www.butsch.ch/post/Mcafee-EPO-Server-4X-Database-or-Space-growing-EPOeven ts.aspx

                   

                  Nachricht geändert durch bretzeli on 19.09.11 02:13:55 CDT
                  • 6. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
                    bretzeli

                    Update,

                     

                    This case is still unsolved even after migrating the EPO to Server 2008R2. We have two crashdumps showing identical behaviour with the Crypt DLL msvcr80.dll from MS and Apache.exe

                    We have just uploaded a Crash Dump and MER to Mcafee and are waiting for updates from them.

                     

                     

                    Analysis Summary

                    Type

                    Description

                    Recommendation

                      Error

                    In Apache__PID__3004__Date__09_25_2011__Time_04_16_22PM__427__Second_Chance_Except ion_C0000005.dmp the assembly instruction at msvcr80!memcpy+5a in C:\Windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d0 8cc06a442b34fc\msvcr80.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x0c5d9000 on thread 9

                    Please follow up with the vendor Microsoft Corporation for C:\Windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d0 8cc06a442b34fc\msvcr80.dll

                      Error

                    In Apache__PID__3012__Date__09_30_2011__Time_01_15_14AM__557__Second_Chance_Except ion_C0000005.dmp the assembly instruction at msvcr80!memcpy+5a in C:\Windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d0 8cc06a442b34fc\msvcr80.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x0da9b000 on thread 9

                    Please follow up with the vendor Microsoft Corporation for C:\Windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d0 8cc06a442b34fc\msvcr80.dll

                      Warning


                    An Operating System newer than Windows Vista has been detected. At the time this script was written it was only tested on Operating Systems versions of Windows Vista and prior. Analysis results for Apache__PID__3012__Date__09_30_2011__Time_01_15_14AM__557__Second_Chance_Excepti on_C0000005.dmp may be inaccurate or incomplete.

                    It is recommended that you verify whether there is a newer version of Debug Diagnostics available or a more compatible version of this analysis script.

                      Information

                    DebugDiag determined that this dump file (Apache__PID__3004__Date__09_25_2011__Time_04_16_22PM__427__Second_Chance_Excep tion_C0000005.dmp) is a crash dump and did not perform any hang analysis. If you wish to enable combined crash and hang analysis for crash dumps, edit the CrashHangAnalysis.asp script (located in the DebugDiag\Scripts folder) and set the g_DoCombinedAnalysis constant to True.

                      Information

                    DebugDiag determined that this dump file (Apache__PID__3012__Date__09_30_2011__Time_01_15_14AM__557__Second_Chance_Excep tion_C0000005.dmp) is a crash dump and did not perform any hang analysis. If you wish to enable combined crash and hang analysis for crash dumps, edit the CrashHangAnalysis.asp script (located in the DebugDiag\Scripts folder) and set the g_DoCombinedAnalysis constant to True.

                      Warning


                    An Operating System newer than Windows Vista has been detected. At the time this script was written it was only tested on Operating Systems versions of Windows Vista and prior. Analysis results for Apache__PID__3004__Date__09_25_2011__Time_04_16_22PM__427__Second_Chance_Excepti on_C0000005.dmp may be inaccurate or incomplete.

                    It is recommended that you verify whether there is a newer version of Debug Diagnostics available or a more compatible version of this analysis script.

                     

                    MSVCR80!MEMCPY+5AIn Apache__PID__3004__Date__09_25_2011__Time_04_16_22PM__427__Second_Chance_Except ion_C0000005.dmp the assembly instruction at msvcr80!memcpy+5a in C:\Windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d0 8cc06a442b34fc\msvcr80.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x0c5d9000 on thread 9

                    Module Information

                    Image Name:C:\Windows\winsxs\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4940_none_d0 8cc06a442b34fc\msvcr80.dll  Symbol Type: PDB
                    Base address:0x74780000  Time Stamp: Wed Sep 29 05:28:49 2010
                    Checksum:0x000a606b  Comments:
                    COM DLL:False  Company Name: Microsoft Corporation
                    ISAPIExtension:False  File Description: Microsoft® C Runtime Library
                    ISAPIFilter:False  File Version: 8.00.50727.4940
                    Managed DLL:False  Internal Name: MSVCR80.DLL
                    VB DLL:False  Legal Copyright: © Microsoft Corporation. All rights reserved.
                    Loaded Image Name: msvcr80.dll  Legal Trademarks:
                    Mapped Image Name:   Original filename: MSVCR80.DLL
                    Module name: msvcr80  Private Build:
                    Single Threaded: False  Product Name: Microsoft® Visual Studio® 2005
                    Module Size: 620.00 KBytes  Product Version: 8.00.50727.4940
                    Symbol File Name: c:\symcache\msvcr80.i386.pdb\769BC0A2E0054674A3F542BCBBD95BA81\msvcr80.i386.pdb  Special Build: &
                    • 7. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
                      bretzeli

                      Hi Michael,

                       

                      This case is with our Tier III engineer Joe Bidgood who I have asked to contact you.

                       

                      Best Wishes

                       

                      Mandeep

                       

                      -----Original Message-----

                      Sent: 11 October 2011 15:09

                      To: MB McAfee SR Update

                      Cc: Kandola, Mandeep; Kratz, Ralf

                      Subject: AW: McAfee Support Notification - SR # <3-1686673611> has been updated

                      Importance: High

                      • 8. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
                        altair.santana

                        Hi,

                         

                        I have the same issue. And I escalated to Platinum support. But I did not have updates.

                         

                        ePO 4.5 p4 manage ~40k clients

                        Windows 2008 R2 64 - 16GBRAM

                        SQL 2005 remote (dedicated)

                         

                        Log Name:      Application

                        Source:        Application Error

                        Date:          10/10/2011 15:11:14

                        Event ID:      1000

                        Task Category: (100)

                        Level:         Error

                        Keywords:      Classic

                        User:          N/A

                        Computer:      xxxxxxxxxxx

                        Description:

                        Faulting application name: Apache.exe, version: 2.2.9.0, time stamp: 0x48d60715

                        Faulting module name: MSVCR80.dll, version: 8.0.50727.6195, time stamp: 0x4dcddbf3

                        Exception code: 0xc0000005

                        Fault offset: 0x00050c1c

                        Faulting process id: 0x82e8

                        Faulting application start time: 0x01cc876c0ddc7f75

                        Faulting application path: D:\Program Files\McAfee\ePolicy Orchestrator\Apache2\bin\Apache.exe

                        Faulting module path: C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d 09154e044272b9a\MSVCR80.dll

                        Report Id: 3d4efc68-f36b-11e0-81c7-003048ce7839

                        Event Xml:

                        <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

                          <System>

                            <Provider Name="Application Error" />

                            <EventID Qualifiers="0">1000</EventID>

                            <Level>2</Level>

                            <Task>100</Task>

                            <Keywords>0x80000000000000</Keywords>

                            <TimeCreated SystemTime="2011-10-10T18:11:14.000000000Z" />

                            <EventRecordID>1478</EventRecordID>

                            <Channel>Application</Channel>

                            <Computer>CPRODAMIBS23.rede.sp</Computer>

                            <Security />

                          </System>

                          <EventData>

                            <Data>Apache.exe</Data>

                            <Data>2.2.9.0</Data>

                            <Data>48d60715</Data>

                            <Data>MSVCR80.dll</Data>

                            <Data>8.0.50727.6195</Data>

                            <Data>4dcddbf3</Data>

                            <Data>c0000005</Data>

                            <Data>00050c1c</Data>

                            <Data>82e8</Data>

                            <Data>01cc876c0ddc7f75</Data>

                            <Data>D:\Program Files\McAfee\ePolicy Orchestrator\Apache2\bin\Apache.exe</Data>

                            <Data>C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_ none_d09154e044272b9a\MSVCR80.dll</Data>

                            <Data>3d4efc68-f36b-11e0-81c7-003048ce7839</Data>

                          </EventData>

                        </Event>

                         

                        Message was edited by: altair.santana on 10/11/11 10:07:16 AM CDT

                         

                        Message was edited by: altair.santana on 10/11/11 1:47:57 PM CDT
                        • 9. Re: EPO 4.6, SQL Exp. 2005, Event 1000, Apache.exe crash, MSVCR80.dll
                          diwi

                          I have the same error (Event ID 1000 / tomcat.exe / Application error) with ePO 4.5.5 Build 1188 on Windows 2008 x64 (not R2).

                           

                          In the meantime I use a reboot server task to get ePO running again and will soon update to 4.5.6 before I upgrade to ePO 4.6.x !!!

                           

                          Best regards,

                          DiWi