2 Replies Latest reply on Jun 13, 2016 2:51 PM by akerr

    Adding A Disk To Your ESM

    akerr

      Having added/changed event disk on our ESM a couple of times now, I thought I'd share with the community some tips that would have made my experiences go a lot smoother.

       

      First, the purpose of doing this in our environment was to increase retention and reporting time in our ESM.  We like to provide monthly reports for our customers which means we need 31+ days of retention.  The 500GB drive in our VM12 wasn't enough space for event data.  It would start dumping old events when it got to around 90% full, which at the time was about 300GB of data. We were holding about 25 days at the time, so I decided to add a 750GB drive just for event data.  At 90% use, we'd have 675GB of usable space giving us 56 odd days of retention which would handle expected growth for at least a little while.

       

      However, you don't get to use 90% of a secondary drive for event data. You only get about about 67.5%.  This is due to various reservations the ESM makes for growth of indexes and such.  The end result was we barely had over 40 days of retention, so I ended having to do the change again later, adding a 2TB drive.

       

      A virtual ESM will recognize a new virtual disk (you can't use NFS or ISCSI for this operation) in time.  I wasn't able to determine the process that causes this to happen.  Rebooting the ESM will also cause the new disk to be recognized.

       

      The script that actually moves the database to the new drive is /usr/local/bin/SetupVmVolume.   It logs it's activities to /var/log/SetupVmVolume.log when run.  Simply, the script is supposed to shut down cpserviced, dbserverd, and then copy the database to the new drive.  Be prepared, this can take hours in which time the ESM is essentially not running.  In my experience (while running 9.5.1MR2), there are a couple of problems with the script.  First,

      there is a typo in one of the function calls. If you can read diff output, this is the problem.  If you can't, on line 451 the script calls for checkprocName which it should be calling checkProcName.

       

      451c451

      <          $found = checkProcName( 'cpservicectl' ) || checkprocName( 'dbservicectl');

      ---

      >          $found = checkProcName( 'cpservicectl' ) || checkProcName( 'dbservicectl');

       

       

      So you'll need to correct that before you can make this change.  The other problem occurs after the database has moved and dbserver has started back up.  Lines 465 and 466 of the script:

       

            $result = systemwithlog( "/sbin/service dbserver $cmd" );

            $result = $result && systemwithlog( "/sbin/service cpservice $cmd" );

       

      The first line does indeed start dbserver back up, but it fails to start cpservice.  I unfortunately haven't been able to do enough testing to determine why dbserver doesn't return a "true" exit code signalling it's started up properly. The logs indicate it's fine, but cpservice is never started, and the script is no longer running.  In my experience, I've waited until I see dbserver is running again and then manually start cpservice.

       

      I hope this helps anyone looking to add space to their ESM.  The instructions should apply to a physical ESM as well as a virtual one.  Note that on the virtual ESMs, the option to move event data is only available on the ESM12 and above.  I have brought these issues up to Intel, but haven't heard back on any action being taken.