Finally the stars and my projects have aligned and I'm able to get something posted! Standard apologies about the long delay; not only is this my busy season, but 10.0 was released introducing the new HTML5 UI, Enterprise Log Search (ELS) appliance and introduced a number of API changes, some better documented than others. Now that I've had some time to work with the new release, this post is going to focus on some of the key differences in the 10.0 API. For contrast, I'll be updating one of my 9.x scripts posted at Github for 10.x and explain the changes I had to make it get it working again. If you just want a direct link to the script, it can be found at the bottom of this post.

 

First thing is first, the API documentation decided it was time for a nicer URL and now is available at:

 

https://<ESM-IP>/rs/esm/help

 

(vs. the 9.x location at https://<ESM-IP>/rs/esm/help/commands)

 

There are new methods for alarms, blacklists, cases, views, ELS queries, notifications, many new "get info" methods and some fun surprises in how returned results are nested. There have been a lot of great improvements made across the board to the API in 10.0...and some other things got changed too. Let's start with some breaking changes. As usual, all my code is Python3 with Requests. I cover how to create the runtime environment for these scripts in a nearby blog post.

 

Let's get right to it:

 

1. New Token-based login

 

This is a good change to make that makes the API more 'RESTFUL'. The API now requires that a token be embedded in the session. Here's is some bare bones code that shows the necessary steps to authenticate and then prints the current ESM time at the end as verification.

 

import base64
import json
import re
import requests

esmhost = "<ESM-IP>"
user = "<ESM-USER>"
passwd = "<ESM-PASS>"
url = 'https://{}/rs/esm/'.format(esmhost)
login_url = '{}{}'.format(url, 'login')
b64_user = base64.b64encode(user.encode('utf-8')).decode()
b64_passwd = base64.b64encode(passwd.encode('utf-8')).decode()
params = {"username": b64_user, 
          "password": b64_passwd, 
          "locale": "en_US", 
          "os" : "Win32"}
params_json = json.dumps(params)
v10_login_headers = {'Content-Type': 'application/json'}
login_response = requests.post(login_url, 
                               params_json, 
                               headers=v10_login_headers, 
                               verify=False)
cookie = login_response.headers.get('Set-Cookie')
jwttoken = re.search('(^[A-Za-z0-9-_=]+\.[A-Za-z0-9-_=]+\.?[A-Za-z0-9-_.+/=]*)', cookie).group(1)
xsrf_token = login_response.headers.get('Xsrf-Token')
session_headers = {'Cookie' : jwttoken, 
                   'X-Xsrf-Token' : xsrf_token, 
                   'Content-Type': 'application/json'}
print(requests.post(url + 'essmgtGetESSTime', headers=session_headers, verify=False).text)

 

Here's the same code a little better organized in a class with some error checking and validation. This is how it looks in the demo script and the credentials come from the config file.

 

import base64
import json
import re
import requests

class NitroESM(object):
    def __init__(self, esmhost, user, passwd):
        """ Init instance attributes """
        self.esmhost = esmhost
        self.user = user
        self.passwd = passwd
        self._build_login_urls()
        self._encode_login()
        self._build_params()

    def _build_login_urls(self):
        """ Concatenate URLs """
        self.url = 'https://{}/rs/esm/'.format(self.esmhost)
        self.login_url = '{}{}'.format(self.url, 'login')

    def _encode_login(self):
        self.b64_user = base64.b64encode(self.user.encode('utf-8')).decode()
        self.b64_passwd = base64.b64encode(self.passwd.encode('utf-8')).decode()
    
    def _build_params(self):
            self.params = {"username": self.b64_user, 
                           "password": self.b64_passwd, 
                           "locale": "en_US", 
                           "os" : "Win32"}
            try:
                self.params_json = json.dumps(self.params)
            except TypeError as e:
                logging.error("Invalid parameters: %s", self.params)
                sys.exit(1)
        
    def login_v10(self):
        try:
            self.v10_login_headers = {'Content-Type': 'application/json'}
            self.login_response = requests.post(self.login_url, 
                                                 self.params_json, 
                                                 headers=self.v10_login_headers, 
                                                 verify=False)
            self.cookie = self.login_response.headers.get('Set-Cookie')
            try:
                self.jwttoken = re.search('(^[A-Za-z0-9-_=]+\.[A-Za-z0-9-_=]+\.?[A-Za-z0-9-_.+/=]*)', 
                                           self.cookie).group(1)
            except TypeError as e:
                logging.error("Failed login: %s: %s", self.user, self.passwd)
                sys.exit(1)
            self.xsrf_token = self.login_response.headers.get('Xsrf-Token')
            self.session_headers = {'Cookie' : self.jwttoken, 
                                    'X-Xsrf-Token' : self.xsrf_token, 
                                    'Content-Type': 'application/json'}
        except KeyError:
            logging.error("Failed login: %s: %s", self.user, self.passwd)
            sys.exit(1)

 

And for contrast, here is the 9.x version of the login:

class NitroESM(object):
    def __init__(self, esmhost, user, passwd):
        """ Init instance attributes """
        self.esmhost = esmhost
        self.user = user
        self.passwd = passwd
        self._encode_creds()
        self._build_login_urls()

    def _encode_creds(self):
        """ Encode authstring. Returns base64 string """
        self.creds = '{}:{}'.format(self.user, self.passwd)
        self.b64_creds = base64.b64encode(self.creds.encode('utf-8'))

    def _build_login_urls(self):
        """ Concatenate URLs """
        self.url = 'https://{}/rs/esm/'.format(self.esmhost)
        self.login_url = '{}{}'.format(self.url, 'login')
        self.login_headers = {'Authorization':'Basic ' + self.b64_creds.decode('utf-8'),
                              'Content-Type': 'application/json'}

    def login(self):
        """ Authenticate to ESM and establish session """
        self.login_response = requests.post(self.login_url,
                                            headers=self.login_headers,
                                            verify=False)
        self.session_id = self.login_response.headers['location']
        self.session_headers = {'Authorization':'Session ' + self.session_id,
                                'Content-Type': 'application/json'}

 

Now that we are able to log in again, the next error in the script revolves around the results.

 

2. Data return format change

 

In 9.6, the results from all API’s were wrapped in a return JSON object, thus login in 9.6 returns:

 

{
   return: {
     "adminType": 0,
     "autoGetPacket": false,
    "autoRefresh": false,
    "tzId": 26,
     "userAlias": "",
     "userId": 1,
     "userName": "NGCP"
   }
}

 

In 10.0 it has been simplified to:

{
 "adminType": 0,
  "autoGetPacket": false,
 "autoRefresh": false,
 "tzId": 26,
 "userAlias": "",
 "userId": 1,
 "userName": "NGCP"
}

 

One less level of nesting to deal with could break code if you weren't expecting it, so, of course my code broke.

 

When I first wrote this it seemed like a good idea to have this generic function that would just go and get whatever data I wanted without caring much about where it was nested. I did this because I was never quite sure how deep a bit of data was that I might be looking for and I didn't want to catch an error by making an assumption. I continued to think this was a good idea and I just needed to make my function more resilient so that it could handle anything so I added a few lines so that it still worked regardless of whether it was nested in a list or a dict, so I did this, and it worked, but then it dawned on me that due to this change, function wasn't required at all.

 

In fact, where In had earlier had "for ds in search_function" could be changed to "for ds in result" and it loop through the datasources directly. This is cool, now that function can be removed completely and the code is that much more simplified.

 

3. Simplification of queries with "IDs"

 

Another change with the results returned is in 9.x that any method that included an "ID" in the payload would be nested along with an extra key:

 

{
   “resultID”: {
      “id” : 0987654321
   }
}

 

In 10.x this has been simplified to:

{
   “resultID”: 0987654321
}

 

Most of these can be found and fixed with a search and replace.

 

Those are the major changes to know about when moving from 9.x to 10.x.

 

We have come across a few bugs as well. One in particular causes an error similar to:

"Invalid field 'FirstTime' in select state" as a response to a query.

 

Currently the fix is to restart Tomcat on the ESM (/etc/rc.d/init.d/tomcat restart) and it is under investigation. A few documentation errors have been found and fixed and will roll out in the next release also.

 

In the meantime, feel free to send me any questions about specific API nuances or query syntax.

 

As for the updated script, I posted a 10.x (labeled 2.0) version here locally and on Github. This only includes the basic changes to work with a 10.x ESM but I have a few ideas for a future update. Plus it's hard to go back over older code and not want to update it with the new things you've learned since you last looked at it.