3 Replies Latest reply on Aug 6, 2012 3:04 PM by mtuma

    Why should the Firewall suddenly start to ignore rules?

    PhilM

      I thought I'd fire this one at the community before pursuing a more formal route because even after 18 months exposure I am still encountering scenarios where the new 'application signature' functionality doesn't always seem to work - and what's more frustrating is there is no apparent reason why.

       

      A little over two weeks ago I replaced a customer's old 210 appliance (running 7.0.1.02) with a brand new S1104 appliance running 8.2.1.

       

      Because I had received the hardware in advance, had remote access to the old Firewall and knew that the rule set wasn't complex I decided to build the v8 configuration from scratch.

       

      One of the primary services provided by my customer is a public-facing web shop. It consists of a server located on their DMZ with a number of back-end services running on the internal LAN.

       

      There is a single rule, consisting of a service group (or Application Group to use v8 terminology) which allows the IP address of the DMZ server to communicate with these back-end services. One of the applications contained within this application group is the "Kerberos" application. When the Firewall was first deployed, all key services were tested (including the web shop) and were found to be working perfectly. Over the course of that weekend, they continued to work. Then, without explanation, mid-way through the Monday afternoon, the web shop stopped responding. The customer naturally phoned me to see if there was anything on the Firewall which could be contributing to this behaviour. I looked at the audit viewer and noticed that it seemed to be taking issue with connections originating from the web shop IP address for TCP port 88. I checked this and found that it belonged to the Kerberos application signature. Because the web shop had caused the customer some headaches long before the Firewalls were changed he went off to re-boot it.

       

      In the meantime, I created a new user-defined application entry. Calling it "Kerberos-Auth" I configured it to use TCP and UDP port 88. I then created a new rule, placing it immedately before the one which contained the Kerberos application service and used the new user-defined service. As soon as the DMZ server finished re-booting everything came back to life. I tested the web shop and confirmed this to my (naturally delighted) customer. I then disabled the rule containing my custom service and tested the web shop again - everything was still working. Over the course of the next few days I continued to test the service and it appeared to be running solidly.

       

      Fast forward 1 week and, following a power outage, requiring all the servers to be re-booted the web shop stopped working again. Remembering how I had diagnosed the original problem, the customer took it upon himself to check the audit before giving me a call. When he called me he said that every time he initiated a test connection from his DMZ server he could see 'attack' records in the Firewall audit for TCP port 88. Here is one such example:-

       

      2012-08-06 13:58:45 +0100 f_http_proxy a_aclquery t_attack p_major

      pid: 1210 logid: 0 cmd: 'httpp' hostname: xxxx.xxx-xx.com

      category: policy_violation event: ACL deny attackip: xx.x.x.xx

      attackzone: DMZ application: <Unknown TCP> user_name: user@domain.local

      auth_method: Passive (MLC) srcip: xx.x.x.xx srcport: 49253 srczone: DMZ

      protocol: 6 dstip: xxx.xx.x.x dstport: 88 dstzone: internal

      rule_name: Deny All cache_hit: 1 ssl_name: Exempt All

      reason: Traffic denied by policy.

       

      It clearly shows connections on tcp/88, but doesn't explain why it is falling through the entire rule set and hitting "Deny All", because there's a rule in place a lot higher up the rule set which should clearly allow the traffic to pass..

       

      I suggested that he re-enable the rule I had created the previous week (with the separate, user-defined, application on tcp/88) and give it another try. Sucess! the web shop came back to life and the audit showed 'nettraffic' entires for tcp/88 instead of attack records:-

       

      2012-08-06 14:42:07 +0100 f_http_proxy a_libproxycommon t_nettraffic p_major

      pid: 1210 logid: 0 cmd: 'httpp' hostname: xxxx.xxx-xx.com

      event: session end application: Kerberos-Auth netsessid: bd19a501fc9af

      user_name: user@domain.local auth_method: Passive (MLC) srcip: xx.x.x.xx

      srcport: 49704 srczone: DMZ protocol: 6 dstip: xxx.xx.x.x dstport: 88

      dstzone: internal bytes_written_to_client: 186

      bytes_written_to_server: 234 rule_name: AX2009EP-KerberosAuth

      cache_hit: 0 start_time: 2012-08-06 14:42:07 +0100

       

      The "application" entry listed is clearly my user-defined "Kerberos-Auth" application.

       

      Then, just to make my confusion absolute there is then the following audit record less than 10 minutes later:-

       

      2012-08-06 14:51:47 +0100 f_http_proxy a_libproxycommon t_nettraffic p_major

      pid: 1210 logid: 0 cmd: 'httpp' hostname: xxxx.xxx-xx.com

      event: session end application: Kerberos app_risk: medium

      app_categories: auth-services netsessid: b9f2d501fcbf3

      user_name: user@domain.local auth_method: Passive (MLC)

      srcip: xx.x.x.xx srcport: 49829 srczone: DMZ protocol: 6

      dstip: xxx.xx.x.x dstport: 88 dstzone: internal

      bytes_written_to_client: 1368 bytes_written_to_server: 1379

      rule_name: AX2009EP-KerberosAuth cache_hit: 1

      start_time: 2012-08-06 14:51:47 +0100

       

      This references the same rule name, but now appears to be using the signature-based "Kerberos" application, rather than user-defined one it was using a few minutes beforehand.

       

      So I am left with 4 unanswered questions:-

       

      • Why does the service appear to work initially with he original rule (containing the "Kerberos" appliacation) in place?
      • Why does it stop recognising this traffic, resulting in it hitting "Deny All"?
      • Why it is then necessary to enable this rule using a user-defined service on tcp/88 to make it work again?
      • Why, after enabling this temporary rule does the Firewall then continue use the temporary rule, but change the reported application from the user-defined "Kerberos-Auth" application to the original "Kerberos" one?

       

      If anyone can make sense of this and explain to me why it is happening will have my eternal gratitude!

       

      -Phil.

       

      Message was edited by: PhilM on 06/08/12 16:01:24 IST

       

      Message was edited by: PhilM on 06/08/12 16:02:25 IST
        • 1. Re: Why should the Firewall suddenly start to ignore rules?

          Hello Phil,

           

          I think I may have an explanation of why the traffic was denied. If you look at the audit message, you will see that the application is <Unknown>. What this means is that the firewall was unable to identify the application, which is why the traffic was denied. In version 8, to identify some applications, we actually have to allow the connection before identifying the application. I have actually seen situations where the server was not allowing the connection (sending resets), which caused the firewall to not be able to identify the application, which then caused the firewall to audit that the traffic was denied. In reality, the firewall was not denying the traffic, the server was resetting the connection early.

           

          So, in summary, tcpdumps while this problem is occuring is going to be crucial. That will tell us whether or not the client or server are closign the connection before the firewall can identify the application.

           

          -Matt

          1 of 1 people found this helpful
          • 2. Re: Why should the Firewall suddenly start to ignore rules?
            PhilM

            Thanks Matt,

             

            Would this possibly explain why my user-defined application entry works? - because the Firewall doesn't have to concern itself about *what* the application is, simply that it is 'something' trying to pass on TCP port 88.

             

            The customer is understandibly eager not to 'break' it again, now I have implemented something that works for him, but I will speak to him to see if he will allow the fix to be disabled so that we can attempt these tcpdumps, should the service stop working again.

             

            -Phil.

            • 3. Re: Why should the Firewall suddenly start to ignore rules?

              Hello Phil,

               

              Would this possibly explain why my user-defined application entry works? - because the Firewall doesn't have to concern itself about *what* the application is, simply that it is 'something' trying to pass on TCP port 88.


              Correct. When we use a custom app on port 88, the firewall does not need to identify the application which can alleviate this problem.

               

               

              Regards,

               

              Matt