Watchlists: Domain vs web_domain ?

[Former Member]

Something else to keep in mind is that a field such as "Domain" which is an indexed string cannot have Regex used on it within correlation rules. However "web_domain" which is an indexed random string can have Regex applied, which is extremely useful for picking certain elements out of a domain such as an actual IP address being access through a proxy server.

[Former Member]

One other thing to note, as I ran in to this problem...

According to support, a Random String has a maximum of 1,000,000 entries before you are going to have problems.

We were pulling in 3rd party Threat Intel and populating Malicious Domains to the "Web_Domain" field with a Dynamic Watchlist. My original intel data filters had us at about 1.5 Million names. The Dynamic Watchlist pull would crash and then that stopped all other Dynamic Watchlists from running until we bounced the cpservice and dbserverd.

I then modified the filters on my 3rd party Threat Intel feed to get below the 1 Million supposed limitation, got it down to about 500K and it still crashed while importing.

Unfortunately we are parsing the short domain names to the "Web_Domain" field from our Web Proxies, and the full URL to the "URL" field. I believe we are using the Domain field to parse the Source Users domain, or something along those lines.

I have now disabled the import of data to the "Web_Domain" field.