Let me explain my current landscape. We have Secure Web Reporter 5.x with three log sources for 3 proxy servers (let’s call them A, B and C). Log files are received on a daily basis for each of the log sources.
- Log sources A and B are not a problem as they come in one single file per log source with complete days from 00:00:00 to 23:59:59.
- Log Source C is the problem. This log source is composed of 4 proxy servers so we get for each day 4 log files corresponding to each of the proxy servers. In addition, each log file comes always with several entries corresponding to records of the day before, more precisely from the last hour of the day before. I guess this is related to some misalignment on how log rate is done, but I am not sure.
When importing the situation is the following:
- Log sources A and B are not a problem. They get imported correctly.
- Log Source C. For this group of proxy servers I need to do a mass import for complete month of January. So my plan is to uncompress each days log file into the import directory. When I import 2011-01-01 there is no issue, logs get imported correctly. Repeating this process with 2011-01-02 I always get this “funny” message of “Skipping file because it is too old”. My assumption here is that as log files for 2011-01-02 contain the last hour for 2011-01-01 SWR fails because it understands that I am trying to re-import the same log files, although this is not correct because log entries never overlap, I don’t have repeated records with same timestamp, I am sure about that.
I have made a dozen or more log imports with its corresponding manual task to cleanup those days at database level, as far as I understand there is no way to work around this unless I manually (or automatically) move log entries from one day to another to make sure that each day only has records for they day being processed.
Any idea? Is there any other way to get this solved? Can I specify SWR to ignore that log files can contain information for previous day? What is the best method to do a mass import for a complete month?
By the way. It would be a really good and appreciated improvement to have a command line importing tool that would be a bit more verbose and more flexible. I find the current way of importing a bit unfriendly.
I have another question I could not get an answer neither in documentation or KB.
- What happens with log sources when I delete them? How are the records imported through that log source treated? Do they get deleted also? Are they assigned to some other log source?
I hope you can help me solve this as I have already spent too much time on this.