Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meaning of "Processed .. messages" in filtering bots #1237

Closed
ghost opened this issue May 11, 2018 · 0 comments
Closed

Meaning of "Processed .. messages" in filtering bots #1237

ghost opened this issue May 11, 2018 · 0 comments
Labels
component: bots component: core usability
Milestone
1.0.5

Comments

@ghost
Copy link

ghost commented May 11, 2018 •
edited by ghost
Loading

For common bots, the meaning of the log messages "Processed .. messages" is clear. For filtering bots which can send and drop messages, it is not.

In these cases we can use the word "Forwarded" instead of "Processed".

Another idea: An additional counter for dropped messages (-> v 1.1.0) #1238
@ghost ghost added this to the 1.0.5 milestone May 11, 2018
@ghost ghost mentioned this issue May 11, 2018
Open
@ghost ghost closed this as completed in c95d023 Jun 12, 2018
@ghost ghost self-assigned this Jun 14, 2018
ghost pushed a commit that referenced this issue Jun 21, 2018
REL: bugfix release 1.0.5
b2fa783 
 ### Core
- `lib/message`: `Report()` can now create a Report instance from Event instances (#1225).
- `lib/bot`:
  * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237).
  * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped.
- `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184).

 ### Bots
 #### Collectors
- `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988).
- `bots.collectos.file.collector_file`: handle empty files (#1244).

 #### Parsers
- Shadowserver parser:
  * SSL FREAK: Remove optional column `device_serial` and add several new ones.
  * Fixed HTTP URL parsing for multiple feeds (#1243).
- Spamhaus CERT parser:
  * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254).
  * fix `extra.destination.local_port` -> `extra.source.local_port`.

 #### Experts
- `bots.experts.filter`: Pre-compile regex at bot initialization.

 ### Tests
- Ensure that the bots did process all messages (#291).

 ### Tools
- `intelmqctl`:
  * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075).
  * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077).
- `intelmqdump`:
  * has now command completion for bot names, actions and queue names in interacive console.
  * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225).
  * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256).
  * truncated messages are no longer used/saved to the file after being shown (#1255).
  * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258).
  * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020).

 ### Known issues
no known issues
chorsley pushed a commit to chorsley/intelmq that referenced this issue Jul 14, 2021
Merge tag '1.0.5' into 2021-06-14-upstream-1.0.5-merge
233d478 
1.0.5

 ### Core
- `lib/message`: `Report()` can now create a Report instance from Event instances (certtools#1225).
- `lib/bot`:
  * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (certtools#1237).
  * Kills oneself again after proper shutdown if the bot is XMPP collector or output (certtools#970). Previously these two bots needed two stop commands to get actually stopped.
- `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (certtools#1184).

 ### Bots
 #### Collectors
- `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (certtools#988).
- `bots.collectos.file.collector_file`: handle empty files (certtools#1244).

 #### Parsers
- Shadowserver parser:
  * SSL FREAK: Remove optional column `device_serial` and add several new ones.
  * Fixed HTTP URL parsing for multiple feeds (certtools#1243).
- Spamhaus CERT parser:
  * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (certtools#1254).
  * fix `extra.destination.local_port` -> `extra.source.local_port`.

 #### Experts
- `bots.experts.filter`: Pre-compile regex at bot initialization.

 ### Tests
- Ensure that the bots did process all messages (certtools#291).

 ### Tools
- `intelmqctl`:
  * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (certtools#1075).
  * `intelmqctl run [bot-id] mesage send` can now send report messages (certtools#1077).
- `intelmqdump`:
  * has now command completion for bot names, actions and queue names in interacive console.
  * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (certtools#1225).
  * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (certtools#1256).
  * truncated messages are no longer used/saved to the file after being shown (certtools#1255).
  * now again denies recovery of dumps if the corresponding bot is running. The check was broken (certtools#1258).
  * now sorts the dump by the time of the dump. Previously, the list was in random order (certtools#1020).

 ### Known issues
no known issues
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: bots component: core usability
Projects
None yet
Milestone
1.0.5
Development

No branches or pull requests
0 participants