Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File collector does not handle empty files correctly #1244

Closed
ghost opened this issue May 30, 2018 · 0 comments
Closed

File collector does not handle empty files correctly #1244

ghost opened this issue May 30, 2018 · 0 comments
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Milestone
1.0.5

Comments

@ghost
Copy link

ghost commented May 30, 2018

Reported by a user, the collector does not handle empty files correctly:

2018-05-29 09:48:20,659 - ***-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 145, in start
    self.process()
  File "/usr/lib/python3/dist-packages/intelmq/bots/collectors/file/collector_file.py", line 66, in process
    self.parameters.chunk_replicate_header):
  File "/usr/lib/python3/dist-packages/intelmq/lib/splitreports.py", line 125, in generate_reports
    report.add("raw", infile.read(), force=True)
  File "/usr/lib/python3/dist-packages/intelmq/lib/message.py", line 212, in add
    raise exceptions.InvalidValue(key, value, reason=valid_value[1])
intelmq.lib.exceptions.InvalidValue: invalid value '' (<class 'str'>) for key 'raw' is_valid returned False.

The file has to be removed manually.
@ghost ghost added bug Indicates an unexpected problem or unintended behavior component: bots labels May 30, 2018
@ghost ghost added this to the 1.0.5 milestone May 30, 2018
@ghost ghost closed this as completed in 0d777fe Jun 12, 2018
@ghost ghost self-assigned this Jun 14, 2018
ghost pushed a commit that referenced this issue Jun 21, 2018
REL: bugfix release 1.0.5
b2fa783 
 ### Core
- `lib/message`: `Report()` can now create a Report instance from Event instances (#1225).
- `lib/bot`:
  * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (#1237).
  * Kills oneself again after proper shutdown if the bot is XMPP collector or output (#970). Previously these two bots needed two stop commands to get actually stopped.
- `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (#1184).

 ### Bots
 #### Collectors
- `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (#988).
- `bots.collectos.file.collector_file`: handle empty files (#1244).

 #### Parsers
- Shadowserver parser:
  * SSL FREAK: Remove optional column `device_serial` and add several new ones.
  * Fixed HTTP URL parsing for multiple feeds (#1243).
- Spamhaus CERT parser:
  * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (#1254).
  * fix `extra.destination.local_port` -> `extra.source.local_port`.

 #### Experts
- `bots.experts.filter`: Pre-compile regex at bot initialization.

 ### Tests
- Ensure that the bots did process all messages (#291).

 ### Tools
- `intelmqctl`:
  * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (#1075).
  * `intelmqctl run [bot-id] mesage send` can now send report messages (#1077).
- `intelmqdump`:
  * has now command completion for bot names, actions and queue names in interacive console.
  * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (#1225).
  * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (#1256).
  * truncated messages are no longer used/saved to the file after being shown (#1255).
  * now again denies recovery of dumps if the corresponding bot is running. The check was broken (#1258).
  * now sorts the dump by the time of the dump. Previously, the list was in random order (#1020).

 ### Known issues
no known issues
chorsley pushed a commit to chorsley/intelmq that referenced this issue Jul 14, 2021
Merge tag '1.0.5' into 2021-06-14-upstream-1.0.5-merge
233d478 
1.0.5

 ### Core
- `lib/message`: `Report()` can now create a Report instance from Event instances (certtools#1225).
- `lib/bot`:
  * The first word in the log line `Processed ... messages since last logging.` is now adaptible and set to `Forwarded` in the existing filtering bots (certtools#1237).
  * Kills oneself again after proper shutdown if the bot is XMPP collector or output (certtools#970). Previously these two bots needed two stop commands to get actually stopped.
- `lib/utils`: log: set the name of the `py.warnings` logger to the bot name (certtools#1184).

 ### Bots
 #### Collectors
- `bots.collectors.mail.collector_mail_url`: handle empty downloaded reports (certtools#988).
- `bots.collectos.file.collector_file`: handle empty files (certtools#1244).

 #### Parsers
- Shadowserver parser:
  * SSL FREAK: Remove optional column `device_serial` and add several new ones.
  * Fixed HTTP URL parsing for multiple feeds (certtools#1243).
- Spamhaus CERT parser:
  * add support for `smtpauth`, `l_spamlink`, `pop`, `imap`, `rdp`, `smb`, `iotscan`, `proxyget`, `iotmicrosoftds`, `automatedtest`, `ioturl`, `iotmirai`, `iotcmd`, `iotlogin` and `iotuser` (certtools#1254).
  * fix `extra.destination.local_port` -> `extra.source.local_port`.

 #### Experts
- `bots.experts.filter`: Pre-compile regex at bot initialization.

 ### Tests
- Ensure that the bots did process all messages (certtools#291).

 ### Tools
- `intelmqctl`:
  * `intelmqctl run` has a new parameter `-l` `--loglevel` to overwrite the log level for the run (certtools#1075).
  * `intelmqctl run [bot-id] mesage send` can now send report messages (certtools#1077).
- `intelmqdump`:
  * has now command completion for bot names, actions and queue names in interacive console.
  * automatically converts messages from events to reports if the queue the message is being restored to is the source queue of a parser (certtools#1225).
  * is now capable to read messages in dumps that are dictionaries as opposed to serialized dicts as strings and does not convert them in the show command (certtools#1256).
  * truncated messages are no longer used/saved to the file after being shown (certtools#1255).
  * now again denies recovery of dumps if the corresponding bot is running. The check was broken (certtools#1258).
  * now sorts the dump by the time of the dump. Previously, the list was in random order (certtools#1020).

 ### Known issues
no known issues
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior component: bots
Projects
None yet
Milestone
1.0.5
Development

No branches or pull requests
0 participants