Looping on mailbox URL messages with no content

[chorsley]

IntelMQ newbie here: wanted to get feedback on the right way to do this rather than submit a pull request just yet.

I had an issue with an email report containing a URL, where the URL content is 0 bytes. This caused lib/message.py:212 to raise InvalidValue, and the bot exits and restarts.

2017-05-30 23:00:01,926 - shadowserver-url-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/opt/intelmq/venv/lib/python3.5/site-packages/intelmq-1.0.0.dev7-py3.5.egg/intelmq/lib/bot.py", line 150, in start
    self.process()
  File "/opt/intelmq/venv/lib/python3.5/site-packages/intelmq-1.0.0.dev7-py3.5.egg/intelmq/bots/collectors/mail/collector_mail_url.py", line 87, in process
    self.chunk_replicate_header):
  File "/opt/intelmq/venv/lib/python3.5/site-packages/intelmq-1.0.0.dev7-py3.5.egg/intelmq/lib/splitreports.py", line 152, in generate_reports
    report.add("raw", infile.read(), force=True)
  File "/opt/intelmq/venv/lib/python3.5/site-packages/intelmq-1.0.0.dev7-py3.5.egg/intelmq/lib/message.py", line 212, in add
    raise exceptions.InvalidValue(key, value, reason=valid_value[1])
intelmq.lib.exceptions.InvalidValue: invalid value '' (<class 'str'>) for key 'raw' is_valid returned False.

Since the problem report didn't get marked as read before the bot restart, the collector_mail_url.py bot would get permanently stuck in the mailbox trying and failing to process that one report.

A quick fix was modifying this code in collector_mail_url.py:

                        self.logger.info("Report downloaded.")

                        if len(resp.content) > 0:
                            template = self.new_report()

                            for report in generate_reports(template, io.BytesIO(resp.content),
                                                           self.chunk_size,
                                                           self.chunk_replicate_header):
                                self.send_message(report)
                        else:
                            self.logger.error("Report for {} was empty".format(url))

                        # Only mark read if message relevant to this instance,
                        # so other instances watching this mailbox will still
                        # check it.
                        mailbox.mark_seen(uid)

I'd imagine there would be more situations where we have a permanent failure in a report URL, but we never mark the problem mail as read. Perhaps we need to specify other conditions between temporary failures and a permanent inability to process a report - I'm wondering if this is the right place to do it.

[ghost]

Hi,

Thanks for reporting that.
I think that we can treat these cases as "success" - in terms of tried to fetch - and thus mark the email as read, as we don't expect a different result for another run:

• empty webpages
• HTTP status codes 4xx (permanent errors)

I also think that these cases should be logged as they indicate some problem - either the configuration, on the network part or the source of the webpage. Thus warning could be appropriate - as it's an error that we can handle.
@dmth what's your opinion?

[dmth]

Sorry for my late reply.
@chorsley thank your for reporting this issue and providing a workaround.

Marking the mail as sent is a quick fix, but risks losing reports.
We've already seen a some shadowserver reports without content in one (or more) requests, but with content in another one.

So, contrary to @wagner-certat, I do expect a different result on another run.
We maybe want a retry-mechanism for this case.

Proposal:
Do not mark the mail as sent but write it's message-id / uuid (or another identifier the IMAP protocol is providing) and the timestamp of the last try to a list in the bot and retry to download it's contents later. If the content could not be fetched after a certain time, for instance six hours, log an ERROR which identifies the e-mail and maybe the url that was tried, mark the e-mail as read and give up.

[ghost]

Marking the mail as sent is a quick fix,

read, not sent

but risks losing reports.

That risk is communicated to the user by logging a warning.