BatchSMTP crashes on send

[slebrun1]

Debian 12 with native packages, intelmq 3.2.1, envelope 2.0.2

Sending email with the SMTP output bot works as expected. Trying to use the SMTPBatch output bot works up until one actually tries to send, throwing a python error.

intelmq@x:~$ intelmq.bots.outputs.smtp_batch.output SMTPBatch-Output --cli
SMTPBatch-Output: SMTPBatchOutputBot initialized with id SMTPBatch-Output and intelmq 3.2.1 and python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] as process 359502. Standalone mode: False.
SMTPBatch-Output: Bot is starting.
SMTPBatch-Output: Loading source pipeline and queue 'SMTPBatch-Output-queue'.
SMTPBatch-Output: Connected to source queue.
SMTPBatch-Output: No destination queues to load.
SMTPBatch-Output: Bot initialization completed.
Preparing mail queue...
To: slebrun@X; Subject: IntelMQ warning (2023-12-12) (slebrun@X) Events: 378, Size: 95287

Number of mails in the queue: 1
No GPG

What you would like to do?
* enter to send first mail to tester's address None.
* any mail from above to be delivered to tester's address
* 'debug' to send all the e-mails to tester's address
* 'all' for sending all the e-mails
* 'clear' for clearing the queue
* 'x' to cancel
? all
Traceback (most recent call last):
  File "/usr/bin/intelmq.bots.outputs.smtp_batch.output", line 33, in <module>
    sys.exit(load_entry_point('intelmq==3.2.1', 'console_scripts', 'intelmq.bots.outputs.smtp_batch.output')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp_batch/output.py", line 104, in run
    instance.cli_run()
  File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp_batch/output.py", line 171, in cli_run
    if self.build_mail(mail, send=True):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp_batch/output.py", line 342, in build_mail
    .subject(subject)
     ^^^^^^^
AttributeError: 'list' object has no attribute 'subject'
SMTPBatch-Output: Bot stopped.
SMTPBatch-Output: Bot stopped.

[sebix]

@e3rd can you help here?

[e3rd]

Give us your config file, or only its part concerning the bot, @slebrun1 , thanks!

[e3rd]

It seems I found it. Unless the PR is merged, the fix for you is to set the bcc attribute to a mere [].

(The problem was, when the default bcc=None appears, the envelope library tries to read the bcc e-mail header, instead of setting it to the empty recipients list.)

[slebrun1]

Unfortunately, that doesn't seem to have fixed it for me, but the error does look a bit different.

My original config simply didn't have a 'bcc' line (I've noticed that bots sometimes don't like having unused attributes,) but adding 'bcc: []' didn't made any difference, nor did taking it back out. I also tried with and without a custom 'subject' attribute, same issue.

Bot config:

SMTPBatch-Output:
  bot_id: SMTPBatch-Output
  description: Missing description
  enabled: true
  group: Output
  module: intelmq.bots.outputs.smtp_batch.output
  name: SMTPBatch
  parameters:
    allowed_fieldnames: [time.source, source.ip, classification.taxonomy, classification.type,
      time.observation, source.geolocation.cc, source.asn, event_description.text,
      malware.name, feed.name, feed.url, raw]
    attachment_name: intelmq_%Y-%m-%d
    bcc: []
    cli: false
    destination_queues: {}
    email_from: [email protected]
    fieldnames_translation: {classification.taxonomy: class, classification.type: type,
      event_description.text: description, feed.name: feed_name, feed.url: feed_url,
      malware.name: malware, raw: raw, source.asn: asn, source.geolocation.cc: country_code,
      source.ip: ip, time.observation: time_delivered, time.source: time_detected}
    ignore_older_than_days: 4
    mail_template: /var/lib/intelmq/bots/SMTPBatch/default.txt
    redis_cache_db: 15
    redis_cache_host: localhost
    redis_cache_port: 6379
    redis_cache_ttl: 1728000
    send: false
    smtp_server: {host: xxx.xxx.xxx.xxx, password: bigolepassword, port: 25, subject: Lakeland
        IntelMQ Alerts %Y-%m-%d, user: [email protected]}
    subject: IntelMQ Batch Events List
  run_mode: continuous

Output when run against that config:

intelmq@lln-noc-intelmq:/var/lib/intelmq/bots/SMTPBatch$ intelmq.bots.outputs.smtp_batch.output SMTPBatch-Output --cli
SMTPBatch-Output: SMTPBatchOutputBot initialized with id SMTPBatch-Output and intelmq 3.2.1 and python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] as process 370986. Standalone mode: False.
SMTPBatch-Output: Bot is starting.
SMTPBatch-Output: Loading source pipeline and queue 'SMTPBatch-Output-queue'.
SMTPBatch-Output: Connected to source queue.
SMTPBatch-Output: No destination queues to load.
SMTPBatch-Output: Bot initialization completed.
Preparing mail queue...
To: [email protected]; Subject: IntelMQ warning (2023-12-13) ([email protected]) Events: 455, Size: 112871

Number of mails in the queue: 1
No GPG

What you would like to do?
* enter to send first mail to tester's address None.
* any mail from above to be delivered to tester's address
* 'debug' to send all the e-mails to tester's address
* 'all' for sending all the e-mails
* 'clear' for clearing the queue
* 'x' to cancel
? all
Traceback (most recent call last):
  File "/usr/bin/intelmq.bots.outputs.smtp_batch.output", line 33, in <module>
    sys.exit(load_entry_point('intelmq==3.2.1', 'console_scripts', 'intelmq.bots.outputs.smtp_batch.output')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp_batch/output.py", line 104, in run
    instance.cli_run()
  File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp_batch/output.py", line 171, in cli_run
    if self.build_mail(mail, send=True):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/intelmq/bots/outputs/smtp_batch/output.py", line 343, in build_mail
    .smtp(self.smtp_server)
     ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/envelope/envelope.py", line 743, in smtp
    self._smtp = SMTPHandler(**host)
                 ^^^^^^^^^^^^^^^^^^^
TypeError: SMTPHandler.__init__() got an unexpected keyword argument 'subject'
SMTPBatch-Output: Bot stopped.
SMTPBatch-Output: Bot stopped.

[e3rd]

That seems a completely different error. Note that subject is an attribute of the bot, not one of the smtp_server dict keys. The error you've posted complaints about this unexpected subject. Try an original config, just with the bcc: [] line added and give me an error output (because it works at my local station).

(...)
  parameters:
    allowed_fieldnames: [time.source, source.ip, classification.taxonomy, classification.type,
      time.observation, source.geolocation.cc, source.asn, event_description.text,
      malware.name, feed.name, feed.url, raw]
    attachment_name: intelmq_%Y-%m-%d
    bcc: []
    email_from: [email protected]
    fieldnames_translation: {classification.taxonomy: class, classification.type: type,
      event_description.text: description, feed.name: feed_name, feed.url: feed_url,
      malware.name: malware, raw: raw, source.asn: asn, source.geolocation.cc: country_code,
      source.ip: ip, time.observation: time_delivered, time.source: time_detected}
    ignore_older_than_days: 4
    mail_template: /var/lib/intelmq/bots/SMTPBatch/default.txt
    redis_cache_db: 15
    redis_cache_host: localhost
    redis_cache_port: 6379
    redis_cache_ttl: 1728000
    smtp_server: {host: xxx.xxx.xxx.xxx, password: bigolepassword, port: 25, user: [email protected]}
    subject: Lakeland IntelMQ Alerts %Y-%m-%d
  run_mode: continuous

(note I've removed some private parameters like send and cli)

[slebrun1]

Hah! It was indeed the 'subject' attribute in the smtp_server parameter that was the issue:

smtp_server: {host: xx.xx.xx.xx, password: bigolepassword, port: 25, subject: Lakeland
    IntelMQ Alerts %Y-%m-%d, user: [email protected]}

causes the error,

smtp_server: {host: xx.xx.xx.xx, password: bigolepassword, port: 25, user: [email protected]}

works as expected.

The documentation points to the CZ-NIC envelope documentation, and I must have gotten it into my head to include a subject there as part of the info being passed to the SMTP server.

Thanks for your help with this!