Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bulk.pl daemon crashes if it failed to load private key #1110

Closed
Salo15 opened this issue Feb 2, 2021 · 16 comments · Fixed by #1112
Closed

bulk.pl daemon crashes if it failed to load private key #1110

Salo15 opened this issue Feb 2, 2021 · 16 comments · Fixed by #1112
Labels
bug
Milestone
6.2.62

Comments

@Salo15
Copy link

Salo15 commented Feb 2, 2021

We recently had the problem that a moderation email could not be sent to the owners of a list for which there is a certificate and through which encrypted emails can be sent.
The error was that the correct value for the key_passwd parameter was not set in the Sympa config and the error message "failed to load the private key" appeared in the Sympa log.
This was my error and caused the email to the list moderators to get stuck in the queue /var/spool/sympa/bulk/msg/.

The problem with this situation, however, was that during the time Sympa kept trying to send this email, the emails to subscribers of all the other mailing lists were not being sent.
So the mailing list server was accepting and processing emails for the lists, but the emails to the subscribers of the list were sent only after we detected and fixed the above problem.

Do you have any idea why this one email blocked the sending of emails to the subscribers of all other lists?
It looks like this one email blocked the entire bulk spool.
Of course, we would like to avoid that in a similar case in the future the outgoing mail dispatch could be blocked again.

We use Sympa version 6.2.16 and Sympa is installed as Debian Package.

Kind regards,
Sabine
@ikedas
Copy link
Member

ikedas commented Feb 2, 2021 •
edited
Loading

Hi @Salo15 ,
Could you please show us Sympa's log from when the error message "failed to load the private key" appeared to when you knew that an email to subscribers of any other mailing list had not been sent?

@Salo15
Copy link
Author

Salo15 commented Feb 3, 2021 via email

@racke
Copy link
Contributor

racke commented Feb 3, 2021

This is serious issue that I'm encountering once in a while on a 6.2.22 install. Once Sympa dies during processing, it will not retire the culprit to bad directory. The consequences is that it mindlessly tries the same message again and again and again, which means no emails are processed any more.

Log file excerpt:


var/log/sympa.log.5.gz:Jan 29 22:24:04 post sympa_msg[27464]: err main::#243 > Sympa::Spindle::spin#92 > Sympa::Spindle::DoCommand::_twist#120 > Sympa::Spindle::spin#75 > Sympa::Request::Message::next#42 > Sympa::Request::Message::_load#104 > Sympa::Message::get_plain_body#2176 > MIME::Charset::new#433 > MIME::Charset::_find_encoder#467 > (eval)#1 > (eval)#1 > MIME::Charset::BEGIN#1 DIED: Can't locate Encode/EUCJPASCII.pm in @INC (you may need to install the Encode::EUCJPASCII module) (@INC contains: /usr/share/sympa/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at (eval 408) line 1.
/var/log/sympa.log.5.gz:Jan 29 22:24:21 post sympa_msg[27527]: err main::#243 > Sympa::Spindle::spin#92 > Sympa::Spindle::DoCommand::_twist#120 > Sympa::Spindle::spin#75 > Sympa::Request::Message::next#42 > Sympa::Request::Message::_load#104 > Sympa::Message::get_plain_body#2176 > MIME::Charset::new#433 > MIME::Charset::_find_encoder#467 > (eval)#1 > (eval)#1 > MIME::Charset::BEGIN#1 DIED: Can't locate Encode/EUCJPASCII.pm in @INC (you may need to install the Encode::EUCJPASCII module) (@INC contains: /usr/share/sympa/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at (eval 408) line 1.
/var/log/sympa.log.5.gz:Jan 29 22:24:39 post sympa_msg[27586]: err main::#243 > Sympa::Spindle::spin#92 > Sympa::Spindle::DoCommand::_twist#120 > Sympa::Spindle::spin#75 > Sympa::Request::Message::next#42 > Sympa::Request::Message::_load#104 > Sympa::Message::get_plain_body#2176 > MIME::Charset::new#433 > MIME::Charset::_find_encoder#467 > (eval)#1 > (eval)#1 > MIME::Charset::BEGIN#1 DIED: Can't locate Encode/EUCJPASCII.pm in @INC (you may need to install the Encode::EUCJPASCII module) (@INC contains: /usr/share/sympa/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at (eval 408) line 1.
/var/log/sympa.log.5.gz:Jan 29 22:24:57 post sympa_msg[27646]: err main::#243 > Sympa::Spindle::spin#92 > Sympa::Spindle::DoCommand::_twist#120 > Sympa::Spindle::spin#75 > Sympa::Request::Message::next#42 > Sympa::Request::Message::_load#104 > Sympa::Message::get_plain_body#2176 > MIME::Charset::new#433 > MIME::Charset::_find_encoder#467 > (eval)#1 > (eval)#1 > MIME::Charset::BEGIN#1 DIED: Can't locate Encode/EUCJPASCII.pm in @INC (you may need to install the Encode::EUCJPASCII module) (@INC contains: /usr/share/sympa/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at (eval 408) line 1.

So regardless of the reason of the crash, the bad email needs to be removed from the queue without any exceptions.

@racke racke added the bug label Feb 3, 2021
@ikedas
Copy link
Member

ikedas commented Feb 3, 2021 •
edited
Loading

I got it! It‘s duplicate of #8 .

Dependent module MIME-Charset (libmime-charset-perl on Debian) has to be updated. It hss been fixed on 1.011.3 (cf.Changelog.).

@ikedas
Copy link
Member

ikedas commented Feb 3, 2021

@Salo15 , can you find the same log line as that racke quoted?

ikedas added a commit to ikedas/sympa that referenced this issue Feb 4, 2021
@ikedas
bulk.pl daemon crashes if it failed to load private key (sympa-commun…
80d4d29 
…ity#1110)

There is also a case with public key.
@ikedas ikedas mentioned this issue Feb 4, 2021
Merged
@ikedas
Copy link
Member

ikedas commented Feb 4, 2021 •
edited
Loading

@Salo15 , I found the way to fix.

If possible, could you please apply this patch on your Sympa 6.2.16, and check if delivery will no longer be stopped?

@ikedas ikedas added this to the 6.2.62 milestone Feb 4, 2021
@racke
Copy link
Contributor

racke commented Feb 4, 2021

Thanks for the quick fix @ikedas, but it solves the problem only for this specific case. In my opinion any crashes of the bulk daemon should be intercepted.

@ikedas
Copy link
Member

ikedas commented Feb 4, 2021

Thanks for the quick fix @ikedas, but it solves the problem only for this specific case. In my opinion any crashes of the bulk daemon should be intercepted.

Do you know any other reason by which bulk daemon will crash?

@racke
Copy link
Contributor

racke commented Feb 4, 2021

I don't but as Sympa is certainly not bug free it will happen again eventually. Why is it so complicated to move away such emails? It is not likely to work the next time and it brings the system to a grinding halt.

@ikedas
Copy link
Member

ikedas commented Feb 4, 2021

I don't but as Sympa is certainly not bug free it will happen again eventually. Why is it so complicated to move away such emails? It is not likely to work the next time and it brings the system to a grinding halt.

Imagine that we could let the process investigate why the process crashed. How about crashing process that is investigating crash? You ask the endless question.

Instead, when a process crashes, the traceback is recorded on the log, and the human can investigate why the process crashed. That's why I usually ask people "Please show the log from when ... to when ...".

@racke
Copy link
Contributor

racke commented Feb 4, 2021

It isn't an endless question. The expectation of a daemon/service on Linux is that it never crashes and definitely not on user input. So the bulk daemon is currently inferior to other daemon software like nginx, postfix etc. I suggest to move this topic to general discussion / separate issue though.

@ikedas
Copy link
Member

ikedas commented Feb 4, 2021 •
edited
Loading

It's off-topic and I'll stop conversasion by this response:

Postfix, Linux kernel and so on will also crash by unexpected errors. Why they look never crashing is that many people have been eliminating the bugs they could expect. OTOH Sympa has many bugs not yet eliminated. It's simply a matter of degree.

In this time I found that Sympa should not terminate if the methods of Crypt::SMIME fails. By this, Sympa is one step closer to the software that do never crash --- although it is not possible that Sympa will be completely so.

@Salo15
Copy link
Author

Salo15 commented Feb 4, 2021 via email

@ikedas
Copy link
Member

ikedas commented Feb 4, 2021

@Salo15 , thank you for follow up.

But I'd like to ask one thing: As I'd like to confirm that a process has crashed, could you please show us the log line reporting the problem? (confidential information like host name may be masked.) If it is, that is the line including a phrase "DIED:".

@Salo15
Copy link
Author

Salo15 commented Feb 4, 2021 via email

@ikedas
Copy link
Member

ikedas commented Feb 5, 2021

@Salo15 , thanks for information! The information are exactly what I expected, and it confirmed the fix I made was correct.

When the fix will be merged, this issue will be closed. If you would realize something wrong, feel free to request reopen of this issue.

ikedas added a commit that referenced this issue Feb 5, 2021
@ikedas
Merge pull request #1112 from ikedas/issue-1110 by ikedas
5183d92 
bulk.pl daemon crashes if it failed to load private key (#1110)
@ikedas ikedas changed the title Moderation mail that could not be sent blocked all outgoing mail bulk.pl daemon crashes if it failed to load private key Feb 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
6.2.62
Development

Successfully merging a pull request may close this issue.

bulk.pl daemon crashes if it failed to load private key (#1110) ikedas/sympa
3 participants
@racke @ikedas @Salo15