Sync'ing from a file is failing in 6.2.46: "Skip badly formed line"

[pkissman]

I just upgraded from 6.2.44 to 6.2.46 from source on a Debian host.
I have two flat files of email addresses that are included into some lists.
Sync'ing those files now completely fails with "Skip badly formed line" until the process aborts.

I am now seeing this in my sympa log file:

Sep 26 08:58:46 mblc-web01 wwsympa[31083]: err main::#1544 > main::do_sync_include#16517 > Sympa::List::sync_include#4825 > Sympa::Spindle::spin#95 > Sympa::Request::Handler::include::_twist#181 > Sympa::Request::Handler::include::_update_users#292 > Sympa::DataSource::next#149 > Sympa::DataSource::File::_next#75 Skip badly formed line: "[email protected]#15"

Eventually I see the error:
Sympa::Request::Handler::include::_twist#181 > Sympa::Request::Handler::include::_update_users#292 > Sympa::DataSource::next#149 > Sympa::DataSource::File::_next#65 Too much errors in file /etc/sympa/pubdir-announce. Source file probably corrupted. Cancelling

There have been no changes other than the upgrade. The included source file is exactly the same.

[email protected]
[email protected]

with unix linefeeds.

Any help would be appreciated.

Version

Installation method

Expected behavior

Actual behavior

Additional information

[ikedas]

Hi @pkissman,

I guess the error message tells that the file contains lines with Microsoft newlines (CR+LF).

[racke]

@ikedas we should deal with that instead of throwing an error.

[pkissman]

I thought I had tried to convert the included file back and forth between a dos format and a unix format, since that was the first thing I had thought of. But then I went to the script that generates the included file and it is definitely putting in a \r\n, so I guess I was wrong about my own file format. Sorry about that. But this is definitely a change in behavior between 6.2.44 and 6.2.46. I have rolled everything to 6.2.44 since these mailing lists were needed. But I will reinstall 6.2.46 on our development server and try again I’ll report back whether the newline change solves the problem for me. Thanks for your help. From: Stefan Hornburg (Racke) [mailto:[email protected]] Sent: Thursday, September 26, 2019 10:05 AM To: sympa-community/sympa Cc: Kissman, Paul (BLC); Mention Subject: Re: [sympa-community/sympa] Sync'ing from a file is failing in 6.2.46: "Skip badly formed line" (#766) @ikedas<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ikedas&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=tnRMsxswy2fkBpmtJQ_kwpnVQOsNLIVVbJj0W1SUEkM&s=3q7ALA-tCpVxT51vdPASgJ7fexvC73JohGm8rzbAiHo&e=> we should deal with that instead of throwing an error. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sympa-2Dcommunity_sympa_issues_766-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAGILKX2H34EM5WMKSSLGSLTQLS6QBA5CNFSM4I2Z5TJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7VVTMY-23issuecomment-2D535517619&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=tnRMsxswy2fkBpmtJQ_kwpnVQOsNLIVVbJj0W1SUEkM&s=cxHjpasfJS3xlOk1FUb8qqevI2CKwqYVxOWneFowULU&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AGILKX5FTFXF3TPPHKP5K4TQLS6QBANCNFSM4I2Z5TJQ&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=tnRMsxswy2fkBpmtJQ_kwpnVQOsNLIVVbJj0W1SUEkM&s=c4lgJejIlnn50w9y1dtPG0wtYvouQNl9Y43GOdn2pO8&e=>.

[pkissman]

@ikedas and @racke ,

I reinstalled 6.2.46 and regenerated the included file I was having trouble with, but with Unix line breaks, and it did indeed solve the problem. Thank you.

I was also suffering from issues #763 every time I added and removed data sources. #764 solved that problem.

However, I don't remember that email addresses which have been included from a file remained subscribed once the data source is removed. I thought you still had the ability to sync the subscribers so that the included emails were automatically removed. Is this new behavior, or am I misremembering?

[ikedas]

Current code looks supporting both LF and CR+LF:

sympa/src/lib/Sympa/DataSource/File.pm

Lines 58 to 79 in c4087f3

	chomp $line;
	if (++$lines > 49 and not $found) {
	$log->syslog(
	'err',
	'Too much errors in file %s. Source file probably corrupted. Cancelling',
	$self->{path}
	);
	return undef;
	}
	# Empty lines are skipped
	next if $line =~ /^\s*$/;
	next if $line =~ /^\s*\#/;
	# Skip badly formed emails.
	unless ($line =~ /\A\s*($email_re)(?:\s+(\S.*))?\z/) {
	$log->syslog('err', 'Skip badly formed line: "%s"', $line);
	next;
	}
	my ($email, $gecos) = ($1, $2);
	$gecos =~ s/\s+\z// if defined $gecos;

It’s curious... By any chance you have been using CR+CR+LF? It was allowed by previous versions.

Anyways it’s incompatible change. I’ll submit fixes later.

[ikedas]

Hi @pkissman,

Could you please apply this patch and check if the problem will be solved?

[ikedas]

Related to #693.

[pkissman]

Hi @ikedas, Yes. This patch did fix the problem. Files with CRLFs now get included properly. From: IKEDA Soji [mailto:[email protected]] Sent: Friday, September 27, 2019 12:13 AM To: sympa-community/sympa Cc: Kissman, Paul (BLC); Mention Subject: Re: [sympa-community/sympa] Sync'ing from a file is failing in 6.2.46: "Skip badly formed line" (#766) Hi @pkissman<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pkissman&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=ctQDKFEFnwCotUeE_au33eXWbVemo2VKDv7nueg3V9Y&s=zLxwrytBd-GEGJFDilfQcqbjxZ6SOZBAdpvgUQPZGTc&e=>, Could you please apply this patch<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sympa-2Dcommunity_sympa_commit_d21684511ae3e81bd37b2ab2efe6c31eb19f0087.diff&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=ctQDKFEFnwCotUeE_au33eXWbVemo2VKDv7nueg3V9Y&s=8cPM0MWYUZu-YtV9oDnaAO6DAXzCmiN3ZgOo7XoQiGU&e=> and check if the problem will be solved? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_sympa-2Dcommunity_sympa_issues_766-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAGILKX7OTDEAJYN64EZKK3LQLWB5PA5CNFSM4I2Z5TJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7XVHUY-23issuecomment-2D535778259&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=ctQDKFEFnwCotUeE_au33eXWbVemo2VKDv7nueg3V9Y&s=5fBAFrENaxNDzNLGng8gGz_sZluu7yYOSsUTFr1Iweo&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AGILKX4GQIFKZEP2CCQGI5TQLWB5PANCNFSM4I2Z5TJQ&d=DwMCaQ&c=lDF7oMaPKXpkYvev9V-fVahWL0QWnGCCAfCDz1Bns_w&r=MonHUvX27fwQ9bMHWH74_alfWcxUR3Nnj7OGF-5yS48&m=ctQDKFEFnwCotUeE_au33eXWbVemo2VKDv7nueg3V9Y&s=X9PDb8f3c4sw-EawxRr5jdyI7fZGZcDXf40s3miuKy4&e=>.