Create abstraction for SMTP queue storage

[foxcpp]

That would enable writing alternative queue storage implementations. Discussion needed to figure out if there is need and what considerations are.

[joe-getcouragenow]

It would be great is the storage can be integrated into the golang code and for it to be HA.
So for a basic user that just need to start the binary on 1 or 3 servers and the whole thing stays in sync and does leader elections etc automagically.

This does it.
https://github.com/lni/dragonboat

And many devs use it to make KV stores in golang to be HA. So it is mature.
Some examples are: https://github.com/search?q=lni%2Fdragonboat%2Fv3&type=Code

[foxcpp]

Things to be abstracted out from queue code are:

1. Scheduler

Determines when message delivery should be attempted. Interface is roughly this:

Schedule(time time.Time, attemptId string)
Wait() (attemptId string)

This part is already mostly separated - in current implementation that's TimeWheel struct (funny name, should have named it TimerBucket or something :) ).

Such scheduler should be able to preserve information across server restarts and so on. In current implementation queue timers
are recreated from meta-data stored in a filesystem directory.

2. Meta-data & message store.

Message queue needs to keep 3 pieces of data: Meta-data (serialized module.MsgMetaData structure), message header and message body.

This part of queue code is hard-wired with the rest of queue logic so some refactoring is required here.
I would define the interface like this:

StoreMessage(attemptId string, metadata QueueMeta, hdr textproto.Header, body io.Reader)
LoadMetadata(attemptId string) QueueMeta
LoadHeader(attemptId string) textproto.Header
LoadBody(attemptId string) io.Reader

Can take a step forward and use a dumb key-value store interface and let queue itself handle necessary serialization
(like imapsql module has abstracted message contents storage).
Note it is important for such storage to support large blobs and streaming, potentially with local cache since body may be read several times on delivery.

[foxcpp]

It would be great is the storage can be integrated into the golang code and for it to be HA.

Frankly, I am not familiar with consensus protocols like Raft. So I was hoping for feedback on what kind of interface would be appropriate between queue code and storage. I wrote a detailed description above.

[pcmid]

I have an idea to achieve this using NATS Jetstream.
By importing github.com/nats-io/nats-server/v2/server, it can be built into golang code to implement a distributed message queue.