[Messages] Sync job

Added a new job that synchronizes unconfirmed messages across
the network. The goal of this job is to re-send messages missed
by nodes with the ability to push data on-chain. This can happen
because of various issues like server downtimes or bugs.

This job works in three parts:
* the publisher task periodically sends the list of all the messages
  older than the last TX block that have yet to be confirmed by
  on-chain data.
* the receiver task stores the list of unconfirmed messages for
  each peer detected on the network.
* the sync/aggregator task aggregates the confirmation data from
  all the nodes and fetches messages using the HTTP API.
  These messages are added to the pending message queue.

This solution is less expensive that constantly sharing all
the messages across all the nodes and guarantees that the network
will be synchronized eventually as long as the on-chain data
synchronization jobs are working. With the current implementation,
a message can remain out of sync at a maximum until a new TX
is published on-chain + the job period (5 minutes currently).

Author: odesenfans

src/aleph/config.py

@@ -76,6 +76,7 @@ def get_defaults():
             "gateway_port": 8080,
             "id": None,
             "alive_topic": "ALEPH_ALIVE",
+            "sync_topic": "ALEPH_SYNC",
             "reconnect_delay": 60,
             "peers": [
                 "/dnsaddr/api1.aleph.im/ipfs/12D3KooWNgogVS6o8fVsPdzh2FJpCdJJLVSgJT38XGE1BJoCerHx",

src/aleph/jobs/__init__.py

@@ -4,6 +4,7 @@
 
 from aleph.jobs.process_pending_messages import pending_messages_subprocess, retry_messages_task
 from aleph.jobs.process_pending_txs import pending_txs_subprocess, handle_txs_task
+from aleph.jobs.sync_unconfirmed_messages import sync_unconfirmed_messages_subprocess
 from aleph.jobs.reconnect_ipfs import reconnect_ipfs_job
 
 LOGGER = logging.getLogger("jobs")
@@ -32,8 +33,13 @@ def start_jobs(
             target=pending_txs_subprocess,
             args=(config_values, api_servers),
         )
+        sync_unconfirmed_messages_process = Process(
+            target=sync_unconfirmed_messages_subprocess,
+            args=(config_values, api_servers),
+        )
         p1.start()
         p2.start()
+        sync_unconfirmed_messages_process.start()
     else:
         tasks.append(retry_messages_task(shared_stats=shared_stats))
         tasks.append(handle_txs_task())