Issues with reconnecting to p2p network with producers.
Scenario:
node1 - just a p2p node
node2 - just a p2p node (with impatient operator)
node3 - api node
all nodes are in the region that lost connection to the outside world, i.e. block producing nodes, and because they are firewalled they can't be contacted by the outside world via p2p (connections are outgoing to seed nodes).
-
it takes forever to actually reconnect after network connectivity is back
impatient operator of node2 manually restarts their node after 10-15 minutes so it will connect to seed node and sync quickly, in such case node1 and node3 quickly follows well, except not because: -
postponed transactions are now processed because unlike node2
,node1
wasn't restarted so it has a bunch of transactions that were stacked and now getting applied:
2022-08-24T12:54:00.722177 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 653353μs
2022-08-24T12:54:00.942302 db_with.hpp:166 ~pending_transaction ] Postponed 14873 pending transactions. 4115 were applied. 11 expired.
2022-08-24T12:54:01.153829 db_with.hpp:166 ~pending_transaction ] Postponed 11866 pending transactions. 7093 were applied. 18 expired.
2022-08-24T12:54:01.373537 db_with.hpp:166 ~pending_transaction ] Postponed 11819 pending transactions. 7114 were applied. 17 expired.
2022-08-24T12:54:01.373660 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 651468μs
2022-08-24T12:54:01.593316 db_with.hpp:166 ~pending_transaction ] Postponed 14884 pending transactions. 4028 were applied. 15 expired.
2022-08-24T12:54:01.806205 db_with.hpp:166 ~pending_transaction ] Postponed 12183 pending transactions. 6707 were applied. 14 expired.
2022-08-24T12:54:02.025962 db_with.hpp:166 ~pending_transaction ] Postponed 11784 pending transactions. 7086 were applied. 12 expired.
2022-08-24T12:54:02.026085 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 652410μs
2022-08-24T12:54:02.247359 db_with.hpp:166 ~pending_transaction ] Postponed 14977 pending transactions. 3858 were applied. 29 expired.
2022-08-24T12:54:02.457302 db_with.hpp:166 ~pending_transaction ] Postponed 11665 pending transactions. 7121 were applied. 40 expired.
2022-08-24T12:54:02.678741 db_with.hpp:166 ~pending_transaction ] Postponed 11690 pending transactions. 7077 were applied. 15 expired.
2022-08-24T12:54:02.678861 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 652762μs
2022-08-24T12:54:02.898285 db_with.hpp:166 ~pending_transaction ] Postponed 14831 pending transactions. 3918 were applied. 13 expired.
2022-08-24T12:54:03.108626 db_with.hpp:166 ~pending_transaction ] Postponed 11689 pending transactions. 7045 were applied. 9 expired.
2022-08-24T12:54:03.330930 db_with.hpp:166 ~pending_transaction ] Postponed 11767 pending transactions. 6945 were applied. 11 expired.
2022-08-24T12:54:03.331067 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 652192μs
2022-08-24T12:54:03.550495 db_with.hpp:166 ~pending_transaction ] Postponed 14750 pending transactions. 3941 were applied. 15 expired.
2022-08-24T12:54:03.762204 db_with.hpp:166 ~pending_transaction ] Postponed 11520 pending transactions. 7151 were applied. 7 expired.
2022-08-24T12:54:03.982105 db_with.hpp:166 ~pending_transaction ] Postponed 11525 pending transactions. 7106 were applied. 35 expired.
2022-08-24T12:54:03.982223 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 651141μs
2022-08-24T12:54:04.202063 db_with.hpp:166 ~pending_transaction ] Postponed 14724 pending transactions. 3888 were applied. 13 expired.
2022-08-24T12:54:04.412082 db_with.hpp:166 ~pending_transaction ] Postponed 11539 pending transactions. 7042 were applied. 8 expired.
2022-08-24T12:54:04.631771 db_with.hpp:166 ~pending_transaction ] Postponed 11680 pending transactions. 6882 were applied. 14 expired.
2022-08-24T12:54:04.631934 chain_plugin.cpp:419 operator() ] Stopped processing write_queue before empty because we exceeded 500ms, held lock for 649697μs