Unable to sync/replay due to inability to connect to p2p
OS: Ubuntu 22.04 Hived Version: .3, .4, .5
I have been seeing this behavior recently on multiple geographically unique servers. This particular server has been running for over a year and using the same hived binary. One of our data centers had maintence that required taking the server offline. When brought backup it would spam these messages and nothing would happen.
I am using multiple good seed nodes and have verified ping connectivity as well as the ability to talk to them on port 2001.
974903ms p2p_plugin.cpp:466 plugin_startup ] P2P plugin startup...
974903ms chain_plugin.cpp:356 operator() ] Write processing thread started.
974932ms p2p_plugin.cpp:482 operator() ] Configuring P2P to listen at 0.0.0.0:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 51.161.87.119:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 188.165.235.19:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 104.26.6.217:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 104.26.7.217:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 172.67.73.79:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 188.114.97.3:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 188.114.96.3:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 116.202.160.108:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 91.183.35.138:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 51.222.244.100:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 65.21.199.174:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 195.201.172.94:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 65.21.235.184:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 116.202.168.187:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 88.99.165.91:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 78.46.81.76:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 95.217.203.90:2016
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 185.130.44.110:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 54.37.87.31:2001
974932ms p2p_plugin.cpp:490 operator() ] P2P adding seed node 173.249.42.156:2001
974932ms p2p_plugin.cpp:509 operator() ] Setting parameters
974933ms p2p_plugin.cpp:512 operator() ] Listening to P2P network
974933ms p2p_plugin.cpp:520 operator() ] Connection to P2P network
974933ms p2p_plugin.cpp:522 operator() ] Connected to P2P network
974933ms p2p_plugin.cpp:530 operator() ] P2P node listening at 0.0.0.0:2001
974933ms p2p_plugin.cpp:539 plugin_startup ] P2P Plugin started
974933ms witness_plugin.cpp:639 plugin_startup ] witness plugin: plugin_startup() begin
974933ms witness_plugin.cpp:652 plugin_startup ] Launching block production for 1 witness.
974933ms blockchain_worker_thread_pool.cpp:133 lazy_init ] Emplacing worker threads
974934ms blockchain_worker_thread_pool.cpp:145 lazy_init ] Emplacing worker threads done
974934ms witness_plugin.cpp:663 plugin_startup ] witness plugin: plugin_startup() end
Entering application main loop...
1004903ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1034903ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1064904ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1094904ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1124904ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1154904ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1184904ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1214904ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1244905ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
1274905ms chain_plugin.cpp:397 operator() ] No P2P data (block/transaction) received in last 30 seconds... peer_count=0
I have tried multiple reboots as well as rebuilding .3, .4 and .5 from source. The exact same result regardless, this is a known good machine that has been running for well over a year without issues. I saw the same behavior when building a new node also on Ubuntu 22.04 using the same process I have done 100 times.
I even went so far to reinstall a clean Ubuntu 22.04 and only ran the following commands.
sudo apt update
sudo apt upgrade
sudo reboot
git clone --recurse --branch master https://github.com/openhive-network/hive
sudo ./hive/scripts/setup_ubuntu.sh --dev
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -GNinja ../hive
ninja hived cli_wallet -j8
sudo ninja install
hived --version
screen -S witness -L -Logfile witness.log
hived
I've even tried with rm -rf .hived and running no config.ini, using what it provides by default.
I looked through the peers.json log and all seeds show a similar result as this one.
{
"endpoint": "51.161.87.119:2001",
"last_seen_time": "1970-01-01T00:00:00",
"last_failed_time": "2023-11-08T08:33:05",
"last_connection_disposition": "last_connection_failed",
"last_connection_attempt_time": "2023-11-08T08:33:00",
"number_of_successful_connection_attempts": 0,
"number_of_failed_connection_attempts": 4,
"last_error": {
"code": 0,
"name": "exception",
"message": "unspecified",
"stack": [{
"context": {
"level": "warn",
"file": "node.cpp",
"line": 1493,
"method": "terminate_inactive_connections_loop",
"hostname": "",
"timestamp": "2023-11-08T08:33:05"
},
"format": "Terminating handshaking connection due to inactivity of ${timeout} seconds. Negotiating status: ${status}, bytes sent: ${sent}, bytes received: ${received}",
"data": {
"peer": "51.161.87.119:2001",
"timeout": 5,
"status": "connecting",
"sent": 0,
"received": 0
}
}
Yet, this particular one (seed.hive.blog) I have confirmed I can reach. So it isn't a network or firewall issue.
telnet seed.hive.blog 2001
Trying 51.161.87.119...
Connected to seed.hive.blog.
Escape character is '^]'.
��.�����/U~�S��&ytL���x\�1
I have seen this issue on 2-3 servers in the last couple of weeks all fresh Ubuntu 22.04 installs, with minimum install and configuration done. I also built the docker image, and had same result.
I'm not sure why this is all of a sudden happening, I've done this install a hundred times.