Can't launch HAF docker with --skip-hived

Running this image registry.gitlab.syncad.com/hive/haf/instance:d70a9632

@wrona can we get someone to investigate it

Any update on this?

I've pinged bartek directly to get someone to fix it, I don't think it should be hard, and I also like this feature.

assigned to @wrona

mentioned in commit a43dc781

mentioned in merge request !543 (merged)

mentioned in commit ce2cfd63

Done in !543 (merged)

closed

reopened

Removing httpd from docker entry point isn't best solution, as having httpd launched was to allow dependencies to launch gracefully.

I'm reopening this issue, since I think the root problem is a difference between images used for CI and production. IMO the CI image should be some form of production image (possibly with extra stuff). In this case, httpd would be in the image and this problem would not occur. As a side issue, the CI dockers should probably be clearly named so that they won't be used by devs, as this was secondary reason for this problem (mahdi was using wrong image it seems).

what is the correct image?

Ok, I didn't know that opened port is important in this scenario.

Production image just should contain httpd. There should be no dedicated HAF image for CI, since we need to test production version. I have some idea how to solve need to have a lot of dependencies required to perform CI and still use minimized production image, but it can take definitely more time. Atm I am going to extend minimal image for httpd.

httpd would be an overkill
ncat -lk -p 8091 --sh-exec 'echo -e "HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"'
does the thing, sure, it just serves dummy responses but that's exactly what healthcheck is looking for

The thing I'm confused about is httpd is already there and works in the images I am testing (this includes recent develop ones). But bartek said there is no dedicated haf image for CI? So what is different about the one that doesn't work?

We have 2 kinds of images:

instance (derived from ci-base-image) which is huge and contains testing environment. This version also does not contain httpd, so I doubt --skip-hived works there

syncad@vm000:/storage1/src/haf$ docker run -it --rm --entrypoint bash registry.gitlab.syncad.com/hive/haf/instance:local
haf_admin@67b90614053c:~$ httpd
bash: httpd: command not found

minimal-instance (derived from minimal-runtime) which has less installed packages than ci-base-image.

My change in !544 (merged) added small package: netcat-openbsd to both and uses it to perform HTTP listening of configured HTTP port.

Current state is bad (according to testing) since on CI we are using instance images instead of minimal-instance what should be changed to always verify production ready version. But to do it, we need to change CI job definitions (especially for hive fork manager unit tests) what can be quite tricky.

I hope !544 (merged) covered completely this issue

closed

mentioned in commit a9e88238

reopened

@wrona skip-hived seems to be broken again. Looks like it tries to pass the argument on to hived itself.

Attempting to collect unknown (hived) option: --block-log-split=-1 --skip-hived
/home/haf_admin/docker_entrypoint.sh
enabling pg_cron
 * Restarting PostgreSQL 17 database server
   ...done.
NOTICE:  extension "pg_cron" already exists, skipping
CREATE EXTENSION
GRANT
GRANT
SET
 schedule
----------
        1
(1 row)

 schedule
----------
        2
(1 row)

[1]+   266 Running                 { sudo --user=hived -En LD_PRELOAD="${OVERRIDE_LD_PRELOAD:-}" /bin/bash <<EOF
echo "Attempting to execute hived using additional command line arguments:" "${HIVED_ARGS[*]}"
set -euo pipefail

if [ ! -f "$DATADIR/config.ini" ]; then
  echo "No config file exists, creating a default config file"

  /home/hived/bin/hived --webserver-ws-endpoint=0.0.0.0:${WS_PORT} --webserver-http-endpoint=0.0.0.0:${HTTP_PORT} --p2p-endpoint=0.0.0.0:${P2P_PORT}     --data-dir="$DATADIR" --shared-file-dir="$SHM_DIR" --psql-wal-directory="$WAL_DIR"     --plugin=sql_serializer --psql-url="dbname=haf_block_log host=/var/run/postgresql port=5432"     ${HIVED_ARGS[@]} --dump-config > /dev/null 2>&1

  # add a default set of plugins that API nodes should run
  sed -i 's/^# plugin = .*$/plugin = node_status_api account_by_key account_by_key_api block_api condenser_api database_api json_rpc market_history market_history_api network_broadcast_api p2p rc_api state_snapshot transaction_status transaction_status_api wallet_bridge_api webserver/g' "$DATADIR/config.ini"

  # set a default logging config.  We will log the usual output both to stderr and to a log file in the
  # haf-datadir/logs/hived/default directory.  Rotate daily, keep for 30 days.
  sed -i 's|^# log-appender = .*$|log-appender = {"appender":"stderr","stream":"std_error","time_format":"iso_8601_microseconds"} {"appender":"p2p","file":"logs/hived/p2p/p2p.log","truncate":false,"time_format":"iso_8601_milliseconds", "rotation_interval": 86400, "rotation_limit": 2592000} {"appender": "default", "file": "logs/hived/default/default.log","truncate":false, "time_format": "iso_8601_milliseconds", "rotation_interval": 86400, "rotation_limit": 2592000}|;s|^log-logger = .*$|log-logger = {"name":"default","level":"info","appenders":["stderr", "default"]} {"name":"user","level":"debug","appenders":["stderr", "default"]} {"name":"p2p","level":"warn","appenders":["p2p"]}|' "$DATADIR/config.ini"
fi

/home/hived/bin/hived --webserver-ws-endpoint=0.0.0.0:${WS_PORT} --webserver-http-endpoint=0.0.0.0:${HTTP_PORT} --p2p-endpoint=0.0.0.0:${P2P_PORT}   --data-dir="$DATADIR" --shared-file-dir="$SHM_DIR" --psql-wal-directory="$WAL_DIR"   --plugin=sql_serializer --psql-url="dbname=haf_block_log host=/var/run/postgresql port=5432"   ${HIVED_ARGS[@]}
hived_return_code="\$?"
echo "\$hived_return_code Hived process finished execution."
EOF
 stop_postresql; } &
waiting for job finish: 266.
Attempting to execute hived using additional command line arguments: --block-log-split=-1 --skip-hived
Error parsing command line: unrecognised option '--skip-hived'
Hived process finished execution: return status: 1.
Performing cleanup....
Hived pid:
Waiting for hived finish...
Hived finish done.
Attempting to stop Postgresql...
 * Stopping PostgreSQL 17 database server
   ...done.
Waiting for postgres process: 232 finish...
Postgres process: 232 finished.
Cleanup actions done.

This is the 2nd time it has broken, so I suppose it is another argument for separating hived and haf (i.e. postgres) into separate dockers.

This issue doesn't occur if you only pass --skip-hived as a argument. But it is easy to duplicate if you pass a hived argument as well. For example, in a docker compose .env file, try:

ARGUMENTS="--split-block-log=-1 --skip-hived"

mentioned in commit haf_api_node@f7ae991f

mentioned in merge request haf_api_node!72 (merged)

Fix is here: args have been improperly passed at docker compose side. I Also think there is no need to split images (we have too much images in each app) what will make more troubles than benefits

Thanks for the fix. However, I strongly disagree on need to split hived and postgres into separate apps. This change will allow for very useful configuration of running each on separate server, enabling operation on servers with fewer individual resources.

closed

Can't launch HAF docker with --skip-hived

Designs

Child items ...

Activity