---
title: titles.haf_api
position: 5
description: descriptions.haf_api
exclude: true
layout: full
canonical_url: haf-api.html
---


### Using docker compose to install and maintain a HAF server and HAF apps

#### System Requirements

We assume the base system will be running at least Ubuntu 22.04 (jammy).  Everything will likely work with later versions of Ubuntu. IMPORTANT UPDATE: experiments have shown 20% better API performance when running U23.10, so this latter version is recommended over Ubuntu 22 as a hosting OS.

For a mainnet API node, we recommend:
- at least 32GB of memory.  If you have 64GB, it will improve the time it takes to sync from scratch, but
  it should make less of a difference if you're starting from a mostly-synced HAF node (i.e.,
  restoring a recent ZFS snapshot) (TODO: quantify this?)
- 4TB of NVMe storage
    - Hive block log & shared memory: 500GB
    - Base HAF database: 3.5T (before 2x lz4 compression)
    - Hivemind database: 0.65T (before 2x lz4 compression)
    - base HAF + Hivemind:  2.14T (compressed)
    - HAF Block Explorer: xxx

# Install prerequisites

## Install ZFS support

We strongly recommend running your HAF instance on a ZFS filesystem, and this documentation assumes
you will be running ZFS.  Its compression and snapshot features are particularly useful when running a HAF node.

We intend to publish ZFS snapshots of fully-synced HAF nodes that can downloaded to get a HAF node
up & running quickly, avoiding multi-day replay times.

```
sudo apt install zfsutils-linux
```

## Install Docker
Install the latest docker.  If you're running Ubuntu 22.04, the version provided by the
native docker.io package is too old to work with the compose scripts.  Install the latest
version from docker.com, following the instructions here:

https://docs.docker.com/engine/install/ubuntu/

Which are:
```
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

## Create a ZFS pool

Create your ZFS pool if necessary.  HAF requires at least 4TB of space, and 2TB NVMe drives are
readily available, so we typically construct a pool striping data across several 2TB drives.
If you have three or four drives, you will get somewhat better read/write performance, and
the extra space can come in handy.

To create a pool named "haf-pool" using the first two NVMe drives in your system,
use a command like:
```
sudo zpool create haf-pool /dev/nvme0n1 /dev/nvme1n1
```
If you name your ZFS pool something else, configure the name in the environment file,
as described in the next section.

Note: By default, ZFS tries to detect your disk's actual sector size, but it often gets it wrong
for modern NVMe drives, which will degrade performance due to having to write the same sector multiple
times.  If you don't know the actual sector size, we recommend forcing the sector size to 8k by
specifying setting ashift=13 (e.g., `zfs create -o ashift=13 haf-pool /dev....`)

## Configure your environment

Make a copy of the file `.env.example` and customize it for your system.  This file contains
configurable paramters for things like
- directories
- versions of hived, HAF, and associated tools

The `docker compose` command will automatically read the file named `.env`.  If you want to
keep multiple configurations, you can give your environment files different names like
`.env.dev` and `.env.prod`, then explicitly specify the filename when running `docker compose`:
`docker compose --env-file=.env.dev ...`

## Set up ZFS filesystems

The HAF installation is spread across multiple ZFS datasets, which allows us to set different
ZFS options for different portions of the data. We recommend that most nodes keep the default
datasets in order to enable easy sharing of snapshots.

### Initializing from scratch

If you're starting from scratch, after you've created your zpool and configured its name in the .env file
as described above, run:
```
sudo ./create_zfs_datasets.sh
```
to create and mount the datasets.

By default, the dataset holding most of the database storage uses zfs compression. The dataset for
the blockchain data directory (which holds the block_log for hived and the shared_memory.bin file)
is not compressed because hived directly manages compression of the block_log file.

If you have a LOT of nvme storage (e.g. 6TB+), you can get better API performance at the cost of disk
storage by disabling ZFS compression on the database dataset, but for most nodes this isn't recommended.

#### Speeding up the initial sync

Following the instructions above will get you a working HAF node, but there are some things you can
do to speed up the initial sync.

##### Replaying
If you already have a recent block_log file (e.g., you're already running another instance of hived
somewhere else on your local network), you can copy the block_log and block_log.artifacts files
from that node into your /haf-pool/haf-datadir/blockchain directory.  After copying the files,
make sure the ownership is set to the same owner as the /haf-pool/haf-datadir/blockchain directory
so hived can read/write them: `chown 1000:100 block_log block_log.artifacts`

Before brining up the haf service, you will also need to add the `--replay-blockchain` argument to
hived to tell it you want to replay.  Edit the `.env` file's `ARGUMENTS` line like so:
```
ARGUMENTS="--replay-blockchain"
```
Once the replay has finished, you can revert the `ARGUMENTS` line to the empty string

##### Shared Memory on Ramdisk
If you have enough spare memory on your system, you can speed up the initial replay by placing the
`shared_memory.bin` file on a ramdisk.

The current default shared memory filesize is 24G, so this will only work if you have 24G free
(that's in addition to the memory you expect to be used by hived and HAF's integrated PostgreSQL
instance).

If you have a 64GB system, ensure you have a big enough swapfile (32GB is recommended
and 8GB is known to not be sufficient) to handle peak memory usage needs during the replay.
Peak memory usage currently occurs when haf table indexes are being built during the final
stage of replay.

To do this, first create a ramdisk:
```
sudo mkdir /mnt/haf_shared_mem

# then
sudo mount -t tmpfs -o size=25g tmpfs /mnt/haf_shared_mem
# - or -
sudo mount -t ramfs ramfs /mnt/haf_shared_mem

# then
sudo chown 1000:100 /mnt/haf_shared_mem
```

Then, edit your `.env` file to tell it where to put the shared memory file:
```
HAF_SHM_DIRECTORY="/mnt/haf_shared_mem"
```

Now, when you resync / replay, your shared memory file will actually be in memory.

###### Moving Shared Memory back to disk
Once your replay is finished, we suggest moving the shared_memory.bin file back to NVMe storage,
because:
- it doesn't make much performance difference once hived is in sync
- you'll be able to have your zfs snapshots include your shared memory file
- you won't be forced to replay if the power goes out

To do this:

- take down the stack (`docker compose down`).
- copy the shared memory: `sudo cp /mnt/haf_shared_mem/shared_memory.bin /haf-pool/haf-datadir/blockchain`
- destroy the ramdisk: `sudo umount /mnt/haf_shared_mem`
- update the `.env` file's location: `HAF_SHM_DIRECTORY="${TOP_LEVEL_DATASET_MOUNTPOINT}/blockchain"`
- bring the stack back up (`docker compose up -d`)

### Initializing from a snapshot

If you're starting with one of our snapshots, the process of restoring the snapshots will create the correct
datasets with the correct options set.

First, download the snapshot file from: TODO: http://xxxxxx

Since these snapshots are huge, it's best to download the snapshot file to a different disk (a magnetic
HDD will be fine for this) that has enough free space for the snapshot first, then restore it to the ZFS pool.
This lets you easily resume the download if your transfer is interrupted.  If you download directly to
the ZFS pool, any interruption would require you to start the download from the beginning.

```
wget -c https://whatever.net/snapshot_filename
```
If the transfer gets interrupted, run the same command again to resume.

Then, to restore the snapshot, run:
```
sudo zfs recv -d -v haf-pool < snapshot_filename
```

## Launch procedure

---

start/stop HAF instance based on profiles enabled in your `.env` file

```
docker compose up -d

docker compose logs -f hivemind-block-processing # tail the hivemind sync logs to the console
docker compose down hivemind-block-processing # shut down just the hivemind sync process
docker compose up -d hivemind-block-processing # bring hivemind sync process back up

docker compose down # shut down all containers
```

This will start or stop all services selected by the profiles you have
enabled in the `.env` file's `COMPOSE_PROFILES` variable.

Currently available profiles are:
- `core`: the minimal HAF system of a database and hived
- `admin`: useful tools for administrating HAF: pgadmin, pghero
- `apps`: core HAF apps: hivemind, HAfAH, haf-block-explorer
- `servers`: services for routing/caching API calls: haproxy, jussi,varnish

# Observing node startup

After you start your HAF instance, hived will need some time to catch up to the head block
of the Hive blockchain (typically a few minutes or less if you started from a snapshot,
otherwise it will take many hours or even days depending on your hardware). You can monitor
this process using: `docker compose logs -f haf`

If syncing or replaying for the first time, HAF will delay creating indexes on its tables until the blockchain data has mostly been added to the database. This means there will be a noticeable delay near the end of the catchup period while these indexes get created. Even on a fast machine this post-sync/replay process currently takes over 2 hours to create the indexes, and another two hours to cluster the account_operations table, so be patient. Do not interrupt the process or your database will be left in an invalid state and might require another full replay.

If you enabled the "admin" profile, you can use pghero's "Live Queries" view to monitor this process (e.g https://your_server/admin/pghero/live_queries). If not, you can still observe the cpu and disk io usage by postgresql during this process if you run a tool like htop.

# After startup: Monitoring services and troubleshooting failures on your API node

Haproxy can be used to monitor the state of the various services on your HAF server:
`https://your_server_name/admin/haproxy/`

If you see a service is down, you can use an appropriate `docker compose log` command to
diagnose the issue. When diagnosing issues, keep in mind that several services depend on other services
(for example, all haf apps depend on the hived service) so start by checking the health of the lowest level
services.

You can diagnose API performance problems using pgAdmin and PgHero. pgAdmin is best for diagnosing severe problems (e.g. locked tables, etc) whereas PgHero is typically best for profiling to determine what queries are loading down your server and can potentially be optimized.

https://your_server_name/admin/

# Creating a ZFS snapshot to backup your node
Creating snapshots is fast and easy:

```
docker compose down  #shut down haf
./snapshot_zfs_datasets.sh 20231023T1831Z-haf-only # where 20231023T1831Z-haf-only is an example snapshot name
docker compose up -d
```
Note: snapshot_zfs_datasets.sh unmounts the HAF datasets, takes a snapshot, and remounts them. Since it unmounts the datasets, the script will fail if you have anything accessing the datasets. In particular, be sure you don't have any terminals open with a current working directory set to those datasets. In theory, the script shouldn't have to unmount the datasets before taking the snapshot, but we have occassionally encountered issues where the snapshots didn't get all needed data.

# Deleting Hivemind data from your database (or a similar app's data)

You may want to remove the Hivemind app's data from your database -- either because you no longer
need it and want to free the space, or because you want want to replay your Hivemind app from
scratch, which is required for some upgrades.

To delete the data:
- stop Hivemind, but leave the rest of the stack running: `docker compose down hivemind-install hivemind-block-processing hivemind-server`
- run the uninstall script: `docker compose --profile=hivemind-uninstall up`
- you'll see the results of a few sql statements scroll by, and it should exit after a few seconds

The Hivemind data is now gone.

If you're uninstalling Hivemind permanently, then remember to remove the `hivemind` profile from your `.env` file's `COMPOSE_PROFILES` line so it doesn't start automatically next time you do a `docker compose up -d`.

If you're upgrading to a new version of hivemind:
- if you're upgrading to a pre-release version, you'll need to set `HIVEMIND_INSTANCE_VERSION` in your `.env` file to the correct tag for the version you want to run.  If you're just upgrading to a new release version (the ones tagged `haf_api_node`), you can leave this alone.
- run `docker compose pull` to grab the new version
- run `docker compose up -d` to bring up all services.  This should run hivemind's install, then launch the block processing container.
- you can monitor Hivemind's sync process by watching the logs from `docker compose logs -f hivemind-block-processing`.  In a few short days, your Hivemind app should be fully synced and ready to handle API requests.
