diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000000000000000000000000000000000000..9319f5e020dc58c94b182dceffbe2e2b618fd21c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,169 @@ +# HafSQL + +## Project Overview + +HafSQL is a HAF (Hive Application Framework) application that creates a modular SQL interface to blockchain data within the HAF database. It provides: + +- **Operations & Virtual Operations**: Access to 50 standard operations and 43 virtual operations with searchable parameters +- **Parsed Data Structures**: + - Posts/comments (searchable by author/permlink, parent relationships, tags, pending payout) + - Reblogs with account and post tracking + - HP and RC delegations + - Community subscriptions and roles + - Followers, mutes, and blacklists + - Proposal voters + +The application runs as a HAF schema module (`hafsql` schema) and maintains live sync with blockchain data via configurable modular syncing. + +## Tech Stack + +- **Language**: JavaScript (Node.js 18+, ES modules) +- **Database**: PostgreSQL (HAF Block Log database - v1.27.4 required) +- **Process Manager**: PM2 + +**Key Dependencies**: +- `pg` - PostgreSQL client +- `dotenv` - Environment configuration +- `json-bigint` - BigInt JSON parsing for blockchain numbers +- `diff-match-patch` - Diff/patch logic for comment body editing + +**Dev Dependencies**: +- `pm2` - Process manager for daemon +- `standard` - JavaScript linter (StandardJS code style) + +## Directory Structure + +``` +hafsql/ +├── src/ # Main application files +│ ├── main.js # Entry point for live sync daemon +│ ├── indexes.js # PostgreSQL index creation +│ ├── setup.js # Database initialization orchestrator +│ └── dropIndexes.js # Index cleanup +├── helpers/ +│ ├── database.js # PostgreSQL pool configuration +│ ├── validateUsername.js # Account name validation utility +│ ├── syncs/ # Live sync modules per feature +│ │ ├── comments.js # Comments, body edits, tags +│ │ ├── delegations.js # HP delegations +│ │ ├── rcDelegations.js # RC delegations +│ │ ├── proposalApprovals.js +│ │ ├── follows.js # Followers, mutes, blacklists +│ │ ├── reblogs.js +│ │ ├── rewards.js +│ │ ├── deleteComments.js +│ │ └── communities.js # Community subs & roles +│ └── setups/ # Database schema initialization +│ ├── schema.js # Create hafsql schema +│ ├── tables.js # Create tracking tables +│ ├── functions.js # SQL functions +│ ├── operations.js # 50 operation type views +│ ├── virtualOperations.js # 43 virtual op views +│ └── extraViews.js # Aggregation/helper views +├── ecosystem.config.cjs # PM2 daemon config (4GB memory limit) +├── Dockerfile # Docker build config +├── example.env # Configuration template +├── run.sh # Helper script +└── drop-indexes.sql # SQL script for index removal +``` + +## Development Commands + +**npm scripts**: +```bash +npm install # Install dependencies +npm run create-indexes # Create PostgreSQL indexes (blocking, long-running) +npm run start # Start daemon via PM2 +npm run restart # Restart daemon +npm run stop # Stop daemon +npm run logs # View PM2 logs +npm run list # List PM2 processes +``` + +**Helper script** (`run.sh`): +```bash +./run.sh install_node # Install Node.js 18 on Ubuntu 22 +./run.sh install # npm install +./run.sh start/stop/restart/logs +``` + +**Docker**: +```bash +docker build -t hafsql-v1.0.2 . +docker run --rm -it --name hafsql-sync hafsql-v1.0.2 +``` + +**Setup flow**: +1. `npm install` +2. Configure `.env` from `example.env` +3. `npm run create-indexes` (24h+ with CONCURRENTLY=true, ~4h with false) +4. `npm run start` + +## Key Files + +- `src/main.js` - Entry point: runs setup, initial batch sync, then live 3-second interval sync +- `helpers/database.js` - PostgreSQL pool configuration and exports +- `helpers/syncs/*.js` - Each feature has `fillFeature()` (batch) + `syncFeature()` (live) functions +- `helpers/setups/operations.js` - Creates 50 operation type views +- `helpers/setups/virtualOperations.js` - Creates 43 virtual operation views +- `ecosystem.config.cjs` - PM2 config with 4GB memory restart limit +- `example.env` - Configuration template with all options documented + +## Configuration (Environment Variables) + +```conf +# Database Connection +PGDATABASE=haf_block_log +PGUSER=haf_app_admin +PGHOST=172.17.0.2 +PGPORT=5432 +PGPOOLSIZE=2 + +# Index Creation +CONCURRENTLY=true # true = non-blocking, false = faster but blocks sync +INDEXMAXTHREADS=4 # Recommend 8-12 for NVMe + +# Modular Feature Flags (all default true) +COMMENTS=true +COMMUNITIES=true +DELEGATIONS=true +RCDELEGATIONS=true +FOLLOWS=true +PROPOSALS=true +SKIPOPERATIONINDEXES=false # Set true to skip op/vop index creation +``` + +## Coding Conventions + +- **Code Style**: StandardJS (zero-config, no semicolons) +- **Tab Size**: 2 spaces +- **Module Type**: ES modules (`import/export`) +- **Auto-fix**: VSCode configured with `standard.autoFixOnSave` + +**Architectural Patterns**: + +1. **Modular Syncing**: Each feature has dedicated `syncs/*.js` module with `fillFeature()` (batch) + `syncFeature()` (live interval) pattern + +2. **Sync Tracking**: `hafsql.sync_data` table tracks `last_op_id` per feature; batch processing fetches operations > last_op_id + +3. **Bulk Inserts**: Uses PostgreSQL UNNEST with multiple arrays for single INSERT with ON CONFLICT upserts + +4. **View-Based Parsing**: Operations views use JSONB operators to extract searchable fields from raw operation bodies + +5. **Database Pattern**: Shared pool from `helpers/database.js`, parameterized queries (`$1, $2` style) + +## CI/CD Notes + +No CI/CD pipeline configured in this repository. + +**Deployment**: +- Containerized via Dockerfile (node:18 base) +- PM2 manages process lifecycle with 4GB memory restart limit +- Requires PostgreSQL accessible at PGHOST:PGPORT +- Requires pre-existing HAF schema with `hive.operations` table + +**Production Considerations**: +- HAF version 1.27.4 exactly required (incompatible with 1.27.3, 1.27.5) +- Index creation: 24+ hours (concurrent) or ~4 hours (blocking) +- 400GB+ disk space for indexes, 56GB for tables +- Each feature can be disabled via env flags for lightweight deployments