Skip to content

Draft: Implementation of `colony` plugin for producing large amounts of transactions.

Andrzej Lisak requested to merge abw_colony into develop

The goal of the plugin is to generate random transactions (from among 5 types) for every block according to given settings. It should work faster than external script, since it is not limited by API and network throughput. It is also capable of configuring itself on top of initial state and then to dynamically adjust production rate in case it is generating too many transactions for the set size of blocks.

The following options configure the plugin:

  • colony-sign-with - set of WIF private keys to be used to sign every transaction produced. The key set must match active authority of at least one account present in initial state. At startup the plugin selects suitable accounts to be used as colony workers.
  • colony-threads - number of threads used during production. Default is COLONY_DEFAULT_THREADS (4). Each thread has its own exclusive set of worker accounts as well as comment buffer (it is in order to allow threads to work with minimal locking).
  • colony-transactions-per-block - target number of transactions produced per block. When not set, plugin uses sum of weights from individual type-of-transaction settings. If those are also omitted, default is 1000. As mentioned above, if the rate exceeds ability of the node to absorb transactions into blocks, the rate is dynamically limited. Value 0 takes away rate limiting - use it only with queen mode witness or your node will be overrun by pending transactions.
  • colony-start-at-block - allows shifting start of work to specific block. The work will start when block with given number becomes head block (or at plugin startup in case that block already passed), which means first transactions will be targeted for next block. The option is most useful when testing with multiple nodes - pushing start of work away from start of nodes allows the nodes to establish p2p communication without extra traffic from transactions. Also in case you are using multiple nodes with colony enabled, it is recommended to start work at different times on different nodes - it is to lessen the chance for different nodes to randomly create duplicate or otherwise conflicting transactions.
  • colony-no-broadcast - disables broadcasting of produced transactions. It is only suitable for situation when you have active witness on the same node, primarily for queen mode or unit tests.
  • colony-article - json with parameters for article generation (root comment with random beneficiaries) - for details see below
  • colony-reply - for replies (comments to articles/other replies)
  • colony-vote - for votes on comments
  • colony-transfer - for minimal HBD transfers between accounts with memo
  • colony-custom - for custom_json operations with random id and one string value If no type-of-transaction setting is used minimal custom jsons will be produced.

Each type-of-transaction setting is a json with the following fields:

  • min - minimal extra size added to operation (f.e. text in body of comment) - cannot be negative
  • max - maximal extra size added to operation, has to be at least as much as min; each type of operation has different max value (60000 for articles, 20000 for replies, 0 for votes, 2047 for transfers and 8184 for custom_jsons)
  • weight - relative share of that type of operation in transaction mix, cannot be negative
  • exponent - parameter affecting randomness, cannot be negative; effective extra size values are calculated as (max-min) * rand(0.0..1.0)exponent + min, so exponents above 1.0 favor small effective extra sizes, while values below 1.0 tend to produce bigger extra sizes

Example parameters:

colony-article = {"min":100,"max":5000,"weight":16,"exponent":4}
colony-reply = {"min":30,"max":1000,"weight":110,"exponent":5}
colony-vote = {"weight":2070}
colony-transfer = {"min":0,"max":350,"weight":87,"exponent":4}
colony-custom = {"min":20,"max":400,"weight":6006,"exponent":1}

These are frequency values taken from mainnet from RC stats, multiplied by 220 and divided by 28800 (one day worth of blocks):

comments: "count": 16537 (1.52% in relation to other selected operations), "history_bytes": 881
votes: "count": 271050 (24.98%), "history_bytes": 142
transfers: "count": 11364 (1.05%), "history_bytes": 210
custom_jsons: "count": 786241 (72.45%), "history_bytes": 327

Run for a short while on top of state from block_log with 100000 accounts with 2MB blocks, colony produced the following results:

Production stats for thread colony_worker_0
Number of transactions: 169922
Articles: 4664 (%2.74 - 0.2% with substitutions taken into account), avg.extra: 1109, avg.size: 1351
Replies: 2015 (%1.18 - 1.30% with substitutions), avg.extra: 197, avg.size: 360
Votes: 38363 (%22.57 - 25.01% with substitutions), avg.extra: 0, avg.size: 131
Transfers: 1781 (%1.05), avg.extra: 67, avg.size: 190
Custom jsons: 123099 (%72.44), avg.extra: 209, avg.size: 325
196 replies and 4135 votes substituted with articles due to lack of proper target comment
3 transactions failed with exception (including 0 due to lack of RC)
...(continued with 3 other threads)

The resulting values are very close to those from mainnet.

Edited by Andrzej Lisak

Merge request reports