Merge branch 'rc-system' into 'develop'

Rc system

Closes #50 and #31

See merge request !100

_tutorials-python/rcdemo.md

+---
+title: 'PY: Resource Credit System Developer Guide'
+position: 36
+description: "The goal of this guide is to demystify how resources and RC's work.  The intended audience is developers working on Hive user interfaces,
+applications, and client libraries."
+layout: full
+canonical_url: rcdemo.html
+---
+Full, runnable src of [Power Up Hive](https://gitlab.syncad.com/hive/devportal/-/tree/master/tutorials/python/36_rcdemo) can be downloaded as part of: [tutorials/python](https://gitlab.syncad.com/hive/devportal/-/tree/master/tutorials/python) (or download just this tutorial: [devportal-master-tutorials-python-36_rcdemo.zip](https://gitlab.syncad.com/hive/devportal/-/archive/master/devportal-master.zip?path=tutorials/python/36_rcdemo)).
+
+### Statelessness
+
+First of all, a word about statelessness.  A great deal of effort has gone into carefully separating stateful from stateless computations.
+The reason for this is so that UI's and client libraries can execute stateless algorithms locally.  Local computation is always to be
+preferred to RPC-API for performance, stability, scalability and security.
+
+Unfortunately, client library support for RC algorithms is lacking.  This tutorial, and its accompanying script, are intended to
+provide guidance to UI and client library maintainers about how to add support for the RC system.
+
+# The RC demo script
+
+To get up and running, I (`theoretical`) have transcribed some key algorithms from C++ to Python.  For example, how many resources
+are consumed by a vote transaction?  The `rcdemo` script allows us to find out:
+
+```
+>>> from rcdemo import *
+>>> count = count_resources(vote_tx, vote_tx_size)
+>>> count["resource_state_bytes"]
+499232
+>>> print(json.dumps(count))
+{"resource_count": {"resource_history_bytes": 133, "resource_new_accounts": 0, "resource_market_bytes": 0, "resource_state_bytes": 499232, "resource_execution_time": 0}}
+```
+
+The `count_resources()` function is *stateless*.  That means all of the information needed to do the calculation is contained in the transaction itself.  It doesn't
+depend on what's happening on the blockchain, or what other users are doing.  [1] [2] [3]
+
+[1] Although it is possible that the calculation will change in future versions of `hived`, for example to correct the [bug](https://github.com/steemit/steem/issues/2972) where execution time is always reported as zero.
+
+[2] For convenience, some of the constants used in the calculation are exposed by the `size_info` member of `rc_api.get_resource_params()`.  Only a `hived` version upgrade can change any values returned by `rc_api.get_resource_params()`, so it is probably okay to query that API once, on startup or when first needed, and then cache the result forever.  Or even embed the result of `rc_api.get_resource_params()` in the source code of your library or application.
+
+[3] `rcdemo.py` requires you to also input the transaction size into `count_resources()`.  This is because `rcdemo.py` was created to be a standalone script, without a dependence on any particular client library.  If you are integrating `rcdemo.py` into a client library, you might consider using your library's serializer to calculate the transaction size automatically, so the caller of `count_resources()` doesn't have to specify it.
+
+### Resources
+
+Let's go into details on the different kinds of resources which are limited by the RC system.
+
+- `resource_history_bytes` : Number of bytes consumed by the transaction.
+- `resource_new_accounts` : Number of accounts created by the transaction.
+- `resource_market_bytes` : Number of bytes consumed by the transaction if it contains market operations.
+- `resource_state_bytes` : Number of bytes of chain state needed to support the transaction.
+- `resource_execution_time` : An estimate of how long the transaction will take to execute.  Zero for now due to [#2972](https://github.com/steemit/steem/issues/2972).
+
+The resources have different scales.  The resources use fixed-point arithmetic where one "tick" of the resource value is a "fractional" value of the resource.  Right now, the resource scales are scattered in different places.  The `count_resources()` result has the following scale:
+
+- `resource_history_bytes` :  One byte is equal to `1`.
+- `resource_new_accounts` :   One account is equal to `1`.
+- `resource_market_bytes` :   One byte is equal to `1`.
+- `resource_state_bytes` :    One byte which must be stored forever is equal to the `hived` compile-time constant `STATE_BYTES_SCALE`, which is `10000`.  Bytes which must be stored for a bounded amount of time are worth less than `10000`, depending on how long they need to be stored.  The specific constants used in various cases are specified in the `resource_params["size_info"]["resource_state_bytes"]` fields.
+- `resource_execution_time` : One nanosecond of CPU time is equal to `1`.  The values are based on benchmark measurements made on a machine similar to `hive.blog` servers.  Some rounding was performed, and a few operations' timings were adjusted to account for additional processing of the virtual operations they cause.
+
+### Resource pool levels
+
+Each resource has a global *pool* which is the number of resources remaining.  The pool code supports fractional resources, the denominator is represented by the `resource_unit` parameter.  So for example, since `resource_params["resource_params"]["resource_market_bytes"]["resource_dynamics_params"]["resource_unit"]` is `10`, a pool level of `15,000,000,000` actually represents `1,500,000,000` bytes.
+
+### Resource credits
+
+The RC cost of each resource depends on the following information:
+
+- How many resources are in the corresponding resource pool
+- The global RC regeneration rate, which may be calculated as `total_vesting_shares` / ([`HIVE_RC_REGEN_TIME`]({{ '/tutorials-recipes/understanding-configuration-values.html#HIVE_RC_REGEN_TIME' | relative_url }}) / [`HIVE_BLOCK_INTERVAL`]({{ '/tutorials-recipes/understanding-configuration-values.html#HIVE_BLOCK_INTERVAL' | relative_url }}))`
+- The price curve parameters in the corresponding `price_curve_params` object
+
+For convenience, `rcdemo.py` contains an `RCModel` class with all of this information in its fields.
+
+```
+>>> print(json.dumps(model.get_transaction_rc_cost( vote_tx, vote_tx_size )))
+{"usage": {"resource_count": {"resource_history_bytes": 133, "resource_new_accounts": 0, "resource_market_bytes": 0, "resource_state_bytes": 499232, "resource_execution_time": 0}}, "cost": {"resource_history_bytes": 42136181, "resource_new_accounts": 0, "resource_market_bytes": 0, "resource_state_bytes": 238436287, "resource_execution_time": 0}}
+>>> sum(model.get_transaction_rc_cost( vote_tx, vote_tx_size )["cost"].values())
+280572468
+```
+
+The `model` object created in `rcdemo.py` is an instance of `RCModel` which uses hardcoded values for its pool levels and global RC regeneration rate.  These values were taken from the live network and hardcoded in the `rcdemo.py` source code in late September 2018.  So the RC cost calculation provided out-of-the-box by `rcdemo.py` are approximately correct as of late September 2018, but will become inaccurate as the "live" values drift away from the hardcoded values.  When integrating the `rcdemo.py` code into an application, client library, or another situation where RPC access is feasible, you should understand how your code will query a `hived` RPC endpoint for current values.  (Some libraries will probably choose to do this RPC automagically, other libraries may want to leave this plumbing to user code.)
+
+### Transaction limits
+
+Suppose an account has 15 Hive Power.  How much can it vote?
+
+```
+>>> vote_cost = sum(model.get_transaction_rc_cost( vote_tx, vote_tx_size )["cost"].values())
+>>> vote_cost
+280572468
+>>> vote_cost * total_vesting_fund_hive / total_vesting_shares
+138.88697555075086
+```
+
+This is the amount of Hive Power (in satoshis) that would be needed by an account to transact once per 5 days ([`HIVE_RC_REGEN_TIME`]({{ '/tutorials-recipes/understanding-configuration-values.html#HIVE_RC_REGEN_TIME' | relative_url }})).
+Our 15 SP account has 15000 SP, so it would be able to do `15000 / 138`, or about `108`, such transactions per 5 days.
+
+You can regard the number `138` (or `0.138`) as the "cost" of a "standardized" vote transaction.  It plays an analogous role to a
+transaction fee in Bitcoin, but it is not exactly a fee.  Because the word "fee" implies giving up a permanent token with a limited,
+controlled emission rate.  It is the amount of SP which will allow a user an additional vote transaction every 5 days (but it might
+be slightly more or less, if your vote transactions use a slightly different amount of resources.)
+
+### Integrating the demo script
+
+The `rcdemo.py` script is a standalone Python script with no dependencies, no network access, and no transaction serializer.  It is a port
+of the algorithms, and a few example transactions for demo purposes.
+
+Eventually, client library maintainers should integrate `rcdemo.py` or equivalent functionality into each Hive client library.  Such integration
+depends on the idioms and conventions of each particular client library, for example:
+
+- A client library with a minimalist, "explicit is better than implicit" philosophy may simply rename `rcdemo`,
+delete the example code, add a tiny bit of glue code, and give it to clients largely as-is.
+- A client library with an automagic, "invisible RPC" philosophy might provide a transaction may make substantial modifications to `rcdemo`
+or expose an interface like `get_rc_cost(tx)` which would conveniently return the RC cost of a transaction.  Since the RC cost depends on
+chain state, this `get_rc_cost()` function would call `hived` RPC's (or use cached values) to get additional inputs needed by stateless
+algorithms, such as resource pools, `total_vesting_shares`, etc.
+- A client library which has a general policy of hard-coding constants from the `hived` C++ source code might distribute
+`rc_api.get_resource_parameters()` as part of its source code, as `rc_api.get_resource_parameters()` results can only change in a new
+version of `hived`.  (Perhaps this kind of thing is somehow automated in the library's build scripts.)
+- A client library whose maintainers don't want to be obligated to make new releases when values in `hived` change as part of a new release,
+might instead choose to call `rc_api.get_resource_parameters()`.  Adding extra performance and security overhead in exchange for the
+convenience of asking `hived` to report these values.
+- A client library with its own `Transaction` class might choose to implement class methods instead of standalone functions.
+- A client library in JavaScript, Ruby or Go might transcribe the algorithms in `rcdemo.py` into a different language.
+
+As you can see, integrating support for the RC system into a Hive client library involves a number of architectural and
+technical decisions.
+
+### To Run the tutorial
+
+1. [review dev requirements](getting_started.html)
+1. `git clone https://gitlab.syncad.com/hive/devportal.git`
+1. `cd devportal/tutorials/python/36_rcdemo`
+1. `python rcdemo.py`

_tutorials-recipes/calculate_rc_recipe.md

@@ -7,7 +7,7 @@ layout: full
 canonical_url: calculate_rc_recipe.html
 ---
 
-Since HF20 a Resource Credit (RC) system has been implemented to manage the number of transactions (comments, votes, transfers, etc) you can execute on the blockchain at any given time. This recipe will look at how to calculate your current RC and also what the current RC cost is for a given transaction. This recipe is far more 'basics oriented' than most. For a more in-depth description of how RC's work consume [this excellent RC demo](https://github.com/steemit/rcdemo) created by Hive's Blockchain Team.
+Since HF20 a Resource Credit (RC) system has been implemented to manage the number of transactions (comments, votes, transfers, etc) you can execute on the blockchain at any given time. This recipe will look at how to calculate your current RC and also what the current RC cost is for a given transaction. This recipe is far more 'basics oriented' than most. For a more in-depth description of how RC's work consume [this excellent RC demo]({{ '/tutorials-python/rcdemo.html' | relative_url }}) created by Hive's Blockchain Team.
 
 ## Intro
 
@@ -86,4 +86,4 @@ Additional info can also be found in [this article by hive user @holger80](https
 
 ## Allocation of RC to blockchain resources
 
-An in depth look at how RC's are assigned to each of the three resources (CPU megacycles/state memory/history size) can be found in Hive's wiki articles for [RC Bandwidth System](https://github.com/steemit/steem/wiki/RC-Bandwidth-System) and [Parameters](https://github.com/steemit/steem/wiki/RC-Bandwidth-Parameters)
+An in depth look at how RC's are assigned to each of the three resources (CPU megacycles/state memory/history size) can be found in Hive's wiki articles for [RC Bandwidth System]({{ '/tutorials-recipes/rc-bandwidth-system' | relative_url }}) and [Parameters]({{ '/tutorials-recipes/rc-bandwidth-parameters' | relative_url }}).

_tutorials-recipes/rc-bandwidth-parameters.md

+---
+title: RC Bandwidth Parameters
+position: 1
+description: Analyze the dynamics of the resource budget pool.
+exclude: true
+layout: full
+canonical_url: rc-bandwidth-parameters.html
+---
+
+### Parameters
+
+Each pool has its own values of the following constants:
+
+- `budget_time` : Time interval for budgeting.
+- `budget` : How much the resource increases linearly over `budget_time`.
+- `half_life` : Time until the resource would decay to 50% of its initial size with no load or budget.
+- `drain_time` : Time until user RC would decrease to 0% of its capacity in the hypothetical world where all users have saved maximum RC, use all RC and RC regen on this resource, and the resource pool is empty (so the price is at its maximum value and there is budget but no drain).
+- `inelasticity_threshold` : Percentage of the equilibrium value that serves as a boundary between elastic and inelastic prices.
+
+### Computed values
+
+Let us analyze the dynamics of the resource budget pool as outlined [here]({{ '/tutorials-recipes/rc-bandwidth-system' | relative_url }}) with an eye to establishing values of the parameters.
+
+### Setting p_0
+
+Let `p(x)` be the RC cost curve, `p(x) = A / (B + x)`.
+
+Let's denote as `p_0` the value of `p(0)`, which is of course `p_0 = A / B`.  In other words,`p_0` is the price when there are no resources in the pool.  Let's set up a worst-case scenario:
+
+- Suppose there are no resources in the pool
+- Suppose all users have maximum RC reserves
+- Suppose all users spend all RC reserves and regen on this resource
+- Suppose all users run out of resources at time `t_drain`
+
+Then we can solve for `p_0` in terms of `t_drain` and the budget:
+
+```
+global_rc_capacity + global_rc_regen * drain_time = p_0 * budget * drain_time
+p_0 = (global_rc_capacity + global_rc_regen * drain_time) / (budget * drain_time)
+```
+
+Suppose `drain_time = 1 hour`.  Then the interpretation of this equation is that, in an hour, users in the aggregate can only muster RC equal to their RC capacity plus an hour's worth of RC regen, call this much RC `rc_1h`.  If we say this RC "should" be *just enough* to pay for `res_1h`, 1 hour's worth of a single resource (say, an hour's worth of history bytes).  That implies a resource price of `rc_1h / res_1h`, which is the price we set the value of `p_0` to.
+
+If we substitute for `global_rc_capacity` and simplify, we get an alternative expression for `p_0` which may be useful for some analysis:
+
+```
+p_0 = (global_rc_capacity + global_rc_regen * drain_time) / (budget * drain_time)
+    = (global_rc_regen * rc_regen_time + global_rc_regen * drain_time) / (budget * drain_time)
+    = (global_rc_regen / budget) * (1 + rc_regen_time / drain_time)
+```
+
+The first multiplicative term, `global_rc_regen / budget`, is the *balanced budget price* which would cause spending to match outflow; let us say `p_bb = global_rc_regen / budget`.  This is the equilibrium price if users have no reserves and spend all RC's on this resource.  This is multiplied by a multiplier which represents the temporary level the price can rise to, which is ultimately unsustainable due to finite reserves.
+
+### Setting B
+
+What about the constant `B`?  When `x` is much smaller than `B`:
+
+- The value of `p(x)` is approximately `A / B`
+- Price `p(x)` is (almost) *inelastic* with respect to pool size `x`
+- A small percentage change in `x` doesn't change `p(x)` much)
+
+When `x` is much larger than `B` the opposite is true:
+
+- The value of `p(x)` is approximately `A / x`
+- Price `p(x)` is (almost) *elastic* with respect to pool size `x`
+- A small percentage change in `x` causes an almost equal in magnitude percentage change in `p(x)`)
+
+So while `p(x)` is never fully elastic or inelastic, you may think of `B` as being near the "midpoint" of the "transition" from almost perfectly inelastic (for small `x`) to almost perfectly elastic (for large `x`).  This makes sense, since a perfectly elastic price is desirable (when possible) but unstable when the pool goes to zero (a halving of the pool results in the doubling of the price, meaning the price "doubles infinite times" as the pool goes toward zero, "halving infinite times").
+
+Where should this transition occur?  The simple answer is a small fraction of the pool's equilibrium size at no load, say `1%`, or perhaps `1/128`.
+
+#### Inelasticity proof
+
+A more rigorous way to say that `p(x)` is (almost) inelastic is that, for any small value `∊`, there exists some `x` small enough such that, for any small value `δ`, we have `f(x) / f((1+δ)*x) < 1+δ*∊`.  (This can be interpreted as saying the percentage change in `f(x)` in response to a percentage change in `x` dies out as `x →0⁺`.)
+
+To prove this, set `x = B*∊`.  Then:
+
+```
+f(x) / f((1+δ)*x) = (A / (B+x)) / (A / (B+(1+δ)*x))
+   = (B+x+δ*x) / (B+x)
+   = 1 + δ*x / (B+x)
+   < 1 + δ*x / B
+   = 1 + δ*∊
+```
+
+#### Elasticity proof
+
+TODO:  State/prove similar condition for elasticity case.
+
+### Decay rate
+
+Let `τ` be the [time constant](https://en.wikipedia.org/wiki/Time_constant) for the decay rate.  (The decay constant is [proportional to](https://en.wikipedia.org/wiki/Time_constant#Exponential_decay) the half-life.)  The value of `τ` is essentially how much time it takes for the system to "forget" about past non-usage, and places a "soft limit" on the possibility of a very long burst of very high activity being allowed as a result of "saving up" months or years worth of inactivity.  The value of `τ` is per-resource, and we've determined that resources should have short-term and long-term versions with different `τ` values.  For short-term `τ` value, I recommend 1 hour.  For the long-term `τ` value, I recommend 15 days.
+
+If `Δt` time passes, then the exponential decay step of updating the pool should be `pool -= Δt * pool / τ`.
+
+The no-load equilibrium pool size then occurs when `Δt * pool_eq / τ = Δt * budget_per_sec`, or `pool_eq = τ * budget_per_sec`.
+
+### Discount term
+
+Above, we set implicitly the largest (zero-stockpile) price `p_0` based on `drain_time`.  It may also be useful to set the smallest possible price `p_eq` to something other than the value implied by the above equations.  To do this, we can simply shift the curve downward by a factor of `D`, and scale the curve up so `p(0) = p_0`.
+
+To elaborate:
+
+First, define `p(x)` with discount term `D` such that
+
+```
+p(x) = (A / (B + x)) - D
+```
+
+Then set `B = inelasticity_threshold * pool_eq`.  For `A` and `D` we have two equations in two unknowns:
+
+```
+p_0 = (A / B) - D
+p_eq = (A / (B + pool_eq)) - D
+```
+
+Solve both equations for `A`, set the two expressions equal to each other, and solve for `D`:
+
+```
+A = B*(p_0 + D) = (p_eq+D)*(B+pool_eq)
+-> B*p_0 + B*D = B*p_eq + B*D + p_eq*pool_eq + D*pool_eq
+-> B*(p_0 - p_eq) - p_eq*pool_eq = D*pool_eq
+-> (B / pool_eq)*(p_0 - p_eq) - p_eq = D
+```

_tutorials-recipes/rc-bandwidth-system.md

+---
+title: RC Bandwidth System
+position: 1
+description: All about the RC bandwidth system, the complete rewrite of the bandwidth system.
+exclude: true
+layout: full
+canonical_url: rc-bandwidth-system.html
+---
+See also [RC Bandwidth Parameters]({{ '/tutorials-recipes/rc-bandwidth-parameters' | relative_url }})
+
+### Introduction
+
+The *RC bandwidth system* is a complete rewrite of the bandwidth system.  Its goals include:
+
+- Enable simple, effective UI feedback to users about bandwidth usage and remaining bandwidth
+- Simplify the mental model of what buying additional SP gives users
+- Reduce or eliminate unstable feedback in current bandwidth system
+
+### History
+
+HF20:  Initial implementation.
+
+### Resource credits
+
+Each account has a manabar called "resource credits."  Resource credits have the following characteristics:
+
+- RC's are attached to a particular account and cannot be transferred
+- An account's maximum RC is proportional to its VESTS
+- Transacting uses RC
+- Transactions which would cause a negative RC balance are blocked
+- RC regenerates over time
+
+### Resources
+
+How many RC's are required for a transaction?  Statelessly compute, for each transaction, how many of each *resource* it takes.  Resources include:
+
+- CPU (mega)cycles
+- State memory
+- History size
+
+Then each resource has an exchange rate.  If CPU cycles cost 5 RC / megacycle, state memory costs 8 RC / byte, and history size costs 4 RC / byte, a transaction which takes 2 megacycles, creates 50 bytes of state, and has a 150 byte transaction size will cost `2*5 + 50*8 + 150*4 = 1010 RC`.
+
+### Resource budget pools
+
+A *resource budget pool* for each resource type will be established.  The resource budget pool will have a per-block linear increase, a per-block percentage decrease, and a per-transaction decrease.
+
+For example:
+
+- Suppose the per-block resource budget is 2500 megacycles, 5000 state bytes, and 25,000 history bytes.
+- Suppose the per-block percentage decrease is 0.02%
+- Suppose the pool currently contains 12,000,000 megacycles, 20,000,000 state bytes, and 80,000,000 history bytes.
+- Suppose the above transaction (consuming 2 megacycles, 50 state bytes, and 150 history bytes) is the only transaction which occurs.
+
+We can compute the new values as follows:
+
+```
+// when transaction is processed
+bp.megacycles -= 2;
+bp.state_bytes -= 50;
+bp.history_bytes -= 150;
+
+// per block additive
+bp.megacycles += 2500;
+bp.state_bytes += 5000;
+bp.history_bytes += 25000;
+
+// per block multiplicative
+// of course this would be implemented as integer arithmetic
+bp.megacycles *= 0.9998;
+bp.state_bytes *= 0.9998;
+bp.history_bytes *= 0.9998;
+```
+
+### Resource pricing
+
+The resource budget pool can be viewed as the blockchain's "stockpile" of each resource, which it "sells" for RC.  The price of each resource is based on the current level of the stockpile.  Exactly how the price
+is determined isn't very important, as long as it is a decreasing, smooth curve.
+
+The specific cost curve is:
+
+```
+p(x) = A / (B + x)
+```
+
+where `A` and `B` are parameters which may be set to different values for different resources.