... | @@ -10,7 +10,7 @@ We've already discussed the fact that Bayes pulls in favor of certainty. If we c |
... | @@ -10,7 +10,7 @@ We've already discussed the fact that Bayes pulls in favor of certainty. If we c |
|
|
|
|
|
See what's happening here? The combined opinion gets pulled very strongly toward the 1st opinion the more certain the first opinion becomes. It seems a little strange that what seems like small differences in the first opinion should have such a pronounced effect.
|
|
See what's happening here? The combined opinion gets pulled very strongly toward the 1st opinion the more certain the first opinion becomes. It seems a little strange that what seems like small differences in the first opinion should have such a pronounced effect.
|
|
|
|
|
|
## An intuitive explanation
|
|
## A more intuitive explanation
|
|
|
|
|
|
The Bayes eqn will confirm the truth of this but this situation begs for a more intuitive explanation. Although 99% and 99.9% don't seem like much of a difference, they represent a huge difference in sample sizes. To be able to say 99%, one must perform at least 100 experiments, 99 of which succeeded and 1 which failed. To be able to say 99.9%, one must perform at least 1000 experiments where 999 succeeded and 1 failed. And so on.
|
|
The Bayes eqn will confirm the truth of this but this situation begs for a more intuitive explanation. Although 99% and 99.9% don't seem like much of a difference, they represent a huge difference in sample sizes. To be able to say 99%, one must perform at least 100 experiments, 99 of which succeeded and 1 which failed. To be able to say 99.9%, one must perform at least 1000 experiments where 999 succeeded and 1 failed. And so on.
|
|
|
|
|
... | @@ -62,7 +62,7 @@ Let's juxtapose the two relevant calculations: |
... | @@ -62,7 +62,7 @@ Let's juxtapose the two relevant calculations: |
|
|
|
|
|
We see here that the numerator increases by a factor of 10, corresponding to the 10 times increase in sample size, and representing the number of people who test positive twice who are actually sick. The denominator is composed of this same number plus a constant, the number of people who falsely test positive twice. This constant remains so because as the test becomes more accurate, ever fewer results are bad (in percentage terms). For every 9 added to the decimal place, it becomes about 10 times harder to have a bad result. This is the basic reason why certainty pulls so hard in its favor.
|
|
We see here that the numerator increases by a factor of 10, corresponding to the 10 times increase in sample size, and representing the number of people who test positive twice who are actually sick. The denominator is composed of this same number plus a constant, the number of people who falsely test positive twice. This constant remains so because as the test becomes more accurate, ever fewer results are bad (in percentage terms). For every 9 added to the decimal place, it becomes about 10 times harder to have a bad result. This is the basic reason why certainty pulls so hard in its favor.
|
|
|
|
|
|
## Counteracting Certainty and OOM
|
|
## Counteracting certainty and OOM
|
|
|
|
|
|
Indeed, to counteract the effect of certainty we would need an equal and opposite level of certainty for the opposite opinion. If 99% represents our probability of a True result, then a 1% opinion for True is it's opposite (99% for False). When combined these two yield, via Bayes,
|
|
Indeed, to counteract the effect of certainty we would need an equal and opposite level of certainty for the opposite opinion. If 99% represents our probability of a True result, then a 1% opinion for True is it's opposite (99% for False). When combined these two yield, via Bayes,
|
|
|
|
|
... | @@ -76,7 +76,7 @@ To an extent we can also understand this in terms of decimal places. The more de |
... | @@ -76,7 +76,7 @@ To an extent we can also understand this in terms of decimal places. The more de |
|
|
|
|
|
This only works at the certainty (99%, 99.9%, 99.99%, etc.) and uncertainty end (1%, 0.1%, 0.01%, etc.) of the probability spectrum, which is why the OOM view is the more accurate way to look at it. If someone says they are 10.001% certain of something they must have done at least 100,000 experiments to confirm that (10001 succeeded, 89999 failed). But in Bayes, this is not very different than someone saying 10% where only 10 experiments were conducted for confirmation (9 succeeded, 1 failed). When combined with a 99.999%, which has the same number of decimal places, the result will be 99.991%, very close to 99.999%. The "decimal place heuristic" clearly only works at the ends of the spectrum (where the smaller opinions exhibit a difference in OOM).
|
|
This only works at the certainty (99%, 99.9%, 99.99%, etc.) and uncertainty end (1%, 0.1%, 0.01%, etc.) of the probability spectrum, which is why the OOM view is the more accurate way to look at it. If someone says they are 10.001% certain of something they must have done at least 100,000 experiments to confirm that (10001 succeeded, 89999 failed). But in Bayes, this is not very different than someone saying 10% where only 10 experiments were conducted for confirmation (9 succeeded, 1 failed). When combined with a 99.999%, which has the same number of decimal places, the result will be 99.991%, very close to 99.999%. The "decimal place heuristic" clearly only works at the ends of the spectrum (where the smaller opinions exhibit a difference in OOM).
|
|
|
|
|
|
## Using Decimal Places to Combine and Evaluate Evidence
|
|
## Using decimal places to combine and evaluate experimental evidence
|
|
|
|
|
|
Nevertheless, this idea asks us to consider what happens when two opinions differ in the evidentiary weight behind them. Clearly an experiment with 1000 tests is not equivalent to one with 10. The Bayes equation, however, has no mechanism to judge the quality of the probabilities inserted into it except when those probabilities clearly differ in terms of their OOM.
|
|
Nevertheless, this idea asks us to consider what happens when two opinions differ in the evidentiary weight behind them. Clearly an experiment with 1000 tests is not equivalent to one with 10. The Bayes equation, however, has no mechanism to judge the quality of the probabilities inserted into it except when those probabilities clearly differ in terms of their OOM.
|
|
|
|
|
... | @@ -98,7 +98,7 @@ Given how close the result is to $`P_1`$ we might just consider skipping the cal |
... | @@ -98,7 +98,7 @@ Given how close the result is to $`P_1`$ we might just consider skipping the cal |
|
|
|
|
|
This idea is also useful for evaluating the relative merits of two experiments without necessarily combining them via Bayes. If two scientists disagree and one has significantly more experimental evidence, then we could use the above idea to perform a weighted average of their opinions.
|
|
This idea is also useful for evaluating the relative merits of two experiments without necessarily combining them via Bayes. If two scientists disagree and one has significantly more experimental evidence, then we could use the above idea to perform a weighted average of their opinions.
|
|
|
|
|
|
## Reporting Results Correctly using Decimal Places
|
|
## Reporting results correctly using decimal places
|
|
|
|
|
|
This discussion assumes that probabilities will be reported to the correct number of decimal places. If we do 11 experiments and 9 succeed we could claim
|
|
This discussion assumes that probabilities will be reported to the correct number of decimal places. If we do 11 experiments and 9 succeed we could claim
|
|
|
|
|
... | @@ -106,4 +106,4 @@ P = 9/11 = 0.818181818181 |
... | @@ -106,4 +106,4 @@ P = 9/11 = 0.818181818181 |
|
|
|
|
|
but this would be to represent the result with far too many decimal places, implying that many more experiments had been done to confirm it. The correct decimal representation is 0.8, implying that about 10 experiments have taken place, leading to one decimal place.
|
|
but this would be to represent the result with far too many decimal places, implying that many more experiments had been done to confirm it. The correct decimal representation is 0.8, implying that about 10 experiments have taken place, leading to one decimal place.
|
|
|
|
|
|
Therefore we need to watch when reports are made with a suspiciously large number of decimal places. Sources should be encouraged to report their experimental results as fractions where we can see openly in the denominator how many experiments were done. |
|
Therefore we need to watch when reports are made with a suspiciously large number of decimal places. Sources should be encouraged to report their experimental results as fractions where we can see openly in the denominator how many experiments were conducted. |
|
\ No newline at end of file |
|
\ No newline at end of file |