Let's talk statistics

[Dragavon]

This thread was split off from another thread (http://forum.thesettlersonline.com/t...ster-Architect)

A sample of around 50 should give a statistically significant result. not an accurate percentage, but enough to rule 16% drop rate as either true or false.

Example: opening 50 boxes without a single xmas tree - probability of 0.84^50= 0.0002 - thus statistically the 16% drop rate has been shown false with less than 0.1% error margin!

You would need thousands only to find the exact percentage (like if it's 16% or 17%). Hundreds will do fine in determining a reliable interval (like 15-20%) and even less when only testing for a certain percentage value (as above).

With regard to Peajay's post, 5 tries is indeed too few to rule out luck. Especially when there are hundreds if not thousands of players attempting this. ONE of them is bound to have exceptionally bad luck. If you throw heads and tails long enough, you are bound to reach a long streak of only heads or only tails at some point. The coin's odds of landing heads or tails did not magically change - random outcomes always include variation. If you throw a die 6 times, it's more likely that you get some number twice rather than getting exactly 1,2,3,4,5,6 - one of each.

You are totally wrong. 50 samples is not even close to getting a useful result. If you think it is, then you do not understand statistics at all.

[Dorotheus]

Come back when you when you have several days records at a sampling rate of 50 a second.

[King-Fero]

You are totally wrong. 50 samples is not even close to getting a useful result. If you think it is, then you do not understand statistics at all.

If you claim to understand statistics, then it surprises me that you have been able to avoid the fact that the large majority of statistical tests give perfectly reliable results with sample sizes of n=30 and up. Obviously this does not mean that if there are 9 christmas trees in a pool of 50 architect boxes, the likelihood could be accurately estimated to be EXACTLY 18%. but 9 out of 50 is a reliable indicator that the chance is not, say, less than 10% or above 30%. This was already made clear in my previous post should you have read it.

A "statistic" is pooled data on an entire set. Let's say that BlueByte checks player data for Server N, where there are 6000 players, and can tell that the average player level is 33. A "sample" is a small portion of the players drawn randomly, like 100. Taking the average of these 100 players might be 34,2. This will not be 100% accurate with the true average, but using a statistical test will tell you whereabouts the true average is. This is referred to as the confidence interval. A larger sample will of course provide you with a more narrow confidence interval.

Knowing that the average player level is between 30-36, or 31-35, can be quite trivial. What is considered a "useful result", to put it in your own words, depends on what the information is needed for. Regarding the items contained in the master architect, knowing that the chances are around 15% should be useful enough. Whether the actual chance is 14% or 16.6% is quite irrelevant unless someone is planning to buy hundreds of euros worth of them. Those purchasing a few, some tens, or even a hundred of them, will be more affected by random variation.

I don't know the policy on posting non-game related links to the TSO forum so I dare not link you directly to greater enlightenment. I encourage you to use a common search engine with key words such as 'small sample' + 'statistical tests'.

Here's a few quotes such a search might lead you to:

"Some people think that if you have a small sample size you can't use statistics. Put simply, this is wrong, but it's a common misconception"

"Just because you don't have a large sample size doesn't mean you cannot use statistics. Again, the key limitation is that you are limited to detecting large differences between designs or measures."

[Dragavon]

If you claim to understand statistics, then it surprises me that you have been able to avoid the fact that the large majority of statistical tests give perfectly reliable results with sample sizes of n=30 and up. Obviously this does not mean that if there are 9 christmas trees in a pool of 50 architect boxes, the likelihood could be accurately estimated to be EXACTLY 18%. but 9 out of 50 is a reliable indicator that the chance is not, say, less than 10% or above 30%. This was already made clear in my previous post should you have read it."

If you know anything about statistics you would understand that with a sample of 50 Master Architect boxes there is a good chance than one or several of the possible results will not show up at all. I stand by my previous statements

[King-Fero]

If you know anything about statistics you would understand that with a sample of 50 Master Architect boxes there is a good chance than one or several of the possible results will not show up at all. I stand by my previous statements

If one of the possible items does not show up at all (like the improved store house), then it's an indication that it's chances of appearing are low. An example like this does not contradict with anything I've written so far as I'm not claiming it to be proof that the chance would be zero. The confidence interval would cover probabilities close to zero. In a sample of 50 the more likely items will be abundant, whereas the rarer items will be scarce if not nonexistent. This is not only statistically true, it is also common sense.

There is nothing wrong with my understanding of statistics nor statistical analysis. I don't want to hijack this thread for a pointless debate that is beyond it's original point, but I also feel difficulty letting go of it while you remain so far off the target. Your original claim was this:

"You need several thousand samples before you get a statistically significant result."

If that is a statement you wish to remain to stand by despite all the points I have made to support the contrary, then this conversation truly has gone nowhere. But I challenge you to find someone who you consider that does understand statistics (yourself excluded), and run this by them before you continue with your false arguments.

[Fexno]

Hey King-Fero and Dragavon,

I moved your posts to a separate thread, since your discussion about wether or not something is statistically significant was becoming more and more off-topic.
I did leave the first two posts in the original thread so you still gave your 2 cents there.

For those who want to read how this discussion started please read from here: http://forum.thesettlersonline.com/t...rchitect/page2

Cheers,
Fexno

[DrinkMoreMilk]

Tree points (ho ho ho):

1) Of course, it doesn't matter how many tests we do - it's not possible to find the exact percentage. Unless we were to stumble upon BB's source code.

2) If the probability of getting a tree is 16%, the probability of 5 trees in a row is 0,01% - quite unlikely.

3) Say we think the probability is 16%, but we tested it and got 17 trees out of 50 tests. With 17/50, a 99% confidence interval for the probability is between 18% and 53%. We would have highly significant evidence to doubt that the true probability was 16%. So yes, we might be able to draw useful conclusions from 50 tests, as long as we aren't talking about tiny probabilities.

[King-Fero]

On another thread corona88 was friendly enough to share findings from 130 Master Architects:

24 pirate residence 18.4%
27 raving rabbit 20.7%
20 christmas tree 15.3%
20 frozen manor 15.3%
18 arctic iron mine 13.8%
2 witch tower 1.5%
13 pavilion 10%
4 improved storehouse 3%
1 angel monument 0.76%
1 dark castle 0.76%
0 white castle 0%

Now, even if we were to open a thousand more MA, the probabilities are likely to stay within 2%-3% points (either up or down) from above values. If that can't be considered a useful result, then so be it. For me this sample of 130 MA is evidence enough of the approximate likelihoods of each item, and so it would be for most statistical analysts.

[fishslice]

If it helps here's a sample for you to analyse to your hearts content. All based upon a single player, on live server purchasing in groups of 50 - 100 MA boxes spread over 3 days at random times.....


Pirate Residence, 96 16.0%
Raving Rabbid Statue, 84 14.0%
Arctic Iron Mine, 91 15.2%
Frozen Mansion, 110 18.3%
Witch Tower, 9 1.5%
Pavillion, 83 13.8%
White (Sir Robin's) Castle, 7 1.2%
Christmas Tree, 98 16.3%
Dark Castle, 8 1.3%
Improved Storehouse, 8 1.3%
Angel Monument 6 1.0%
Total boxes 600

[Dorotheus]

I will return to this discussion after proof is offered that 1 sample is in fact a sample of 1.