![\"\"](\"http:\/\/rankexploits.com\/musings\/wp-content\/uploads\/2012\/10\/BareProjection.png\" "\\"BareProjection\\"")
<\/a>\nThe dashed lines represent the standard error in the mean quatloo winnings per bet.<\/p>\n
To the right of the red line, is my prediction<\/i> of number of quatloos I will earn in the future if I don’t come up with some nifty system to improve my betting performance.<\/p>\n
As you can see, in this example, I am predicting the future “Quatloo anomaly” will be zero. The uncertainty intervals are the uncertainty in the mean over the baseline<\/i> (left) and the uncertainty in the mean over the forecast period<\/i> which has not yet occurred. <\/p>\n
We’ll now pretend I concoct a ‘system’ that I think will improve my ability to win Quatloos. It will take practice, so I think my winnings will tend to increase at a rate of Q(i)=mi where ‘m’ is the trend. Of course, my null hypothesis remains m=0, and I don’t really know how large ‘m’ is. I merely hope it’s positive.<\/p>\n
I’m going to use my system for 120 plays. At the end of that time, I want to test the null hypothesis that <\/p>\n
- \n
- I will earn 0 Quatloos each month in the future period. That is E[Q]=0<\/li>\n
- The trend m is zero. That is m=0. <\/li>\n<\/ol>\n
I can test either using an appropriate ‘t’ test. Each would involve finding a ‘t’ value for the data in the upcoming period and comparing that to a student t-distribution. I can call the two t value tmean<\/sub> and ttrend<\/sub>. <\/p>\n
I now have two possible tests. Strictly speaking, I am testing two separate hypotheses both of which are related since the amount to “the system doesn’t work”. So, I really only want 1 test. <\/p>\n
If I’m limited to one test or the other, I’d really like is to pick the more statistically powerful<\/I> of the two tests. That is: When applied at the same statistical significance (that is a pre-defined type I error rate) , I’d like the test that results in the greatest statistical power (which is the same as the lowest type II error rate.) <\/p>\n
Oddly enough, I also know that I can show the errors<\/i> in both tests are independent. That is, if the null hypotheses are true and both m=0 and E[Q]=0, then the slight deviations in the actual ‘m’ and average quatloo winnings I will observe during the upcoming periods are statistically independent. (This was discussed here here<\/a>.)
\nSo the fact that there are two methods and the errors in the two methods are statistically independent leads me to an interesting situation. <\/p>\nThere should be some method of combining the tests to create a pooled test that is more powerful than either alone. That is, there should be a method of creating a pooled <\/p>\n
tpooled<\/sub>=wtrend<\/sub>ttrend<\/sub>+wmean<\/sub>tmean<\/sub><\/p>\n
with weights selected such that the standard deviation of the pooled ts is 1 and the statistical power of the test is maximized. (Note that the standard deviations of ttrend<\/sub> and tmean<\/sub> are one by definition. That’s a requirement for t values.) <\/p>\n
So, I concocted the following rule, which seems to work:<\/p>\n
- \n
- Define a effect size for an alternate hypothesis: that is the hypothesis that might be true. This can be any value other than zero since the magnitude cancels: I pick me<\/sub>=0.1 quatloo\/decade. If that hypothesis is true, I find I will win Q=m*120 months quatloos. <\/li>\n
- Based on data available prior to the forecast period, estimate the ‘te,m<\/sub>‘ and te,mean<\/sub> that would exist if<\/I> this size effect is observed during the upcoming prediction period. To estimate these I need to estimate the variability of 120 month trends given data prior to the forecast periods, and I need to estimate the variability in the difference between the mean in the prediction and baseline period. (Those are provided in the figure and are based on the noise prior to the forecast.) <\/li>\n
- Compute relative weights for each test variable ‘i’ as as wi<\/sub>= te,i<\/sub>^2\/ sum(te,j<\/sub>^2). (Notice the size effect cancels out when the t’s are normalized.)<\/li>\n
- Collect data during the sample period. Afterwards, compute the t value based on the sample data (using normal methods.) Then create a pooled t using tpooled<\/sub>=wtrend<\/sub>ts,trend<\/sub>+wmean<\/sub>ts,mean<\/sub>. Here the ‘s’ denotes sample data. <\/li>\n<\/ol>\n
I’ve done a number of test and this weighting does appear to always<\/I> result in a more powerful test than either<\/i> the trend or mean tests alone provided the weights are based on the size effect during the forecast period<\/i> and the individual t-tests properly account for all errors. For the current case, I collected data on Quatloo winnings and the results are as below:<\/p>\n
![\"\"](\"http:\/\/rankexploits.com\/musings\/wp-content\/uploads\/2012\/10\/TestPooled-500x500.png\" "\\"TestPooled\\"")
<\/a><\/p>\nThe actually important part of the graph is the information in the lower left. What I want you to notice is the based on synthetic data I created, ‘t’ for the mean anomaly test was 18.44. The t for the trend test was 12.57. I weighted the two t’s using relative weights of (0.66,0.34)\/sqrt(0.66^2+0.34^2). this resulted in a tpooled<\/sub> of 22.13. The significant feature is 22 is larger than either 18.4 or 12.6. This isn’t important for the synthetic data I show which has so much power it’s ridiculous. But the feature is very important if we have just barely enough data<\/I> to generate sufficient power to hope for “fail to rejects”. <\/p>\n
Synthetic tests using 10000 repeat trials indicate that this pooled t has the proper characteristics and is– on average– larger than either of the two other ‘t’. That means the resulting test has higher power than either the test comparing trends or the test for comparing anomalies alone. Consequently, it is preferred. <\/p>\n
So, this is sort of a “goldilocks” test! <\/p>\n
I know this is sketchy. I’ll be happy to answer questions. But this post is mostly so I have a place holder to remember what I did. (I uploaded the really badly organized script with embarrassing incomprehensible notes and a bunch of side tests. And… yes… this can be applied to model tests. I haven’t done it yet because I hadn’t verified I knew how to weight the two tests. Did model tests with weights of (0.5, 0.5 )\/sqrt(0.5). Now I need to repeat that with the correct weights. \ud83d\ude42 <\/p>\n
==========
\nNote:<\/b>:I’ve discussed this before and Paul_K was dubious.<\/a> So, this result is something of a turn about because while the first test I though up way back then did not seem to optimize this one does. I didn’t upload that script, so I’m not sure if I applied the test entirely properly in that post. I may have forgotten to account for the ‘running start’ aspect of testing the means making the test of the means appear more powerful than it ought to.)<\/p>\n","protected":false},"excerpt":{"rendered":"The purpose of this post is to discuss a method of testing a projection that I think has greater power than testing trends alone. (Yes, Paul_K. We discussed this before. I now think it can be done!! \ud83d\ude42 ) To discuss this, I’m going to discuss how I might test a hypothetical prediction. Suppose I’ve … Continue reading Pooled Method of Testing a Projection<\/span>
- Based on data available prior to the forecast period, estimate the ‘te,m<\/sub>‘ and te,mean<\/sub> that would exist if<\/I> this size effect is observed during the upcoming prediction period. To estimate these I need to estimate the variability of 120 month trends given data prior to the forecast periods, and I need to estimate the variability in the difference between the mean in the prediction and baseline period. (Those are provided in the figure and are based on the noise prior to the forecast.) <\/li>\n
- Define a effect size for an alternate hypothesis: that is the hypothesis that might be true. This can be any value other than zero since the magnitude cancels: I pick me<\/sub>=0.1 quatloo\/decade. If that hypothesis is true, I find I will win Q=m*120 months quatloos. <\/li>\n