{"id":15735,"date":"2011-06-20T15:58:49","date_gmt":"2011-06-20T21:58:49","guid":{"rendered":"http:\/\/rankexploits.com\/musings\/?p=15735"},"modified":"2011-06-20T15:58:49","modified_gmt":"2011-06-20T21:58:49","slug":"whats-uncorrelated-with-what-for-paulk","status":"publish","type":"post","link":"https:\/\/rankexploits.com\/musings\/2011\/whats-uncorrelated-with-what-for-paulk\/","title":{"rendered":"What’s uncorrelated with what? For PaulK"},"content":{"rendered":"

This post is mostly for Paul_K; he and I discussed on of my claims in my previous article discussing the NOAA anomaly. <\/a> Specifically, I claimed to compute two possible test variables from linear fit to observed data which in the current post I will call d*a<\/sub> and d*m<\/sub> and said the errors in these tests parameters were uncorrelated. I probably worded it in a confusing way, and I’d like to clarify for Paul_K. After clarification, it may turn out Paul_K has a suggestion, but for the time being, we seem to be talking at cross purposes. To do so, I’m going to discuss some observations based on trendless synthetic data.<\/p>\n

Suppose I generate a series of 137 months of Gaussian white noise with standard deviation 1 and mean zero, and call that “Yi<\/sub>“. I can now compute the mean of Yi<\/sub> in the normal way. We know the true<\/I> mean of this data should be 0. But instead, we will get some sample mean value Ybar<\/sub>, which we recognize as an error<\/i>. That is: it’s different from 0, the known correct value if we repeat the experiment over and over and over and take an average over the ensemble of all possible experiments. Let’s call this \u00ce\u00b5Y<\/sub>.<\/p>\n

Next, I can also compute the best fit trend through <\/p>\n

Yi<\/sub>=m i + b + ui<\/sub><\/p>\n

where the ‘i’ values are (1,2,3,…. 137), m is the best fit trend, and b is the intercept at i=0, and ui<\/sub> are the residuals. The best fit values of m and b are those which minimize the sum of the squares of the 137 residuals. <\/p>\n

Because I’ve generated the Yi<\/sub> as gaussian white noise, we know the true<\/I> value of m for the process is m=0. Nevertheless, when I generate 137 data points, I’ll nearly always get a non-zero value for the trend based on the sample, m. The difference between the sample value of the trend, m, and the true trend is the error in the trend. Let’s call this \u00ce\u00b5m<\/sub>.<\/p>\n

My claim is that the errors \u00ce\u00b5Y<\/sub> and \u00ce\u00b5m<\/sub> are uncorrelated. I also make a separate claim that if we do the least squares fit to the <\/p>\n

Yi<\/sub>=m (i – imean<\/sub>) + a + ui<\/sub> <\/p>\n

where imean<\/sub> is the mean value for the indices of the data. So, if the indices were 1 through 137, imean<\/sub>=69.<\/p>\n

We’ll find that the value of ‘a’ is equal to the sample mean of the Yi<\/sub>; I leave showing this as an exercise to the reader. \ud83d\ude42 But it’s obvious from this if ‘a’ is equal to the sample mean Yi<\/sub>; then the error in a just the error \u00ce\u00b5Y<\/sub>. So, I am now claiming the errors in the determination of ‘a’, the intercept in the second form of the fit is uncorrelated from the error in the determination of the trend ‘m’. <\/p>\n

To provide an empirical demonstration of this when the residuals are white noise, I created 105<\/sup> 137 month sequences of white noise and computed a the sample mean and trend for each series. This resulted in 105<\/sup> pairs of trends and means. I then took the step of normalizing with the estimate of the uncertainty based on the least squares fit. This results scaled errors with mean equal to zero and standard deviation equal to 1. I then computed the cross correlation between the scaled error for the mean and the trend which for the most recent case tested happened to be -0.0004674765; and also made a scatter plot of the two:
\n\"\"<\/a>
\nThe more or less circular appearance is typical of uncorrelated data.<\/p>\n

Since the main purpose of this post was to clarify a point for PaulK, and I think<\/i> this might be sufficient for the clarification, I’m going to stop here so we can continue our conversation on my attempting to develop a sort of “portmanteau” test involving testing the combination of the trend and the constant term from a least squares fit. I think such a test would have the properties of being a bit more difficult to cherry pick and under some circumstances can be more powerful than testing either the trend alone or the constant term alone. (There are circumstances where I think it may be worse– specifically, if the baseline was too short in time, it can become less powerful. But testing power is something that can be done after we figure out if I’m missing something that makes my portmanteau test untenable. )<\/p>\n

R File<\/b>: This R file (MonteCarlo d_star<\/a>.) was used to generate the plot. The demonstration with AR1 is also included. So are various cryptic and obscure comments. <\/p>\n","protected":false},"excerpt":{"rendered":"

This post is mostly for Paul_K; he and I discussed on of my claims in my previous article discussing the NOAA anomaly. Specifically, I claimed to compute two possible test variables from linear fit to observed data which in the current post I will call d*a and d*m and said the errors in these tests … Continue reading What’s uncorrelated with what? For PaulK<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-15735","post","type-post","status-publish","format-standard","hentry","category-statistics"],"_links":{"self":[{"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/posts\/15735","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/comments?post=15735"}],"version-history":[{"count":0,"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/posts\/15735\/revisions"}],"wp:attachment":[{"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/media?parent=15735"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/categories?post=15735"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rankexploits.com\/musings\/wp-json\/wp\/v2\/tags?post=15735"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}