In comments, Joshua described, and so indirectly proposed, a statistical test one might use to determine whether a particular data set for the number of trans mass murders, k<\/i>, out of a collection of n<\/i> mass murders could be used to test the hypothesis that trans individuals commit mass murders at the same rate as non-trans individuals. I don’t think his intention in posting this was to propose a test. I think his intention was to tell me how woeful my view that the main problem with running statistical tests on hypotheses about the rate at which trans individuals commit mass murders is that we don’t have reliable data. But, whether or not he intended to do so, he did propose<\/i> a method which, of course, we could use to test actual samples.
\n<\/p>\n
In this post, I’m going to quote from the (currently reserved) comment in which he describes a general statistical test. He applied the method to hypothetical data. I’ll also be correcting some of the numerical errors that crept into his analysis of that hypothetical data. I will also be releasing that comment and placing it in the comments thread for this post so you can see the full comment.<\/p>\n
On to Joshua’s comment! I’ll skip the first bit as irrelevant to the proposed statistical test and start with this.<\/p>\n
\nOh, and it\u2019s cute that you\u2019d want to take the time to show me a test, but really don\u2019t waste your time. I can just ask Grok to run some numbers for me:<\/p>\n<\/blockquote>\n
Joshua then post Grok’s answer to a question Joshua asked; the question itself is not. So we will call the entity responding “Joshua-Grok”. Content from Joshua’s comment is in blockquotes. I will break that up and add my own observations. <\/p>\n
Joshua-Grok’s answer begins:<\/p>\n
\n\nExpected Proportion Under the Null<\/p>\n
If trans people make up 1% of the population, we expect 1% of shooters to be trans by chance. For a sample of n shooters, the expected number of trans shooters is n \u00d7 0.01.\u201d .<\/p>\n<\/blockquote>\n
That is indeed what we expect if individual trans individuals become shooters at the same rate as non-trans individuals.<\/p>\n
\n\n2. Binomial Test Setup<\/p>\n
Let n be the number of school\/church shooters in the sample (e.g., 50 or 100 over a decade).
\nLet k be the number of trans shooters observed.<\/p>\nThe probability of observing k or more trans shooters by chance, given a population proportion of 0.01, is calculated using the binomial distribution:P(X ? k) = 1 \u2013 P(X < k), where X follows a binomial distribution with parameters n and p = 0.01.<\/p>\n<\/blockquote>\n
This would seem to tells us something about the situation Joshua wanted Joshua-Grok to analyze. It seems to be school or church shooters (as opposed to other possible shooters.) Joshua wants Grok to assume 1% of the general population is trans. The variables ‘n’ and ‘k’ are the total number of shooters and total number of trans shooters respectively.<\/p>\n
I’ll now tease out his notation so we agree what it is. I’ll also translate this into R commands. This will allow those familiar with R to repeat the calculations.<\/p>\n
1) The notation P(X ? k) contains a ? which likely means WordPress stripped something. Based on the wording “P(X ? k)”, we can infer P(X ? k) must be P(X ? k). That is the probability that “k or more” will be observed out of whatever sample size “n” is later used and single probability “p” are used. Joshua-Grok says it will use p=0.01.<\/p>\n
2) Based on the above, P(X < k) must mean \"the probability of observing fewer than k shooters\" given the parameters which will be n shooters, with the probability that any individual shooter is trans is 0.01\".\n\nJoshua appears to be rounding to three decimal places. So if n=50 and the population of trans is 0.01 in R P(X < k) would be calculated as \nP(X < k) = round(pbinom(k-1,50,0.01),3);\n\nThe reason for k-1 is that, pbinom calculates \"the probability of observing k or fewer\", Joshua is saying just \"fewer\". \n\nJoshua-Grok continues:\n\n\n\n\n
\nWe want the smallest k where P(X ? k) < 0.05, indicating statistical significance.<\/p>\n<\/blockquote>\nThis statement tells us Joshua-Grok has picked a single sided<\/i> test. He doesn’t say why he picked a single sided test, but this is his prerogative. Some might have picked a two sided test. He’s also picked a confidence level of 5%. That’s also his prerogative and pretty conventional.<\/p>\n
Joshua-Grok now starts to calculate.<\/p>\n
\n\n3. Calculating the Threshold<\/p>\n
Let\u2019s test for different sample sizes (n) to find the minimum number of trans shooters (k) that would reject the null hypothesis at p < 0.05.Sample Size: n = 50 (e.g., 5 shootings\/year \u00d7 10 years):<\/p>\n<\/blockquote>\n
Okie dokie! Let’s do that!<\/p>\n
\nExpected number of trans shooters: 50 \u00d7 0.01 = 0.5.<\/p>\n<\/blockquote>\n
Yes.<\/p>\n
\n\nUsing a binomial calculator (or cumulative distribution function), we compute P(X ? k) for increasing k until p < 0.05:\n \t\n\n\n\n
For k = 2: P(X ? 2) ? 0.264 (not significant).<\/li>\n For k = 3: P(X ? 3) ? 0.080 (not significant).<\/li>\n For k = 4: P(X ? 3) ? 0.019 (significant at p < 0.05).<\/li>\n Proportion: 4\/50 = 8%. So, if 8% or more of shooters are trans, the result is statistically significant.<\/p>\n<\/blockquote>\n
If<\/i> P(X ? 2) means what Joshua says it means “The probability of observing k or more trans shooters by chance”– the correct results for the calculation above should be:<\/p>\n
> round(1-pbinom(c(2,3,4)-1,50,0.01),3)
\n[1] 0.089 0.014 0.002<\/p>\nWe can now replace Joshua’s numbers with the correct values and also correect the declarations of significance.<\/p>\n
For k = 2: P(X ? 2) ? 0.089 (not significant).<\/li>\n For k = 3: P(X ? 3) ? 0.014 (significant at p < 0.05).<\/li>\n For k = 4: P(X ? 3) ? 0.002 (significant at p < 0.05).<\/li>\n Proportion: 3\/50 = 6%. So, if 6% or more of shooters are trans, the result is statistically significant.<\/p>\n
But maybe some of you don’t trust my R. So I can also ask Grok:<\/p>\n
\nThanks! I’m going to divert a little to something “Joshua” claimed. He was interested in a coin that got 0.01 heads in a single flip. He wanted to flip it 50 times. And he wanted the probability of getting the following three things:<\/p>\n
“k or more” heads, for k values of 2, 3 and 4.
\nCan you give me those values? (You need not explain.)<\/p>\n<\/blockquote>\nInitially, Lucia-Grok gave me ~ 0.091, 0.014, and 0.002. Those did not<\/i> exactly match my values from R, but they were close<\/i>. I told Grok what I got using R and asked it if it had approximated. Grok gave me a long winded answer which ended with “The correct probabilities, using the binomial distribution, are: [..] 0.089 [..] 0.014 [..]. 0.002 . These match your R output.”<\/p>\n
Back to Joshua-Grok which repeats the analysis, but this time with a different number of shooters. The new sample size is n=100. <\/p>\n
\n\nSample Size: n = 100 (e.g., 10 shootings\/year \u00d7 10 years):Expected number: 100 \u00d7 0.01 = 1.<\/p>\n
Testing values of k:<\/p>\n
For k = 4: P(X ? 4) ? 0.189 (not significant).<\\li>\n For k = 5: P(X ? 5) ? 0.081 (not significant).<\\li>\n For k = 6: P(X ? 6) ? 0.029 (significant).<\\li><\/blockquote>\n How do these<\/i> match what I get with R?
\n> round(1-pbinom(c(4,5,6)-1,100,0.01),3)
\n[1] 0.018 0.003 0.001<\/p>\nCorrecting Joshua’s text we get:<\/p>\n
Sample Size: n = 100 (e.g., 10 shootings\/year \u00d7 10 years):Expected number: 100 \u00d7 0.01 = 1.
\nTesting values of k:
\nFor k = 4: P(X ? 4) ? 0.018 (significant).
\nFor k = 5: P(X ? 5) ? 0.003 (significant).
\nFor k = 6: P(X ? 6) ? 0.001 (significant).<\/p>\nAs before, I asked Grok to estimate.<\/p>\n
\nJoshua next moved on to a different claim . He now wanted to flip the same coin 100 times. He wanted the probability of k or more heads for k values of 4, 5 and 6. Can you give me those?<\/p>\n<\/blockquote>\n
As before, Grok first used a Poisson estimate and returned (0.015, 0.003, and 0.001).<\/p>\n
So I asked “Did you do this exactly? Or are they approximated?” It admitted to approximating and now returned these numbers as correct. (0.018, 0.003, and 0.001). These match my R values– which is not surprising because Grok hooked in R to do the calculation. Grok even gives the code!<\/p>\n
Back to Grok answering Joshua: <\/p>\n
\nProportion: 6\/100 = 6%. If 6% or more of shooters are trans, the result is significant.<\/p>\n<\/blockquote>\n
Well… except that 6 out of 100 was not<\/b> the threshold for significance. From the following we can see the threshold for significance is 4. So the correct proportion is 4%. <\/p>\n
Joshua-Grok now repeats but with n= 20 shooters! <\/p>\n
\n\nSample Size: n = 20 (e.g., 2 shootings\/year \u00d7 10 years): Expected number: 20 \u00d7 0.01 = 0.2.<\/p>\n
Testing values:For k = 2: P(X ? 2) ? 0.017 (significant).<\/p>\n<\/blockquote>\n
Grok says 0.017. Let’s check:<\/p>\n
> round(1-pbinom(c(2)-1,20,0.01),3)
\n[1] 0.017<\/p>\nHey! The Joshua-Grok got a number right!!!!<\/p>\n
Joshua-Grok then generalizes its results. But bear in mind, Joshua-Grok’s conclusions for n=50 and n=100 are based on incorrect results for the cumulative probability. So these generalizations are based on wrong numbers for those two cases. Neverthless, quoting it:<\/p>\n
\n\n4. Generalizing the Threshold<\/p>\n
The required percentage decreases as the sample size increases because larger samples provide more statistical power. Roughly:<\/p>\n
For n = 20, you\u2019d need ~10% (2 trans shooters) to reach significance.<\/li>\n For n = 50, you\u2019d need ~8% (4 trans shooters).<\/li>\n For n = 100, you\u2019d need ~6% (6 trans shooters).<\/li>\n<\/blockquote>\n If we use the correct numbers the above becomes:<\/p>\n
The required percentage decreases as the sample size increases because larger samples provide more statistical power. Roughly:<\/p>\n
For n = 20, you\u2019d need ~10% (2 trans shooters) to reach significance.<\/li>\n For n = 50, you\u2019d need ~6% (3 trans shooters).<\/li>\n For n = 100, you\u2019d need ~4% (4 trans shooters).<\/li>\n Back to Joshua-Grok who continues:<\/p>\n
\n\nHowever, these percentages are sensitive to the sample size and the true population proportion. If the trans population is 0.5% instead of 1%<\/b>, the threshold percentages would be slightly higher<\/b> (e.g., ~10% for n = 50), and if it\u2019s 2%, they\u2019d be lower<\/b> (e.g., ~6% for n = 50).<\/p>\n<\/blockquote>\n
Bold mine. <\/p>\n
Of course<\/i> the percentages of shooters that would make us reject the hypothesis that trans become mass murders at the same rate as non-trans are sensitive to the number of mass shootings observed in a period and true population proportion of trans which may not be 1%.<\/p>\n
But, before we debate if the proportion of trans in the general population is doing, let’s discuss what Joshua-Groke claims. Focus on the part I made bold. As a generalization, something is backwards. Because if the proportion trans in the general population is lower<\/i> than 1% the threshold percentage of shooters required to reach significance is lower<\/i> than if the percentage of trans is 1%. You don’t even need to do the numbers to know this. After all, if there were 0.00001% trans in the general population, you’d marvel if 2 out of the 50 randomly draws of people were trans! Likewise if the proportion of trans is higher, the threshold value to reject the proposition they commit mass murder at the same rate is higher. Say 50% were trans. You would be unsurprised if 50% of the shooters were trans. That’s about what you would expect!<\/p>\n
So what are<\/i> the right numbers for 0.005% trans?<\/p>\n
For the 0.5% trans, the threshold for 50 is 2, and 2\/50 is 4% not 10%. For 2% trans the threshold is 4, and 4\/50 is 8%. Here’s the R.<\/p>\n
> round(1-pbinom(c(1,2,3,4)-1,50,0.005),3)
\n[1] 0.222 0.026 0.002 0.000<\/p>\n> round(1-pbinom(c(1,2,3,4)-1,50,0.02),3)
\n[1] 0.636 0.264 0.078 0.018<\/p>\nOk… so I’ve also been replicating with Grok. Naturally, I asked Grok. What does Grok say? Now things get a little interesting. <\/i><\/p>\n
Turns out Grok makes mistakes!<\/h2>\n
\nSuppose the probability of heads on a single toss of a coin is 0.5%. If we toss it 50 times, what is the probability that we would get “k” or more heads. Please do this for k=c(1,2,and 3).<\/p>\n<\/blockquote>\n
Grok came back formatted nicely but basically told me c(0.221, 0.080 and 0.013). It should be evident these do not<\/i> match c(0.222 0.026 0.002). Grok conveniently also gave me R code to run for myself.<\/p>\n
I ran the code Grok gave me and told it the R code it supplies gives c(0.222 0.026 0.002). Those match my final numbers, not Grok’s.<\/p>\n
Grok redoes stuff, tells me his numbers and accused me of having the wrong value for “p” or “n” stored.<\/i> His code snippet provided both values! So, as if! So, I give Grok simpler code with “p” and “n” hard wired in. Once again, R returns my previous numbers. I tell Grok Grok still decides it’s right!! It does post details of what it does. I read that long, long long discussion and tell Grok,<\/p>\n
\n\nHmmm…. for P(X=1) you do P(X=1)=50*0.005*0.995^49. Then you say that’s 0.14095.<\/p>\n
My R says
\n> 50*0.005*0.995^49
\n[1] 0.1955559\n<\/p>\n<\/blockquote>\nIt took a little more bullying on my part, but finally Grok found it’s mistake. It had computed 50*0.005*0.78271435 = 0.1409514, which is wrong. Feel free to check. Grok had stored that as a result and re-used that number for subsequent calculations. <\/p>\n
Interestingly, this is an incredibly unsophisticated mistake. Grok was able to do all sorts of nifty binomials, explain and so on. And then, just like any human, it is capable of the equivalent of a transcription error. Once it makes that error, and “writes that number down”, it just steam rolls ahead with the wrong number.<\/p>\n
Once it corrected its mistake, Grok replicates my numbers within rounding.<\/p>\n
\nYour output is essentially correct, and the differences are due to R\u2019s rounding behavior (e.g., 0.221199217 rounds to 0.222, 0.001158029 rounds to 0.002). My previous calculations for k = 2, 3 were incorrect because I used an erroneous P(X = 1) ~~ 0.1409514 , likely from a miscalculation of (0.995)^49 or a typo in my manual process.<\/p>\n<\/blockquote>\n
So, Grok can do great stuff. It can also go wrong and really stick to that. You have to check things with R!<\/p>\n
Returning from the Grok makes mistakes interlude<\/h2>\n
Yes, the results of the threshold value (whether by number of %) depends on the sample size and the probability– in the trans case, the proportion of trans in the population. Estimating either are actually very contentions matters; but Joshua tells me the proportion of trans is “(roughly 0.5-1.5%, per Gallup and Williams Institute estimates”. Well, that’s a factor of 3. And is that range even wide enough? Who knows? <\/p>\n
As I previously wrote as I previously wrote <\/a> when discussing analyzing data on the issue of whether or not trans individuals were more or less likely to commit mass murders of some sort, ‘Most of us actually touched on much of the \u201cmain\u201d problem in any statistical analysis which is we simply don\u2019t know numbers. There are some others\u2013 including cherry picking and also\u2026 I don\u2019t know it\u2019s name. It\u2019s not the texas sharp shooter. It\u2019s\u2026. something to do with picking the observation and then only doing the statistic because you saw that thing. \u2019 I guess that’s called “post hoc theorizing”. Wikipedia says post hoc theorizing not so bad<\/a> as long as you recognize and admit you are doing it and don’t pretend the analysis is based on a pre-planned experiment.<\/p>\n
But at least now Joshua has suggested a “method”. I’m not sure he’ll continue to “like” it now that it gives answers he probably doesn’t like. And he’s suggested a range for p. So, there is that. We can now apply the Joshua-Grok method (with correct numerical results) to other problems. Of course, we can also criticize the method. I mean… “Why a one sided test?” \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"
In comments, Joshua described, and so indirectly proposed, a statistical test one might use to determine whether a particular data set for the number of trans mass murders, k, out of a collection of n mass murders could be used to test the hypothesis that trans individuals commit mass murders at the same rate as … Continue reading Statistical Test Described by Joshua<\/span>