While I was cursing at R, Anthony posted to inform us that THE password has been released.
My favorite line is
I wasn’t aware of the arrangement but warmly welcome their decision to support my project. For that end I opened a bitcoin address: 1HHQ36qbsgGZWLPmiUjYHxQUPJ6EQXVJFS.
I’m going to go back to cursing at R. I anticipate reading all sorts of juicy (or not so juicy) quotes over the next few weeks.
Open thread.
I am not a regular visitor, nor do I use decimal points. Is it too late to bet on March UAH?
I’m not believing the text was written by the white/black-knight hacker.
What’s the cursing of R about? (I’m a padawan statistics learner, but pretty good at programming.)
On the topic, I’d love to see redacted text of the emails released, but also a network structure of the email senders and recipients. (Some named and perhaps peripheral characters abbreviated to preserve privacy.) I’d love to unleash some social network packages on it.
Wayne2–
The cursing is that I’m trying to set up a monte-carlo thing and I’m organizing into functions and so on. Like 99% of people who programs, I curse when programming. The cursing can be triggered by:
1) poor typing.
2) inserting ‘-‘ when I need +.
3) thinking I found a bug. (Then, wondering if it is a bug and so on.)
4) The cat coming and walking on the keyboard in an attempt to get treats. (His chin is resting on my left hand as I type. I kid you not.)
5) Things that just drive me nuts about R and R documentation.
And so on.
On the social networking packages…. Carnac predicts that the whole thing will be public in a week. I mean… secret passwords? To a small group? Like that’s gonna stay secret forever? Pfffttttt!
I’d be surprised to find much this time around, given that the crop has already been creamed twice (so to speak :-p )
Wayne2 (Comment #111296)
March 13th, 2013 at 1:01 pm
What’s the cursing of R about? (I’m a padawan statistics learner, but pretty good at programming.)
On the topic, I’d love to see redacted text of the emails released, but also a network structure of the email senders and recipients. (Some named and perhaps peripheral characters abbreviated to preserve privacy.) I’d love to unleash some social network packages on it.
###########
write the code. I have the data.
Need to research the R text processing
Speaking of cursing, I enjoyed the comments in your ‘KnightEtAll_ENSO_Adjust.txt’ all out of proportion for any reasonable person. I’m a software engineer and I’m not permitted to curse in my comments, however appropriate it often would be to do so, which is probably why I found it so darn entertaining.
Time to generate some bitcoins. FOIA provided me with quite a few hours of amusement at the expense of some unprofessional folks who couldn’t be more deserving of ridicule and derision. I think it’s only fair that I pay him back.
And yeah, the secret password will probably be out in a week, and along with it the inevitable emails of verbotten prof-on-grad-student action…
Wayne2
I wonder if you saw this post by Warwick Hughes after Climategate 1:
http://www.warwickhughes.com/blog/?p=324
Steven: OK. Depends on what “the code” is. I’d think the first step is to put all the messages into a DB indexed by message id or file name or something unique, and each entry of the mail header (From, Date’s, To’s, CC’s, Subject, etc).
Someone on one of these threads has claimed that they’ve had good success with putting past CG email dumps into Thunderbird and doing various searches there, and it might magically contain a lot of nice email parsing features to piece things together and it might be a better option than a general-purpose DB.
Then write something to take such a DB and create nice json (or other nicely-readable format) files based on from-to relationships, combined with content searches. (Or maybe Thunderbird can be made to do it.)
It may be easy to find email reply chains, but perhaps more interesting — and a bit harder to find — would be email forwards. (Depending on how munged/edited the forwarded email is.)
I think analysis will ultimately be in R using the sna package or something like that, though Cytoscape and other tools come to mind as well. I think that the initial (mainly textual) processing would be much faster in Python or perl or something that works nicely with text. I don’t think R would be the way to go for that part.
Any thoughts?
If you haven’t yet, you might want to try using R Studio. I use it occasionally when I want to analyze more complex data sets. It isn’t quite my favorite on a linux system since you can’t highlight and paste without first copying, but it does provide a good display of data, output, code, etc. in a graphical interface.
Mark, Lucia,
Programming and cursing go together like glove and hand. Hard to imagine one without the other.. BTW, I regularly curse in comments. Since I own the company… nobody complains ;-).
Mark… I was looking at the code as well and saw Lucia’s comment regarding her luv of the R documentation 🙂
BTW… I wonder if FOIA’s opening of the BitCoin account is a mistake. Not sure if there is a legal difference between unauthorized disclosure/entry for fun or profit?
Wayne2 (Comment #111304)
March 13th, 2013 at 2:27 pm
I reverse engineered the CG1-CG2 emails in to a single Thunderbird compatible inbox file. It’s Relatively easy to add many more. No vexatious R script needed.
Screencap-
http://imageshack.us/a/img708/2041/tbcg1cg2.png
Lucia- Corrupt link in my previous post. can you replace with this one? thanks.
http://imageshack.us/a/img708/2041/tbcg1cg2.png
Duke: I think the first thing with the new batch will be to try to figure out a way to extract useful information without revealing private information of parties who might be peripheral players. The old redaction step.
If we could boil down each message to who it was from and to, what other message it was a reply to, or perhaps what other message it was a forward of, we could pretty much have a totally-redacted dataset (if we were careful with email addresses) that could be analyzed with social network tools. That would be interesting in itself: what reporters were best buds with what climatologists, which climatologists wrote to which journal editors, etc.
Perhaps if we find interesting things there, we could give some sense of direction to those who have direct access to the actual emails.
Oh, and in addition to the from-to stuff, the dates. It would be nice to come up with a timeline that might guide folks based on known events, known emails from CG1 and CG2, and FOIA requests.
wayne2.
The first order of business for me is going to be sorting mails to see which have personal info and which dont. because they cant be released without redaction. I’m still noddling around for an approach.
I think I have one, goes something like this:
Build a concordance; this gives me a word to document link.
based on the concordance, mails get selected and dumped into a
bin for redaction: redaction will probably require human eyes. personally I read the first 2000 mails in about 2 days…
After mails have been redacted they can go into a public database where your approach seems fine.
Nobody wants to publish the internal love notes or family stuff that may be in the pile or medical stuff.
ha, maybe there is dump of mails every week for the 5 years..
AJ…
I forgot the comment about trying to figure out a simple filter was there when I uploaded. 🙂 (I could figure out the Butterworth filter. But argh!!!)
In the US his opening a BitCount account won’t do him any harm. Anyway, no matter what country, someone has to find him first. He’s probably using TOR, lots of fake disposable emails and so on. Someone might be scouring https://lists.torproject.org/pipermail/tor-talk/ for names of known climate denizens who can be shown to know about tor. That said, I have a comment up there and have never used TOR. I started reading to around the time I started banning TOR connections from this blog. (Speaking of which, I only partially automated. So, a few TOR connections might get through. But most can’t.)
Steven: The more I think about it, the more I think that a from/to/in-reply-to/date output would be very useful to make public first. Then, let people munch on that. Find the journalist who talks a lot to Mann, find the flurry of emails on the eve of a paper being approved, find emails in a period that a university says there were no emails to be found, etc.
THEN that would guide good folks like you as to what to read/redact/release.
I think any automated redaction is bound to fail: it will probably set aside too many emails to be human-redacted and it will probably release emails that are very personal that don’t happen to have the right words/patterns in them.
So a guided human-redaction based on social networking, dates, and in-replys would be a good guide for what might be truly important. Along with keyword searches for ‘hockeystick’ or the names of those who were forced out of journal/academic jobs because of their beliefs.
Ultimately, a full-scale search-n-redact will have to be done. But I’d rather do something straightforward and have crowd-sourced targets than rush the redaction process.
hey Wayne, what If i strip off the header ( to from subject cc etc )
into a seperate file and the connectivity stuff can proceed in parallel and then we can link via index to the content?
yes, I was thinking along the same lines.
Will have to wait for tonight.. need to get back on some other stuff
and I have R maintainer Issues banging on my door.
if every thing goes right I should have something done.
Do we know if the attachments sent with emails are recoverable?
Looking at drafts of papers would be so nice.
Lucia… Opening a bitcoin account might not harm him, but politics could possibly. Hypothetically, if this “offence” happened in Chicago and he was caught, do you think the DA would press charges? My guess is that they wouldn’t. The Heartland/Gleick episode must have set a precedent 🙂
Humm…. FOIA seems to be toying with us a bit. If we take him at his word, and he is neither British nor from the States, then where else might considerable fluency (native, or nearly so) in English be found? New Zealand, Australia, Canada, India, and a few smaller countries, plus maybe South Africa or the Netherlands, or perhaps someone from most anywhere who spent quite a lot of time living/studying in an English speaking country. That doesn’t narrow it down very much!
Steven: I think that would work. Each file would have its name (a unique ID I assume) and then the header fields.
Not sure what fields might exist, but I’d want to see all of them. (May even leave the Subject: field out if you’re concerned about sensitive info in them.) I believe Reply-to: and Message-id: fields exist on some systems, and that would helpful.
You’d have to actually dig into the messages and look for quoted header fields to try to detect forwarded emails and who they were originally from/to/ID, which would be less simple but would be useful in the long run since it seems a lot of emails I’ve seen are chains of forwarded messages.
Ok,
The concordance is written and running.
Its not perfect but it will give you a normalized table
of “token” and filename. The filenames are numbers
and so represent a natural index.
220K mails, I fear the concordance will be massive.
but I built it as a normalized table fwiw. converted everything to lower case and some other stuff.
the header code should be easy if the whole stack looks like the first few hundred. As a first pass I think I’ll just grab the “top”
header. any embedded mail ( threaded crap) will be a parsing nightmare.
for database tables i’d assume a table of from, to, cc,
maybe words in subject line.
Hopefully the concordnace will finish running soon, and I write the header stripper
10% done, and 150MB concordance.. hmm that stays under the R memory limits so I dont have to use LaF or Big Memory.
Maybe it’s phi. In Switzerland. 🙂
uggg.
50% mark. my machine is a pig.
Next up, I’ll create normalized tables for
words in the To:
words in the From:
words in the subject:
so yu can FIND mail Where From contains “lucia” AND To: contains
“jones” and subject contains “knitting” and body contains
“crazy”
or sumthing like that. hmm and at first the search will just return a number to the file.
so effectively there be a database prepared before the mails get released. Some other person will have to that
Is there any coordinance between the password recipients re the forensic work?
Is there any coordinance between the password recipients re the forensic work?
##########
huh? what you mean?
DOne!@
1.4 Gbs, complete concordance.
Mosher-
I’m thinking mysql and phpadmin would be ideal for this. the WAMP package…
http://www.wampserver.com/en/
Mosh says, “Done.”
So, you see how patient we are, waiting for the link?
@mosher, sorry if I was unclear, I meant to say whether are all people who received the password doing their own forensic work or is there contact between your guys like who’s going to look for what and where, otherwise there could be duplicate research? Otherwise I can understand that there may be a rush who has the first juicy scoops.
@steven mosher (Comment #111312)
It’s just a matter of time before all the personal stuff comes out as well.
Don’t you have anything useful to do?
Bugs,
Please note that while a fair number of skeptical bloggers have received the password, nobody has simply posted the password for anyone to use. Do you think this is just an odd coincidence, or can you accept that all those skeptical bloggers have no interest in the personal stuff? I think you will find very few people care about the personal secrets of climate scientists, even if they think those same scientists acted less that professionally, and even if they think those same scientists have had their thumbs on the balance for a long time.
mosher
I think he wants to know if there is some sort of coordination where all those who received passwords have created some sort of project with tasks, milestones, a GANT chart, deliverables and so on. That’s the sort of thing a project lead might do if they wanted to avoid duplicate effort and so on. I would suggest the answer is “Heck no”.
What has happened is whoever was sent the email, read it. They saw they were permitted to distributed to “trusted” people. The delivered it to “trusted” people and each of those people is doing whatever the heck they find interesting to do– provided they have the time.
I would even go so far as to say that most people who have the email don’t know who else has the email.
We know Jeff Id has a password because he wrote
“The password for the remaining emails was sent to me and several other blogs last night.”
http://noconsensus.wordpress.com/2013/03/13/climategate-3-0/#comments
I betcha dollars to donuts he is doing almost nothing. As it happens, I have the password. And you know what I did. I looked up from my R coding, checked to see if Anthony had announced, and wrote this post. That’s it. As a result, I learned Anthony had posted and now I know Mosher is sifting through.
(As you might suspect, I agree with Zeke when he says “I’d be surprised to find much this time around,”) That said: I think we’ll be reading quotes over the next few weeks. People will find quoteables and we’ll read them.
Oh. And I’ve predicted the 1/2 life for the “secret” password is 3 days. If FOIA doesn’t want the truly personal stuff to get out, he’d better change the password or take the files down because someone is going to give it to someone who is going to give it to someone who is….
Mosher –
I had a few ideas on what could be done for analysis but it seems you are well advanced along that path already:
http://joannenova.com.au/2013/03/climategate-iii-the-password-is-out/#comment-1252598
Also thank you for recognising that it is only proper to try to withhold emails that aren’t relevant to scientific global warming skepticism. I think CG3 has the capacity to rapidly turn into a bad joke if the FOIA annointed skeptics don’t stay on target.
If FOIA doesn’t want the truly personal stuff to get out, he’d better change the password or take the files down because someone is going to give it to someone who is going to give it to someone who is….
He can’t – it’s embedded in the all.7z zip, and already distributed – unless he’s using an IRM package which requires the zip to phone home for instructions before opening. I doubt that, however. That requires him to host a server somewhere, which is easily traceable.
I see Lucia.
Hmm.
1. I have the password.
2. Nobody will ever get it from me.
3. I have the mails.
4. I will release mail on a case by case basis to any interested party who is mentioned by name in the mail provided there is no personal information in the mail.. a judgement call of course. One person, has requested this. he got one mail that said absolutely nothing about anyone but him.
5. I won’t release all the mails, they need to be gone through.
6. I’ll finish some prototype databse work and walk away
a) a concordance; ( code for it if you have the mails)
b) a deconstructed header: ( code for it if you have the mails)
Then I’m walking away. Reading through the first 2000 mails was something I did on autopilot over the course of 2 days. It was a puzzle, prove that these mails are real or fake. They stick in my head. I could not bring myself to read CG2.0 and can’t see reading CG3.0 except for the 90 or so mails that refer to me. Even there it’s the past and I’m not really terribly interested in it. For me I have reduced CG3.0 to a programming problem, not an emotional thing.
I have to be careful what I put in my head because it stays there. Obviously I dont watch horror films and sifting through CRU mails is at once boring, infuriating, and creepy with emphasis on the latter.
Steven:
I agree that you should not release personal stuff but for the heck of it, can you run “chiffon”, “spiked heels” and “teletubbies” through the concordance to see how many hits?
Just a hunch. Thanks.
“Hoi Polloi (Comment #111340)
March 14th, 2013 at 2:00 am
@mosher, sorry if I was unclear, I meant to say whether are all people who received the password doing their own forensic work or is there contact between your guys like who’s going to look for what and where, otherwise there could be duplicate research? Otherwise I can understand that there may be a rush who has the first juicy scoops.
No coordinated work. The few people I have talked to are all going to honor FOIA request not to distribute the password.
And nobody wants to just publish all the mails without redaction.
Trivia question: who was the only indvidual to have personal information released in CG 1?
I’m doing a concordance. Actually just the code to compile it and passing that code out. But if you dont have the mails its worthless.
So, no plan no project schedule no agreed approach. Obviously some folks are getting legal opinions ( as we did in the first case ).
Scoops? na. I think the approach will be a slow and prolonged
Imagine word of the week. Where readers request a word and mails that have that word get reddacted and posted.
Traffic for life.
If one was so inclined.
‘It’s just a matter of time before all the personal stuff comes out as well.
Don’t you have anything useful to do?”
1. In my mind the only way to prevent personal stuff coming out is to
get the tools put in place to allow for a redaction project.
2. I consider that useful.
3. the code took 15 minutes. Its not pretty, it gets the job done.
it even uses loops and mapply() woo hoo.
4. Tonights code will probably take 45 minutes.
So, donate an hour of my time to try to keep the personal stuff off the web. Worthwhile and fun.
Mosh….Was it Jeff? That was part of the investigation so does that count?
Big. Whoop.
i.e. Why bother, SM?
@SM,
Never mind- I misread something you said.
We have a situation where good people are held hostage to very bad ideas, and I for one think we should honor the goodness in man and let actions be questions for the law. Where everyone else is let free. Manipulation can happen to us all.
Now is the winter of our discontent
Made glorious summer,
For the third time.
I’d drop the entire lot into splunk and wring it out there. Unless there’s a lot of html or encoded content I’ve never seen it take more than a few minutes to redact email addresses. Normalize it by taking out all line breaks and html tags. In other words, treat it like a spam engine does mail – take out the kruft, scan the rest. Splunk will find the rest by using lists in “interesting” words and phrases.
Speaking of spam tools, ClamAV, a free anti-virus tool lets you create your own signatures which can be very complex. It is a very fast parser. Beats awk, sed, grep all to hell for speed though those remain critical tools, too.
Cheers @ mosher, good luck!
Referring back to CG2.
“steven mosher (Comment #86227)
November 26th, 2011 at 3:53 am
Bugs.
There is a reason why the case is open.
There is a reason why the file is in the open password protected
No deal has been done. Yet.
He may want to come in from the cold.
He has contacted the other side.
he’s not a hacker.”
Looks like that theory is dead.
bugs–
How is the theory dead? FOIA didn’t say anything I read to suggest he is a hacker.
lucia: The larger question is why bugs would somehow feel vindicated if the emails turned out to be “stolen” by a hacker? No one has been able to question their authenticity and no one has tied FOIA to a well-known skeptic — no Gleickness in the air. So what difference does it make to the debate if it was a hacker?
I’d remind bugs that an “insider” has never had to be, say, Phil Jones or some other major player on The Team. FOIA could be a small cog in the machinery. FOIA might’ve simply had access to one of the many closets into which The Team stuffed their skeletons to avoid FOIA’s reach (the law, not the person). In fact, the the nom de plume FOIA might be a reference to this.
At any rate, in CG1 and CG2 we’ve seen how The Consensus was enforced by The Team, we’ve seen that there was internal dissent and not a true consensus, we’ve seen that The Team was willing to hide things, punish “enemies”, and request its members to break the law (delete emails). This all affects the credibility of The Team and the reality of The Consensus, and bugs simply can’t abide that.
On a side note, I’d say that FOIA appears to be extremely smart and skilled so if FOIA doesn’t think there’s much left after CG1 and CG2, I’d tend to believe it. On the other hand, I also think that FOIA may have been doing quick analyses like grepping for keywords and once people do network analysis, we may find that certain journalists and policy makers — perhaps using pseudonyms — were also on The Team.
Wayne2–
Sure. But sometimes if one focuses on the “larger” question it gives the impression that bug’s claim on the “smaller” question was correct and now all we want to do is discuss whether that’s “meaningful”.
In fact: Nothing about what FOIA wrote points to FOIA being a hacker or having gotten the files as a result of hacking. Nothing. So, we don’t need to delve into the pyschological make up of bugs’ desire that he be a hacker and/or what “it would mean” if FOIA were a hacker.
Frankly, I have no idea what “it would mean” if he were a hacker vs. a leaker. Certainly, it wouldn’t change what we learn about the climate scientists based on the contents of the emails! At this point, the statute of limitations has passed, so it wouldn’t affect the actions of the UK constabulary. So… what?
lucia: Agreed.
I wasn’t trying to challenge or deflect your question about bugs’ specific assertion. I was trying point out that bugs’ comment was what slavish worship of The Team and their Consensus demands. It’s bugs’ attempt to be a TV lawyer and say, “OBJECTION! That damning and incontrovertible evidence is non-admissable!”. It wouldn’t be sustained — even on TV — and it doesn’t affect the truth of the matter, but it’s all bugs has left to try to remain a true believer.
As you point out, the latest FOIA email says nothing that indicates that they were a hacker. Rather, it seems very plausible that they were inside The Team’s domain, though probably peripherally and only briefly. Someone has suggested that the bitcoin address was not actually to raise funds, but to provide a public key that FOIA could use to prove their identity if they choose to step forward, which is also the sign of a non-hacker. (A hacker would have other proof, such as trophies from their penetration of the site/server, and wouldn’t need to create anything to prove their feat.)
FIOA could be neither a hacker nor an insider. Given UEA practice of leaving files on non indexed urls, I think the most likely scenario is that FOIA simple found the email volume in an unsecured, open spot on the UEA website. It’s no crime to download a document from an unsecured web page.
Yes. Or they could create a bitcoin address because the figure they might get a few bucks in donations. I suspect they’ll get enough to buy a few six packs but not enough to retire.
Why is it whenever I hear someone say, “A hacker would…,” the speaker is inevitably wrong?
Wayne2, a hacker might keep “trophies” (whatever those are supposed to be), but that’s not something every hacker will always do. I’m not sure what you think one would have kept, but I can assure you there are plenty of hackers who wouldn’t keep incriminating evidence. Quite a few would prefer a tactic like the one you say they wouldn’t use.
lucia (Comment #111379)
March 15th, 2013 at 7:57 am
He is politically motivated, not ‘scientifically’. His language is not English, he is not an insider at the university. The server the files were taken from is not a web server, no way that they could be obtained with just an open URL.
bugs–
English is not his first language which you stretch to mean…. what? That he can’t have been inside the university ever? I’m sure plenty of Europeans regularly visit CRU and are insiders if only briefly. Heck, I bet Chinese visit and are “insiders” briefly for post docs graduate work and so on.
I have no idea if the server the material was on were connected to the “web”. I haven’t followed that detail. But that’s pretty irrelevant to figuring out if he was a hacker or not.
Even if the server was only an internal server, files could be obtained by connecting through a network.
Mosher had FOIA as being one of the FOIA team there. If you are working as an administrator or FOI, your English is going to be fine. If you are going to get data off a server that is connecting from the outside, that is not directly facing the web, you are going to have to do some hacking. He says the leaks were deliberately timed to do the most damage.
bugs–
Moshwers theory being wrong doesn’t make FOIA a hacker.
Mosher was wrong, though.
The hacker issue is a different argument, but I’m happy to go with hacker, still. If what FOIA is saying is true, he acted alone. My theory of the rich moron who can’t understand the science hiring some hackers is also wrong.
@lucia (Comment #111298)
Yes, this is where we get the vindictive and petty side of FOIA confirmed. He gets to humiliate them, while pretending he didn’t do it.
bugs,
If FOIA wanted to be vindictive and humiliate climate scientists, he would just have posted the damned password. Your rather twisted logic about FOIA’s motives reminds me a bit of Dr. Strangelove. The simplest explanation is that FOIA does not want to release private information.
A foreign post doc or grad student at UEA seems the most likely identity for FOIA. Based on his errors in English, most likely northern or eastern European, but he could be from almost anywhere.
SteveF (Comment #111403)
Lucian doesn’t believe that password is secure, other bloggers don’t, I don’t believe it for a second. His rationale is just a fig leaf.
SteveF (Comment #111404)
Mosher’s theory is dead wrong, as are various other ‘names’ from the department such as Briffa.
bugs,
So referring to the “not a hacker theory”, you said:
You’ve now downgraded to:
Yes. We all understand that you like the hacker theory and are happy to go with it. But that’s an entirely different from decreeing the non-hacker theory is dead. In fact, even if it’s a theory you don’t like and have never like (likely owing to your political outlook) , based on the actual evidence the “not a hacker theory” is at least as plausible ad the ‘hacker theory’.
I specifically referred to Mosher’s theory. He was dead certain he even knew who it was. Completely wrong.
bugs
But you know what? I might be wrong. I haven’t released it. In fact, I immediately took steps to make sure I couldn’t leak it accidentally or on purpose. It may be that everyone who has gotten it is the sort who is not going to leak it. That’s rather amazing to me… and yet it may happen that way.
Bugs,
we know nothing more today than we knew before.
let’s draw some distinctions so I can help you understand.
1. Insider/outsider: insider: employee of UAE; outsider: non employee.
2. Hacker/not hacker: Hacker: broke into the system: Not hacker: did not break into the system.
I will explain to you what I explained to the last journalist who called me: I have a person of interest. Here are the questions I would ask him, when you go to talk to him. I suspect he will not meet you face to face, he prefers written misdirection. and then I explained my questions. One of the issues was the use of the word “we” in the first mail which seemed to me to be a mis direction and reminded me of the letter in the patsy ramsey case.
And I likened this to a kid napping..mails instead of a human of course. So the questions were aimed at exposing potential personal conflicts. I count it as a coincidence that this discussion happened this month and count it as a coincidence that FOIA felt the need to discuss the issue of the papal “we”. I have not heard back from the journalist, wait for the book.
That said: I would say the probablity of me being right is single digit land. The person of interest would have to be a fairly good student of style to fake both sincerity and the little tweaks to the language that were employed. But interestingly this is the first time he has felt the need to excuse his command of the language.
I’d want to look at the other letters and see if there were similar oddities or if they just appeared in the latest letter which of course excused itself for linguistic oddities.
FOIA only released the emails, no more, no less. If there’s humiliation involved then it’s not his\her fault but because of the authors themselves.
Personal messages to each other about confidential, personal matters.
Mosher
Strangely, they aren’t oddities that *immediately* bring to mind any specific country. Everyone sort of notices distinctive markers in “chinglish” (missing articles… other hard to describe stuff.) or “romance language-english” (overabundance of latin cognates relative american or english usage, funny idioms etc.) But that letter is written by someone whose written English is good.
I say it’s phi. 🙂 (Not really. I don’t really say that.)
Hoi Polloi
Well… sort of both. Most people don’t expect emails to become widely circulated. They don’t expect it even though everyone knows they might be and has known since… on… the early 90s? If you say something “private” in an email, you should be aware it might get circulated. If it is, you– the person who wrote the email– are at least partly responsible for any humiliation that ensues.
Are the files made accessible by the password not previously plumbed for interesting emails or have they been looked at for the more obvious hits but maybe not scanned in their entirety?
I always wondered when bags of dimes were offered as “unsearched for rare coins”, whether they really were?
This is not at all to suggest any misrepresentation by FOIA, but more to ponder the possibility that there may be useful information in the files undiscovered in FOIA’s earlier screenings – files not earlier detected due to messages not having the usual signals.
John Ferguson,
I suspect what was released was all pulled out via some kind of keyword search. What is left does not contain those keywords.
lucia:
I think we were aware of it even in the ’80s. In those days we shared terminal rooms, and somebody literally could be looking over your shoulder (writing like somebody is looking over your shoulder at you typed is still good practice).
I was a grad student and some of my emails were published in a book without my prior consent (formally that is, I wasn’t asked, not shown which emails would be published, nor did I sign a form stating it was OK). They were given by one of my colleagues to a third party, and I was informed of it after the fact.
I didn’t really care, but it did seem a bit odd it wasn’t handled any better than that, especially by the publishing company.
What I find odd about some of the emails is the ones that basically refer to Mann as a d**bag or the equivalent. You shouldn’t write things like that about a colleague without the expectation he will eventually see what you wrote.
Is CG3 an entirely new batch, or does it contain previously released emails? If E= Everything “obtained” by FOIA, is CG3 = E, or do CG1+CG2+CG3 = E?
Maybe today, Mosher can tell us.
Carrick–
The reason I say the 90s is by then the wider public was email. Things like AOL existed and so on. Prior to that, it was a smaller circle. So people could at least imagine that emails weren’t going to get circulated to ‘everyone’. But by the 90s, really, everyone knew emails could easily be accidentally or on purpose disseminated to “all” because it was happening on a daily basis, discussed in comedy sketches — with comedians knowing the audience would “get” the joke and so on.
Any scientist or even just “garden variety person” who, in the 90s, didn’t know emails could be circulated “to all” was an blithering idiot. That’s not to say they expected most emails would be circulated. But they knew how easily they could be circulated.
Mr Mosher, you say “Nobody wants to publish the internal love notes or family stuff that may be in the pile or medical stuff.” yet you have no qualms about reading this private correspondence yourself?
I believe you studied English and so are familiar with the meaning of the word hypocrisy.
Louise,
There is absolutely nothing hypocritical about reading something you (a) intend to not publish, (b) prefer not be published and (c) don’t publish.
@bugsy #111415,
and that’s why FOIA sent the emails only to a few people with the remark:
He\she wants to reveal the whole set-up between this small congsi of climate manipuators.
“Scientists” who write that they will re-invent peer-review to keep out contrarian opinions deserve to be humiliated for their abuse of science and that’s what FOIA wants to achieve.
bugs,
“Personal messages to each other about confidential, personal matters.”
Humm.. I do have to admit that rigging peer review, coordinating email deletions to avoid having to disclose information under FOIA laws, and pressuring journal editors to keep them from publishing ‘the wrong kind of papers’ are things those folks considered confidential and personal, and very much wanted not to be publicly disclosed. Indeed, Phil Jones several times asked for information about these shenanigans to not be disclosed to anyone outside the ‘Team’. Which is as clear an indication as there can be that Phil clearly knew what they were doing was unethical.
.
I am confident we completely disagree about whether they were entitled to confidentiality to hide professional misconduct.
Louise (Comment #111426),
Are you really suggesting ‘privacy’ considerations should be used to shield people from the consequences of professional misconduct? Or perhaps you think professional misconduct is generally OK? You seem to understand English at least as well as FOIA; did you read what he wrote about his ethical justifications for releasing the emails? Did you take note that virtually all the damaging emails were about political advocates masquerading as scientific researchers? Had the emails shown nothing unethical happened, FOIA would probably have done nothing but delete them.
.
Just about nobody cares about love affairs, family conflicts or medical problems. Lots of people care about professional misconduct by publicly funded researchers. I hope it will not happen, but if embarrassing personal information is disclosed (and virtually none has so far), then those involved will have my sincere sympathy. They get no sympathy from me for the disclosure of professional misconduct.
You know what’s worth looking for in the climategate 3.0? Business emails from people at private companies that work on government funded “green” or “clean energy” projects. We could find a trail to lobbying, etc. Those should certainly be released and I hope if any are in there, Mosher should keep them in.
lucia (Com<ment #111434)
No conspiracy theories here. No sir.
SteveF (Comment #111432)
Realclimate agreed that deleting the emails was an error. Jones knew that this was a war he was caught up in. The hacking of their server, the people accusing them of doing it all for the money, to bring down capitalism, the hatred and threats made. The ‘redefine peer review’ quote being a case in point. They could see garbage being published that should not have been. The journal that published that paper had people resign from the board, because a sympathizer to the ‘skeptics’ had used subterfuge to make sure only the people he wanted to review the paper did so.
The leaks, according to the mail, were politically motivated. The selective quoting out of context, is also political.
Bugs,
These lines you wrote made me laugh out loud:
“The ‘redefine peer review’ quote being a case in point. They could see garbage being published that should not have been.”
For some reason this made me think of Mann. Reading some of the Climategate emails, it made his colleagues think that as well. The only difference here is that since it supported their professional misconduct, they did nothing about it.
Either you’re willfully being a hypocrite or you’re so hopped up on kool-aid that you can’t tell that you are. I’m not sure which it is.
bugs,
RealClimate (in the person of Gavin) admitted that deleting emails to avoid FOIA was a mistake. Big whoop. A moron, on a really bad day, maybe half drunk, could see that. What about those who were deleting those emails (like Mike Mann)? Did Mike admit that agreeing to delete emails was not a good thing to do?
.
Nobody really thinks these folks were motivated by money… at least not money directly in their pockets. (It is nice to be richly funded from the public trough, of course.) It is obvious to anyone who has looked at it that they were motivated mainly by ‘higher goals’…. political goals in particular, like ‘save the Earth from humans’.
.
Yous see, bugs, the issue is pretty simple: allowing political goals to influence science is always wrong. Trying to influence what science is ‘acceptable’ for publication and what is not is always wrong. Imagining that you (or anyone!) are uniquely able to decree that certain scientific POV’s are not acceptable is always wrong. Such arrogance is obnoxious and destructive, nothing less. The entire cabal was rotten to the core because it was motivated not by a desire to advance science, but rather by an overwhelming desire to achieve a set of specific policy goals.
.
Policy goals, honestly stated, and outside of science, are fine. Using the cloak of ‘climate science’ to advance specific policy goals is utterly wrong, and worthy of the kind of rebuke the UEA emails received.
SteveF (Comment #111444)
March 16th, 2013 at 7:21 pm
People say it all the time on the web, they’re in it for the money. As for higher political goals, I thought they were ‘saving the earth for humans’, not to mention the other species that share it with us.
bugs,
” I thought they were ‘saving the earth for humans’, not to mention the other species that share it with us.”
.
And that my friend is the political disagreement. Many people honestly see the policy actions that are claimed by many (including those involved in the UEA emails) to be required to achieve those goals as destructive and wasteful, or perhaps better stated, an expensive cure that is worse than the disease. Such policy goals should not be influencing what science is allowed to be pursued and reported. Science is not politics; the ends do not justify the means.
bugs, Steve F
â€I thought they were ‘saving the earth for humans’, not to mention the other species that share it with us.â€
.
I wonder how bugs would feel if people with entirely different political ideas dominated climate science and used their position. to try to maintain a kind of ‘politico-scientific correctness’.
.
No, I don’t really wonder about that 🙂
“but if embarrassing personal information is disclosed (and virtually none has so far),”
Except of course that all those bloggers that have received the access to the e-mails do also have access to this personal information. They now have access to personal and private details that they have no right to. Whether they publish this or not does not mean that the information is still private.
My employer allows me to use my professional e-mail for personal correspondence as long as this is not excessive. I have had several e-mail exchanges with my doctor regarding some medical tests. What right would Mr Mosher or similar have to read this information, regardless of whether they then publish it?
That’s what I call hypocrisy – believing it is OK to read peoples’ private correspondence as long as one does not publish it.
Louise,
Maybe so. Still, it’s a lesser damage than letting everybody on the internet read it. The world isn’t a perfect place. Do you have a better alternative?
For example, when processing an FOIA request through the normal course of things, ~somebody~ has to wade through emails that may contain private information that isn’t relevant to the request. They have no more ‘right’ to do so than anyone else. So, what? We disallow FOIA entirely because somebody has to separate the relevant from the irrelevant?
Intent matters at least a little. The whole point of the exercise is to minimize the exposure of private correspondence and nothing else.
Louise–
I don’t call it hypocrisy. I also don’t consider it hypocrisy to accidentally overhear someone discussing their medical condition at the local bar, coffee shop or on a bus. I once overheard a person discussing her psychiatric problems at a local Starbucks. She chose to post this in public. My ears were there. I don’t think I’m required to excuse myself to give her privacy when she fails to take care.
With respect to your hypothetical or real habit of interlacing private emails with your company emails: You seem to have figured out how to create and use a non-company email. I suggest you use that private emails for your private communications. Otherwise, when someone goes through a batch of business emails, it is no more fault their eyes land on a private email that it was my fault to accidentally overhear someone’s psychiatric woes at Starbucks.
(The things I have overheard people discuss in Starbucks, on the train or the hairdresser. OMG!)
No. It is not hypocrisy to read through company emails in order to redact those emails that one diagnose has having been of purely private interest. You’re just wrong.
Lucia – if you think reading correspondence not addressed to you and without the permision of the correspondent is the same as accidentally overhearing (as opposed to intentionally eavesdropping) a conversation not addressed to you then I understand why you don’t understand the meaning of hypocrisy either.
That’s still not hypocrisy. Another American who is English challenged, I see.
But you might want to work on your sentence structure a bit. Should it read “If you think reading correspondence not addressed to you ‘is okay'” perhaps? Hypocrisy would involve saying it’s “okay” here but not “okay” there.
(Like it’s okay for Gleick to steal Heartland documents but not okay for FOIA to release the climategate papers.)
Louise–
Exactly the same? I didn’t say two things were exactly the same.
But coming across otherwise personal information a person failed to keep private by throwing in the “business” pile while going through “business” records, is shares this similarity: It’s not the fault of the person going through the business pile to that they came across the private information. It’s the owners fault for failing to take proper care of information they considered private.
Someone sifting through business information to redact truly private stuff is not “hypocricy” simply because you wish to decree that the entire pile of business communication becomes “private” because some idiot toss in a batch of information she now wishes would be really, truly private and beyond the eyes of everyone but her and her physician.
The fact is: Whoever would be implenting FOIA at CRU would have had to read this stuff if the material had been requested under FOIA. Or it would be read by some lawyers should a batch of emails become involved in a law suit. You might not like who would be reading your truly personal private confidential exchanges with your physician that you stupidly mixed in with your business exchanges. But someone other than your physician — i.e. the person to whom the email was addressed– would, and this would violate no medical law and be no ethical breach.
For what it’s worth: I don’t even think intentionally eavesdropping on an conversation two people are having three bus seats away from me would be unethical. If you do, you are going to have to explain why. But I think if people want information kept utterly totally private, they shouldn’t mix it in with their business communications and they shouldn’t discuss in earshot of others and they shouldn’t put it in business emails.
The fact is: lots of individuals do share medical information. For example: My father-in-law had cancer. His friends knew it. Nothing about our medical laws or ethics says that other people have to keep that information “private” never emitting a peep to others. If you want your private stuff kept private: Keep it private. Don’t try to use that as an excuse why people there is some ethical problem with people looking through a pile of business emails.
Carrick–
I suspect Louise would be another Brit who is English challenged.
Lucia, Carrick,
But actually knowing what words mean is so hard! Easier to just use them loosey-goosey and make up your own definitions, then critique people based on those made up definitions. Lots easier.
Using your business email for personal communications is a bad, bad idea. In the U.S., the Supreme Court has ruled that whereas your employer may not bug your phone without a warrant, they may do whatever they please with your work email. Your work email is considered to be the property of your employer. This makes no sense to me, but that is the law of the land.
.
On the technical side, it seems to me that Apache Tika was made for the purpose of creating searchable indexes out of data like these emails:
http://www.abcseo.com/tech/search/integrating-solr-and-tika
.
We have Solr indices at work that contain many millions of documents. Two hundred thousand emails would be just a tasty snack.
My first thoughts were that FOIA had been gotten to. An innocuous release to appease. But now…..? If I were Mosher,et. al., I would not assume too much.
Louise, You are obviously not clear on fundamental ethical distinctions. The emails that were “stolen” were generated by public employees in the course of their duties. Therefore, under freedom of information laws they are public property. The fact that the University of East Anglia managed by various unethical arguments and practices to deny their release does not change that. If you carefully read the login message on most government computing systems it clearly states that this system is soley for the conduct of government business. There is no expectation of privacy and there should be none. My employer has in their login message a similar statement. There is nothing unethical or unclear about it.
If you want something to remain private, use your own private email account for it.
Louise, if you think that somehow your employer only archives the “business” emails, you are deluded.
Sure, they ‘allow’ you to use the email system for limited personal use, but I guarantee they still own everything you type as a work product.And there is no possible way they could remove the “personal” data from the archives without someone in IT reading them.
I for one would feel just at uncomfortable, ff not more so, knowing that the HR and IT departments where I work had access to my private medical info. than largely anonymous, probably fairly ethical, climate sceptics.
To Louise’s point I found myself laughing while perusing the CG1 and CG2 releases at the disclaimers put into the emails from the corporate and legal departments of various entities.
.
The disclaimers all said that if you were not a named recipient of the email it needed to be deleted immediately. Inevitably, the CG cache were all forwards to people not on the original To: or CC: lists.
.
Always write emails with the presumption they will be sent along elsewhere.
Louise, in her defense of deliberatley ignoring cliamtegate e-mails, is either a naive fool or a cynical idiot. These items are in the public arena. They are authentic. They are of relevance in judging the credibility of the authors and those who back them. They are not discussing private family health or financial details.
Feigning some sort of moral outrage on this is ludicrous. Using the arguments Louise chooses to rely on is more than a wee bit desperate, and transparently phony on her part.
I see that Louise is getting ‘hot under the collar’ about ethics, does this represent confirmation that, at last, there’s something getting warmer in the 21st century…
bugs–
How is speculating lobbying might be going on a conspiracy theory?
lucia (Comment #111471)
March 18th, 2013 at 1:41 pm
bugs–
No conspiracy theories here. No sir.
How is speculating lobbying might be going on a conspiracy theory?
—
Don’t let it bother you Lucia, bugs is part of a conspiracy. 😉 Anyone feeling furious yet or is more recursion in order?
Lucia:
See SteveF’s comment.
I’m not sure how well it applies to the Englished-Challeged Louise, but bugs seems to be one of those who’ve never met a word he couldn’t bend the meaning of to further (in his mind anyway) his rhetorical position.
Carrick:
I challenge this English! 😉
j ferguson… “bring it!”
😈
Carrick,
Ah, er …. Your choice of Weapons? Maybe Strunk and White at 50 meters?
Or maybe we can hurl our favorite sentences across a pitch, the ones which are so convoluted we can’t understand them ourselves.
If it is to be that, I concede, your’s of 111476 is unbeatable – world class. 😉
None of this is to suggest that Louise actually has a grasp of the thing. She clearly doesn’t.
Trying to post at RealClimate is evidence for hacker, isn’t it?
MikeN: Because of the “trying” part or the “RealClimate” part?
MikeN–
It’s evidence he tried to hack into RealClimate (and even seems to have succeeded) . That’s not necessarily evidence of having hacked CRU nor of being a skilled hacker.
Did not the FOIA emails have RC passwords laying about in them?
If the FOIA email release was an inside job or were left unsecured, just going through the email may have given access to RC.
If so, I guess it is still a “hack” to get into RC as it was un-authorized, but not to the level any self respecting true hacker would acknowledge as such. The hacking skill level would be a bit low to just key in a known password.
I was about to post a comment similar to Ed Forbes’ above. I think of “hacking into RC” as applying some effort specifically to gain access, e.g. a dictionary attack on passwords, phishing/social engineering to obtain passwords, installation of keyloggers, use of known exploits on their server or blog software…
It seems more likely to me that “Mr FOIA” had the RC password dropped into his lap from a Climategate email. [And thought it a fun lark to post the emails on RC.]
It’s not proof, but it is definitely evidence.
Can we rule out that the letter is a false flag operation, and not from the real FOIA?
MikeN what is the easiest explanation of the letter?
Any papers retracted yet?
I’ll check back in a few days. Basketball.
Boris (Comment #111522)
March 19th, 2013 at 6:37 pm
“Any papers retracted yet?”
.
Nah. And OJ Simpson never killed anyone.
@lucia (Comment #111471)
They might be lobbying? For what? Money? To bring down capitalism? To cause a mass extermination of the human species because they hate people? I have seen all these reasons suggested around the blogs.
Why do you have to speculate? Because it’s a secret and you don’t know if they do, so you have to guess because they won’t tell you.
bugs–
Since when is lobbying secret? I don’t know where you are but in the US lobbying is legal, done all the time, and for many reasons. The reason I don’t know specifically who lobbies is I haven’t looked into it. I think it would be interesting to look in the files and find out who was involved in lobbying and in what ways.
Of course it may turn out that some people involved in lobbying will prefer their roles be kept invisible– but it’s public behavior and the public has a right to know who is lobbying. (At least that’s the way laws work in the US.)
For bugs –
“As of November 2009, there are approximately 40,000 registered lobbyists at the state and federal levels, nearly 15,000 of which are federal lobbyists. Many of the major lobbying firms and advocacy groups are located on K Street in Downtown Washington DC. ”
http://dc.about.com/od/jobs/a/Lobbying.htm
http://www.alldc.org/
http://www.opensecrets.org/lobby/
Not only is it legal, it is written into the 1st amendment.
curious,
It’s a conspiracy I tell you. A conspiracy theory. People who believe lobbying goes one are all conspiracy theory nuts!! OMG. Call Stephan Lewandowky. But yet another of my “theories” in his supplemental materials. OHHHHH!!!!
What’s going to be interesting is if the FOIA release has some of the items that have been FOIAs which UEA claims were not held.
Lucia – one glimpse of the underbelly:
http://lobbyit.com/about/clients
Couple on there I might support myself!
8-0
So, what exactly is in CG3?. If there had been anything earth(?)shattering, presumably we would have had it by now.
Isabelle–
there are a modest number of people sifting through. Snippets are appearing places. If you wait for me to find something you are going to wait forever. The most mind blowing revelations could be in there, I’m not the one who is going to find them.
Good morning Lucia,
The question was rhetorical. But still, IF there had been anything golden nuggetty it would have been out by now. Unlike g1 and g2, where numerous sites, very publicly were churning out the contents, things are very quiet, bar the odd legal letters to Tall Bloke and Air Vent.
Isabelle–
Rhetorical questions are not permitted here. The rule is unique to my blog but I enforce it.
Rhetorical questions are impolite in general because they attempt to take the burden of clarity in making one’s point or burden of supporting it off the person who asked the “question” thereby disrespecting the time of other readers. So, the rule here is: If you want to ask a rhetorical question you must provide your “answer” to that question in the comment where the question is asked. That means you should have posted the discussion you put in “Isabelle (Comment #111561) “.
Obviously, everyone is permitted an infraction because the rule is unique to my blog. But — really– arguing by trying to make points by posing rhetorical questions is a very, very bad habit. I know that Socrates used them as part of the Socratic method– I’m not going to be so draconian as to suggest everyone who asked them be forced to drink hemlock.
As for what’s in c3– FOIA said most of the good stuff was in c1. So, it’s likely that we’ll get snippet slowly, and it won’t be as exciting. Moreover, lots of people are behaving like me. So, exciting stuff could be in there, I’m not going to find it. I’m willing to wait. In 6 months we’ll know a better answer to whats in there.
Was Draco the one who sentenced Socrates?