I think some of you will be interested to learn a tidbit about email repositories at UEA. But I’ll leave that as a teaser.
In comments at CA, Jonathan Jones mentioned the existence of a related ICO ruling on the following request which was refused (pdf):
On 14 August 2009 the complainant requested the following from UEA:
“1. A copy of any digital version of the CRUTEM station data set that has been sent from CRU to [a named individual] and/or any other person at Georgia Tech between January 1, 2007 and June 25, 2009
2. A copy of any instructions or stipulations accompanying the transmissions of data to [a named individual] and/or any other person at Georgia Tech between January 1, 2007 and June 25, 2009 limiting its further dissemination or disclosure.â€
The material in bullet 1 was identical to that requested by J. Jones. The material requested in bullet 2 is new. Naturally, I’m sifting through to find the aspects of the ruling related to bullet point (2).
First: in paragraph 5, we learn
“It explained that regulation 12(4)(a) applied to the request for stipulations accompanying the transmission of the data to Georgia Tech as no instructions or stipulations were held by the University. Any instructions or stipulations related the dataset were verbal and made between the parties at the time.”
(This is a bit ironic given the supposed great concern over possible loss of control of their IP on the part of the NMS’s expressed by certain parties over on other blogs. Clearly, the concern did not rise to the level of getting any sort of formal agreement before sending data any NMS might wish to commercialize to the unnamed party from Georgia Tech. )
The ruling on part (1) is identical to what we see for J.Jones case: UEA violates regulations. However, the ruling on part (2) upholds UEA:
2. The request for a copy of any instructions or stipulations accompanying the sending of the datasets
117. The second part of the complainant’s request was for “a copy of any instructions or stipulations accompanying the transmission of data…†to Georgia Tech. UEA explained that any such conditions were verbal and between the parties involved at that time. Consequently, the exception under 12(4)(a) applied to this part of the request as the UEA did not hold any information falling within the scope of the request. The complainant asked the Commissioner to investigate whether UEA had correctly dealt with this part of his request.118. In scenarios where there is some dispute between the amount of information located by a public authority and the amount of information that a complainant believes may be held, the Commissioner, following the lead of a number of Information Tribunal decisions, applies the civil standard of the balance of probabilities. In other words, in order to determine such complaints the ICO must decide whether on the balance of probabilities a public authority holds any information which falls within the scope of the request (or was held at the time of the request).
119. UEA informed the Commissioner that datasets A and B were sent to Georgia Tech once on, or a few days after, 15 January 2009. It confirmed that there was a cover email to the data sent but this email had been deleted from his personal computer by the member of staff concerned prior to the receipt of the request and in accordance with his usual practice of managing emails. However the exact date that it was deleted was not known.
120. UEA explained that emails for the CRU were held on personal computers and not held centrally. The personal computer for the relevant member of staff had been searched following the request for information and no cover email relating to the data sent to Georgia Tech was found. It confirmed that there was no overarching UEA-wide retention policy or schedule regarding such information. The member of staff concerned routinely deleted emails on a periodic basis.
121. UEA informed the Commissioner that the datasets themselves did not contain any information relevant to this part of the request. The data sent was communicated via email and any relevant information, had it existed, would have been communicated with the data, either in the cover email or within the data file itself. It stated that it believed that there was no information within the deleted email that would be relevant to the request for “instructions or stipulations accompanying the transmission of dataâ€.
122. Based on the information provided by UEA, the Commissioner is satisfied, on the balance of probabilities, that it did not hold any information falling within the scope of the second part of the complainant’s request at the time that it was made and that it, therefore, correctly applied regulation 12(4)(a).
I light of the climategate, I thinkit’s interesting to learn ” that emails for the CRU were held on personal computers and not held centrally” and “It confirmed that there was no overarching UEA-wide retention policy or schedule regarding such information. The member of staff concerned routinely deleted emails on a periodic basis.” Who’d a thunk?
I’m have no computer espionage skills, but I have to imagine that lack of a central repository or back up and staff members who routinely delete email must have made compiling a whole bunch of CRU emails more difficult than otherwise. Maybe some of the more experienced computer types can suggest how it might have been done under these circumstances. ‘Cuz honestly, I’d like to read speculation on this!
The other side of this exchange should be easy to clear up now. We can ask Peter Webster.
Means it was highly un likely that an outside party was responsible. Not that that was ever the most likely case.
I wonder what steps, if any, UEA took to secure personal computers in the wake of Climategate.
If they routinely deleted emails, how did the old emails in the Climategate dossier exist? Stranger and stranger.
All email servers store the emails. It is the client application – usually by configuration settings – that determines if the email is downloaded and deleted from the server, or left there. The only way that they could say all email is stored on the local PC’s (the client application) is if CRU rigorously enforced downloading configurations and locked down all clients so it could not change. Very unlikely.
The other main limitation used is to restrict the server side mail box sizes (in bytes). This limit forces the user to delete/download or beg for more space.
The email server was probably backed up regularly (most certainly daily). Thus, there would be tapes with emails on it. The tapes are most likely kept for 2 – 4 weeks before re-use.
It would be very easy for someone with the correct credentials to capture the emails off the email server over time regardless of download/deletion policies.
It would be very hard to do it in a one shot break-in.
Emails are routinely backed up on servers. Even if the UEA climate guys deleted their emails, copies would still exist on the server. Kan, I used to work at a central computing facility as an opt out from part of my graduate teaching load. I can guarantee that we never deleted emails, or overwrote the tapes for those. (They didn’t take up much space in those days, so that wasn’t a huge problem.)
I will assume that CRU is outright misinformed or just producing a narrative that technically, but not substantively is a truthful response, when they claim that the emails were stored in PCs and not central servers. Anybody who claims that the files were stored locally either doesn’t understand very much about how SMTP works, or is being a bit over clever (and dishonest) in his response. While you can actually do it the way they appear to describe it, howeverI see absolutely no evidence that any of the CRU guys have the technical expertise to set up and run a bullet-proof POPD or IMAP server. Plus you get none of the “value added” by having the emails go to a fixed IP address. Unfortunately the raw headers appear to be missing from the files, so we can’t generally answer at to what the delivery mechanism used was. I also doubt the central system would give permission for this… using poorly set up SMTP servers to reflect spam email is a common exploit, which is why it’s nigh impossible to get permission to do so.
Steve McIntyre #78211
Good point. Seems possible the climategate ‘hacker’ had been squirreling away emails for a long time…
Now that the ICO’s decision is in the Public domain, I’ll “out” myself as the instigator of complaint FER0280033.
Here is some of the comments I made when responding to UEA’s
statements to the ICO. (UEA first, my response, second)
This is a very legalistic answer which is basically saying that CRU, in its official capacity, did not send this data. It does not however mean that an individual (Professor Jones), or individuals within CRU, acting on their own, did not send this data. Of course, since Professor Jones is an employees of UEA/CRU, then UEA is responsible.
If this was indeed the case why then did Mr. Palmer, when rejecting FOI requests, not inform those making the requests where this data could be freely obtained?
I find this statement astonishing because, whilst they initially said that this data was not available, they have managed to find it (despite the fact that their servers are being held by Norfolk Constabulary) and send it to you.
This is a circular argument. They are using information, which is not Publicly available, to produce a graph, to support their contention that the 80% of data released gives the same answer as the 100%. How can this be checked without 100% release?
UEA go into lots of detail about the supposed confidentiality of this data set, however this is a red-herring. What needs to be remembered is that when these FOI request first came in one of UEA’s responses was that the data was subject to third party (Country) confidentiality agreements. Only later did they admit that these “agreements†were verbal, or that the paper versions had been “lostâ€.
It is worth noting that UEA has not subsequently provided any examples of confidentiality agreements which they supposedly held.
In short, despite UEA’s previous claims that the data is held under confidentiality agreements, UEA (with 1 or 2 two exceptions e.g. Trinidad & Tobago) still has not provided any hard evidence to support this claim.
Furthermore, despite continued UEA claims that this data is held under a strict duty of confidence and that releasing it would be an “actionable Breachâ€, I note that it has managed to release it to you (Appendix A and B) and apparently to others (Peter Webster).
I added blockquotes to distinguish claims attributed to UEA from Don’s responses. – Lucia
I agree with Carrick #78214. Email goes through a server somewhere. Usually smtp/pop3, but there are other protocols. You can (of course) configure the server to clear older sent mail, and in the case of received email, you can usually opt to have it removed from the server as it’s downloaded to the client.
In my experience, email usually gets left on the server, because it’s the easiest way of doing archiving and it contains information that no-one would want to risk being permanently deleted.
“UEA explained that emails for the CRU were held on personal computers and not held centrally. The personal computer for the relevant member of staff had been searched following the request for information and no cover email relating to the data sent to Georgia Tech was found. It confirmed that there was no overarching UEA-wide retention policy or schedule regarding such information. The member of staff concerned routinely deleted emails on a periodic basis.”
I found this extraordinary when I was making my complaint and I still find it extraordinary now. As a number of commentators have already pointed out, all emails going in and out of an organisation like UEA (and indeed my own) go through a central server. Everything I receive, or send, via email is recorded centrally, irrespective of whether I “routinely delete” it locally on my “private” PC or not.
So are they really saying that staff PCs, presumably paid for and installed by the university, were not connected externally via a central server?
Or are they saying that staff could routinely bring in their own personal and private PCs and use them to send material, which in itself could be the property of the University?
If either scenario is correct, it would appear that UEA had unique procedures for maintaining data security.
Lucia – I posted the following at Bishop Hill in December. Still remains my view of what probably happened.
If people look at item 6 in this CRU report then they will see there is reference to 3 unnamed researchers and it looks that the Climategate compilation was probably from those 3 only plus anything else of interest found on the server.
http://www.cce-review.org/evidence/Report%20on%20email%20extraction.pdf
The use of thematic inboxes by 2 of the researchers is a clear indication of how the leaker/hacker would have been able to compile emails of interest relatively quickly whist avoiding content of no interest.
Because the server was acting as an email backup for the 3 researchers then we know it was networked which still means an external hack may have been possible. We dont know the physical mode of transport off the server ie did the person have direct physical access or only via the network. I still favour an internal leak myself for other logistical reasons but cant rule out a hack.
What the UEA stated was there was no overarching etc with respect to the individual’s computer/email. This does not mean tape storage for the system was searched as written, to my mind. The climategate emails indicate that UEA is parsing words again. The UEA appears unresponsive to the “all” part of the request. Perhaps Don should point out the irregulaities comparing what the UEA claimed and the Climategate emails showed what UEA is doing; that would indicate that the ICO should have found on the balance of probabilities, the evidence is that the UEA is being unresponsive to the EIR as in the first part that was ruled in Don’s favor.
For some insight into the UEA email infrastructure have a try with google:
east anglia university hp server email infrastructure
I found a linkedin entry with some interesting detail that confirms, at a minimum, that MS Exchange is used and clustered. No certainty that this is includes CRU but at a minimum they have real email systems installed.
We also learn that
.
bob
Given that the Freedom of Information Act 2000 came into force after providing all of five years for public bodies to prepare for it, I wonder if the ICO should have something to say about the UEA’s claimed absence of any Data Retention Policy a full nine years after the Bill’s passing.
Following Climategate a spokesperson for the University of East Anglia said: “We are aware that information from a server used for research information in one area of the university has been made available on public websites ….. and we took immediate action to remove the server in question from operation. We are undertaking a thorough internal investigation and have involved the police in this inquiry.”
Yet:
120. UEA explained that emails for the CRU were held on personal computers and not held centrally.
On ‘the balance of probabilities’ which statement is correct?
What CRU describes is simply not the way it is usually done. While it is possible that emails were downloaded and resident on local PCs, it’s doubtful that they were erased from the servers at that time. This is what happens with Internet Service Providers, but not with organizations. Additionally, servers are usually backed up on a regular basis and some of those backups are permanantly archived. Also, in my organization everyone’s PCs were automatically backed up every day with a delta copy of any files that had been changed on that day. Unless CRU had a highly unusual system, what they claim is unlikely.
Perhaps they just meant they were using a PC as their mail server, and this was kept locally separate from a campus-wide system.
Lucia, the Mac version of the package RghcnV3 has now passed testing.
“Perhaps they just meant they were using a PC as their mail server, and this was kept locally separate from a campus-wide system.”
This is a security breach of gigantic proportions. A hole in the firewall which would not be allowed by any sane network administrator.
As per usual I’m somewhat late to the show.
For those speculating as to what email system CRU used prior to Climategate, here are some useful links.
http://ccgi.newbery1.plus.com/blog/?p=317
http://www.cce-review.org/evidence/Report%20on%20email%20extraction.pdf
It would appear that Prof. Sommer (the independent consultant hired by UEA to
“I am asked by the University of East Anglia to look at the back-ups of the computers of the key researchers in CRU as they are held on the back-up server to see if it is feasible to identify email traffic which was not publicised on the various websites, but nonetheless related to the same issues and might justify further investigation by the Independent Review into the publication of the emails and the allegations of inappropriate scientific and other practice which had subsequently been made.”
is saying that the files supplied to him by Norfolk Police retrieved by Qinetiq (the ‘computer forensics’ consultants hired by Norfolk Police but paid for by UEA to examine the CRU email server in the form of three pen drives) where in ‘Thunderbird’ format.
If you read the PDF link to in the link above you’ll see that Prof. Sommer was unable to accomplish his brief as too much work would have been involved given the short timescle he had to accomplish his task.
It is evident though give the size of the data stored on the three pen drives for researchers A,B an C that unlike the ‘released’ Climategate emails, these files must have contained copies of any file attachments. Prof. Sommer was only allowed to analysis these files in a ‘secure facility’
“The material has been given a very high level of security classification which requires that I work at secure facilities and follow particular protocols which, for example, preclude computers being left to run unattended or overnight and at weekends”
so until such time as these restrictions are lifted by Norfolk Police no one else will be able to examine them to see what files (and their content) were attached to these emails. For example we won’t exactly what files Phil Jones may have sent to Peter Webster (and indeed if he sent them via his UEA email address).