Based on emails from “contact lucia”, I know a few of you want to know what’s up. So, I thought I’d let you know, and then decree this an open thread.
The main thing that’s up: I have been diverted to dealing with ‘bots. This has been and will continue to be bit time consuming both because inspecting serverlogs/ htaccess and web security are not “my thing” and also because using shared hosting deprives me of many of the effective methods of creating a firewall. I won’t discuss that further because those don’t even know what the previous sentence meant will find additional discussion of why it’s time consuming boring.
However, I can report the following:
I expect that I will over time have to add some attack patterns, but I think the current ‘system’ should mostly work.
However, I now plan to turn the scripts into a ‘plugin’ for WordPress. So, for January, my posts will be limited to showing trends when the individual agencies — especially Hadley and GISStemp post. It’s year end after all!
The several people who have invited me to consider posts on things they find particularly interesting: My answer is “not today”. Two visitors have requested I add Scaffetta’s model to the tests. This is a good idea– but implementation will have to wait until later. (My thoughts are that there is little hurry to test it’s ability to predict or forcast. It’s only just been published. It makes more sense to start testing that when we start comparing AR5 models to data.) There have been a few other interesting suggestions– also “wait until later”.
I know some people think it odd that I make writing a plugin the priority since this blog is about climate. I could understand that view. However, I have reason to believe these ‘bots are wrecking havoc at other blogs, and also have a potential for dealing great harm both to those who host blogs and those who read blogs. Explaining fully would be too time consuming. Those who want to know more should read Brandon’s comment (highlighting mine):
lucia, you say:
That is an attempt to upload a script that would do something. What I do not know. Possibly inject viagra ads into my blog. Maybe it wants to turn my blog into a zombie-bot so it can make my bog try to inject viagra ads into your blog. Whatever it is doing, it is not good.
The last sentence is a massive understatement. I only spent a few minutes looking at what that was trying to do, but already, I can tell it’s “evil.†The impression I get is it would give them complete control of your server. They’d download every file on it and copy everything from your database. They’d then give themselves access to a variety of services on your blog, ranging from file storage, IRC channels, mail spamming and various other nasty stuff (though there doesn’t appear to be anything for using it in a DDOS attack). Even worse, I believe all of it would be done without (necessarily) altering anything which would affect your site’s appearance or functionality, meaning they’d have complete control even as your site keeps running.
I didn’t spend a lot of time looking into it (there’s a lot involved), so my impression could be off, but it definitely is worse than just trying to inject ads into blogs.
Those who might want to know what “various nasty stuff” could be– I’ll let Brandon, Kan or nay of the real IT people answer that when they appear.
Meanwhile, I am going to spend the rest of the week “plugin-i-fying” my ‘solution’– which will:
- Monitor and auto-ban IP addresses that attempt really nasty things there by improving security generally. The auto-ban is done at Cloudflare.
- The effect of the above is to dramatically reduce the load from ‘bots and therby prevent blogs from crashing because they are trying to serve pages to ‘bots. The main reason the blogs crash is that ‘bots are guessing 10s of thousands of possible addresses that would permit them to hack in if a blog was vulnerable to those attempts. Oddly enough, most of these attempts are wrong addresses and send the bot to the 404 page. My script analyses these requests and if they are attacks it banns them at cloudflare.
- One side effect is to improve security for visitors and blog hosts at blogs whose operators chose to use the plugin. The security will be improved because — generally speaking– a bot guessing 10s of thousands of addresses will usually guess wrong before they guess a particular uri (i.e. address) that permits them to hack in. When they guess wrong, my script banns them; oddly, my script does not catch them when they guess right. Other measures are required for this. But banning them when they guess wrong makes it more difficult — but not impossible– for bots to ‘hit’ the correct address corresponding to a vulnerability that does exist. If the ‘bots first guess succeeds in finding a vulnerability, a blog will get hacked.
I’ll be explaining this further when the plugin is released. In the meantime… I’m “plugin-i-fying” the script because it seems to be working. (Oddly, hitting “post” will give me more data because posting triggers bot attacks!) So…. if I might post an update revealing that I was mistaken and this doesn’t really work. (More likely, I’ll just discover new attack patterns!)
Oh…. tell me if you can see the pink-ish or yellow images. I need to know because I also need to deal with image scrapers. (You should not see the yellow.) (Grousing that I’m writing the plugin is permitted. It won’t change anything, but it’s permitted. )
You should see an image above.. |
You should see text only. |
And… open thread.
I wonder why you are being subjected to this? You are one of the kindest and most civil of the cliamte related bloggers.
I hope the attackers get bored with trying to overcome your kungfu and move on to other sites more deserving of having their works gummed up.
Best of luck in overcoming this.
respectfully,
hunter
the image on the left, btw, is a light pinkish color. The one on the right has no image, but it does confidentally state it is “yellowl”, in a simple font placed vertically on the right.
Lucia,
On the left, I see a pink box with an ‘X’ through it. On the right I see a broken-link-type box with the text “Yel”
I cannot ser the yellow and blue images, only a pink image.
Pieter
Tim W– Sorry.Yes. I named the file “blue”. It’s really sort of pink! Let me change that
Hunter–
It’s mostly just ‘bots. I could explain more, but that takes away time from implementing the plugin.
Bots never get bored. The best you can do is create a firewall. Shared hosting means I can’t create the type of firewall some of the IT guys visiting keep suggesting I create. But one suggested cloudflare and my script is letting me use it as a firewall.
Lucia-
If I had a question it would be the same as hunter’s, but perhaps from a position of less understanding…. so an answer would be wasted on me…
*
I have a pink square with a cross on the left, and just the text YellowImage on the right, in bold and underlined.
*
Perhaps, actually, my question would be hunter’s in reverse – why are others not being attacked? Is it the case that they are and don’t realise it?
*
Anyway, all power to your anti-bot efforts. I’m surprised that the people at cloudflare or wordpress don’t have a proprietry solution/defence already on offer. It seems very much a defect/weakness in their product – don’t they have the resources to cobble together something effective themselves?
Lucia, I see what you describe above.
I hope you will be able to detail your bots wars in the future. I find these internet annoyances and more than annoyances very frustrating and wonder what motivates people to launch them. Even those that are connected to advertisement do not make sense to me as the annoyance factor should turn anyone off.
The internet is a major step forward in utilizing individual free speech and those people who promote these problems are definitely against free speech. It is like someone being able to enter my home and scream at me while I am attempting to communicate with the outside world.
I suspect too many of us just “put up” with these problems and do not attempt to do anything about them.
I cannot see any of the 4 images (there are 2 above and 2 below the text, right?).
I will. But it’s counter productive to spend too much time describing the method right now because it takes away time from monitoring to verify it’s really working, writing the plugin and verifying the plugin works. I’ll describe everything afterwards. But… it’s working.
The reason I asked about the images is that I want to be sure something I did doesn’t prevent people from seeing them and I also want to be sure a “bait” works to monitor scrapers. Owing to some cloudflare ‘features’ related to their caching I can’t be sure quite what’s going on unless visitors tell me. I’d rather people pipe up with the pink image and the yellow missing image than wait for an image intensive post be impossible for visitors to understand.
So…. lurkers– if you can’t see pink, let me know.
Now I can see the pink boxes at the top-right and the bottom-left. The other two I still cannot see anything, even text.
Kenneth
Maybe. But without advertizing, how are newspapers or bloggers to pay for hosting? They can’t. Donations will never be enough. Subscriptions doesn’t work for most newspapers.
I should run advertising. People could use ad blockers and that would be fine. But I should run it. I do run it at my knitting blog, and that covers costs of hosting this blog. But obviously, it doesn’t cover my time. That’s ok as long as we recognize it’s a hobby.
Still: Speech may be free, but internet hosting costs money. It’s got to come from somewhere.
Anteros
They are.
Those on WordPress.com blogs are protected by WordPress.com who filters for them. I don’t want to host there because– among other things, I want to run the uah betting. I also want to provide visitors with more plugins than possible at WordPress.com and other things.
Everyone who is selfhosted is being attacked. Some have been hacked into. I’m not guessing. I know of specific individuals. If you followed FrankBi’s comments, he mentioned he saw hacked climate blogs– so I’m not the only one seeing them.
This is why the plugin is a priority.
Cloudflare provides some defense already. They block about 1/4th of the spam. They have an API and using is an essential element to prevent the hacking. I’ve asked them to extend their API so I can report connection details. If they do, and write some scripts, they could potentially become very effective at blocking lots of this uhhmmm… sh*t.
A lot of this firestorm of sh*t started becuase of something called the timthumb.php vulnearbility which was only discovered around Sept. 2011. So, it’s pretty new.
WordPress.com does a lot to protect their blogs.
Wordpress.com hosted blogs are somewhat protected because:
1) WordPress.com didn’t ever let ANY use any plugins that contained the timthumb.php extention. This means smart crackers know not to bother guessing at any blog hosted at wordpress.com and
2) WordPress.com can detect and filter stuff because they run so many blogs.
But WordPress.org is just free software used by selfhosted blogs. The timthumb.php problem wasn’t caused by WordPress.org. People who are selfhosted specifically want flexibility. If you want flexibility, you can’t expect an entirely different party to come in and protect you. So, I can’t expect WordPress to swoop in and protect.
Weakness in a product available for free? Sure. WordPress is blogging software, not site security software.
Ok… the logs look ok. I’m going to go out and get my car emissions tested before the snow is 3ft deep!
I have always thought those involved in Internet based attacks (including viruses) ought to face some serious consequences, like public flogging to death…. OK, that is too extreme, but at least long jail sentences. Talk about pure destruction of value…. they are maggots.
Well… I saw the snow and thought it prudent to put off emissions testing and shoveled instead!
SteveF– Yes. They ought to face serious consequences. The difficulty is that many are in other countries, and also many can perfectly well operate using proxies. But some of the hosts that harbor lots of the attack bots are in the US. They may be within the reach of the law, but I have no idea how to get the law to touch them. I suspect the effort involved would be exceed that needed to get Phil Jones to share data!
With respect to my blog and my response, I’m going with technology. If someone wants to figure out how to legally flog these guys, it’s got to be someone who knows how and has the time to do it.
“Everyone who is selfhosted is being attacked.”
Have to agree with that. WUWT and CA are painfully slow at times. And, occasionally, all that my browser loads is the straight text, stripped of the html code.
Bot related, Possibly?
The snow’s just starting on this side of the lake. We’ll be breaking out the snowthrower tomorrow, for only the second time this year. Funny winter for MI.
WRT site funding – I’d buy more mugs but my cupboards are full! The ads on your knitting blog seem fairly unobtrusive. I wouldn’t be put off by them here.
“Still: Speech may be free, but internet hosting costs money. It’s got to come from somewhere.”
Believe me Lucia, you do not need to convince a free market capitalist of the legitimacy of advertising. What I am talking about is those malware ads that can show up on one’s computer (and effect its operation) if they do not seek protection. That is more akin to an advertisor taking a step beyond simply throwing advertisement in my drive but walking into my house and pestering me when I am attempting to eat my dinner. That brings into play my private property rights.
I do not mind at all the advertisements that can appear coincidentally with other information on the internet just as I do not mind seeing it on TV or in the newspapers. Malware (perhaps there is a better term for it) is another matter. Besides advertisements are primarily ignored by people of my age group and most are aimed at the younger people.
If a bot is slowing down or halting your website activity it is violating your private property rights in my book.
SteveF (Comment #88533)
Thanks for that post as I found it very cathartic.
Lucia,
I wonder if you could solve your problems by running two sites. One would be the regular blog, hosted by wordpress.com, say. My blog is hosted by Blogspot, and I don’t have bot problems that I’m aware of (and couldn’t do anything about them anyway). I can and do do a lot of scripting, but I can’t write information back to my own computer, or indeed automatically write info to the blog.
So I have a second site, which is just a web page, not a blog, and not exposed to bot activity. Actually, all I do is automatically upload information, but I think I could run scripts like your betting script. And I embed that in the blog page using a HTML iframe.
With the celebration of Joan of Arc so recently in the news, I have an image (Josh-worthy, I would hope) of Lucia of Naperville clad in shining armor astride her noble steed named Cloudflare, surrounded by the smoking carcasses of slain ‘bots.
Slay on Lucia!
Nick–
I might be able to do that. But it’s a lot of rigamorole futzing to make things work inside a frame inserted into a post. To some extent, I don’t want to do it that way each time. I’ve also got the knitting blog/site where I have scripts that let people create sock and sweater patterns. The bots hit that too– it’s not just here. Reorganizing that to work the way you are suggesting would be time consuming too.
At this point, I honestly think it is less work to thwart the bots than to go back and update all the zillions of “sockulators”, and “sweater-pattern-generators”. Maybe if I were just starting from scratch I’d see things differently. And maybe if it was just betting, I’d see it that way. After all, after betting is done, I don’t need that script to function in that particular post. But I’m not just staring from scratch and I want to keep the knitting patterns working too. (I get thank you notes all the time. I even get an occasional $3 donation!)
SteveF,
‘OK, that is too extreme’
Maybe it shouldn’t be public.
Kenneth
Those ads may have been injected by a ‘bot who hacked in. That’s one of the things ‘bots sometime try to do. Their motivation: money.
Maybe. Or not. But in the case of WUWT and TAV, they are hosted on WordPress.com. So, WordPress.com is the only one who can fix it. If they are under attack– and they could be– I’m sure they will figure out how to respond and ban IPs at a more global level.
BTW: WUWT uses it’s own domain name, so stoooopid bots won’t know it’s not self hosted. In contrast, bots with at least 2 braincells would know to stay away from TAV because it’s address ends with .wordpress.org, so we know it does not have the “timthumb.php” vulnerability.
I probably shouldn’t write “timthumb.php”. I’m convinced writing that attracts bots. But then… that helps me detect more methods of attack and I ban them!
“The snow’s just starting on this side of the lake. We’ll be breaking out the snowthrower tomorrow, for only the second time this year. Funny winter for MI.”
I live in IL not far from where Lucia resides and yesterday I spotted a sight that might appear rather contradictory to a non Midwesterner. A young lady clad in t-shirt was buying supplies for her snowblower. She was, of course, clothed properly and correct in preparing for snow – and cold weather.
Lurker from Japan here. I see the pink squares, not the yellow squares, only text.
Re: snow. We have well over a metre of accumulated/compacted snow lying and are getting 20-30cm fresh per day. We are well on course to exceed our average in town of 12 metres fresh snow per winter.
Kenneth Fritsch –
A young lady clad in t-shirt was buying supplies for her snowblower.
My brother moved to Chicago some 15 years ago (from Southern England, still my home) and a ‘re-interpretation’ has to occur when we talk about changes in the weather. When he says ‘it’s got a bit warmer in the last couple of days’, I have to think about our yearly range, and then add some…. And when I remark how cold it’s been recently he has to remember that this means ‘pretty much mild like it always is’.
It’s a very different world, but hey, we’re adaptable beings and I don’t think moving to Chicago was a ‘catastrophe’ for him. He quite likes it!
lucia,
Please do not hesitate to start running ads here.
By the way, where are the tip jar and Blackboard collectibles links?
I only see pink, the other one has text only and a ‘missing image’ question mark icon.
hunter–
The tip jar is inconspicous– it’s under pages marked as “donations”. I need to get the mug link up.
Kenneth Fritsch
Yes. It went from being unseasonably warm to a blizzard. The blizzard was predicted. We had our warm clothes and shovels ready, and the area was braced for snow. I shovel 4 times yesterday– mostly to make sure I never had to shovel really deep snow. But also, if you shovel, you want to get the wet show off before the temperature drops, otherwise it freezes. In this event, I knew that if I didn’t get off the first batch quickly, I’d have ice under snow. I succeeded in preventing that.
We shoveled this morning. It’s now frigid.
Hector–
My niece is teaching English in Japan until May. One of the other teachers told her a little girl was looking into her class room and asked “Is she American?” The other little girl said “Yes.” Then the first little girl said, “She can’t be. She’s not fat.” 🙂
Evidently, many of the Japanese are just ga-ga over Maggie. She’s blonde, cute, not fat….
Anteros-
It’s 19F (-7C) now. They predict 1F (-17C)tonight.
“I shovel 4 times yesterday– mostly to make sure I never had to shovel really deep snow. But also, if you shovel, you want to get the wet show off before the temperature drops, otherwise it freezes. In this event, I knew that if I didn’t get off the first batch quickly, I’d have ice under snow. I succeeded in preventing that.”
That’s the retired guy’s strategy where he has all day to shovel. I shoveled once in the morning and waited for this AM to use the snow blower. I was worried about the condition you noted of ice under the snow but that did not happen in my neighborhood.
Even went for a walk this PM and promptly fell on my a–. A lady asked me if I was alright and I told her I was – just a little embarrassed. When I was younger I could quickly pop up from a fall before I was spotted. This time I had to take a knee and I do not mean a Tebow knee.
@Lucia. I hope your niece is enjoying her time in Japan. I like the children here. I teach English part-time. It’s a struggle, I’m a geologist, not a linguist.
The 12 metres of snow here (northern Tohoku) is no exaggeration. In winter the prevailing wind is NW, coming from the Siberian High. This arctic airstream crosses the Sea of Japan (about a 1,000km fetch), loads with moisture, and dumps on the W/N coast of Honshu. It’s a mega- version of North America’s Lake Effect. It starts snowing in late December, and stops in early March.
I clear snow twice per day, typically a 2 hour session starting at 6am, and an hour in the afternoon. The system we have is the town pumps river water through the stormwater drains three times per day. Dump your snow in the magic hole, and it’s gone. Like magic 🙂
Hector–
She likes Japan very much. But she also wants to come home. She’s returning in May. She’ll be enrolling in grad school to study marketing.
Chicago used to dump all their snow in the lake but the FEDs don’t permit that anymore. Seems Wisconsin, Michigan and Indiana don’t want Chicago to dump their snow in the lake. Imagine that? ( Likewise, Illinois doesn’t want Wisconsin, Michigan or Indiana to dump their snow in the lake. Chicago was never the sole offender. Dumping in a big body of water is convenient.)
Hector Pascal (Comment #88601) –
Northern Tohoku (Aomori Prefecture?). Hmmm, sounds like you may have had … too much… excitement in the past year, being only ~250 km from Fukushima. I hope your city was spared the worst of the tsunami and aftermath.
Bovine belching is what contributes, not insignificantly, to the methane in the atmosphere. Grass feed cattle produce more than those in a feedlot.
@AMac. Thanks for your concern. We are in northern Yamagata (meaning “mountain shaped”) Prefecture, about 70km west of Sendai. The earthquake was very scary. All you can do is hold on to something and wait, never knowing how long it will last, and how big it’s going to be. It was long and big. Locally we escaped any significant damage, but needed to do a lot of clearing up.
Power and water went off for about 3 days. I was able to fill the bath before the water stopped. Water is the No1 problem, but we were OK and I was able to collect snowmelt from the roof. Sitting in a cold dark house, warming yourself in front of a candle is no fun. Renewables enthusiasts should try it, say for a winter.
We had no fuel supplies for about a month as the tank farm at Sendai was destroyed and about half the tanker fleet washed away. The east coast arterial was closed so supplies had to be trucked in up the west coast, the long way round, and across two mountain ranges on difficult roads.
Yamagata was spared the radioactive fallout. It’s destroyed the livelihoods of all the farmers in Fukushima as well as countless small businesses. We were lucky. No real damage, and my partner’s business has survived, albeit at a thumping loss for the year. The government has intervened, providing low interest loans to keep small businesses afloat, but it’s still debt and will have to be bootstrapped away. Not easy with a dull economy.
Sounds like another brave whistleblower in search of the truth.
I’m saddened to see you fighting the release of your data and your code. What are you hiding?
Robert–
Clearly you either haven’t been paying attention or you lack the ability to comprehend what has been discussed. Almost everything on the server is publicly accessible already– though I admit that the IP addresses you — Robert– use to connect and the email you Robert use to connect etc haven’t been made public. But I guess I don’t mind all that much if the bot gets that info.
But the main problem is that the ‘bots
1) Try to download so much so rapidly they crash the blog making it inaccessible and
2) are likely to try to control my server so they can do this to other people’s blogs and
3) After controlling the server would upload malware, ads and all sorts of other things presenting a danger to my blog readers etc.
This has very little to do with my not wanting to hide any data or code.
That said: If you like, I’d be perfectly happy to release your email address and every IP address on record for your comments including time and meta data indicating any revisions. I mostly keep that private as a courtesy and laziness. But I’m willing to overcome my laziness and dig yours out if you want me to release yours.