Announcement 1: GoogleApp Engines mostly blocked
As some of you know, I recently began banning TOR connections. I’m doing this by reading the IPs of new TOR IPs and banning them at Cloudflare. Naturally, every spam/hack control attempt has a nearly equal and opposite reaction. I now see hack/crack attempts using other quite obvious anonymous proxies. These include connections using proxies set up on GoogleAp engine. (This has been named “CrapEngine” over at webmasterworld.) My method of thwarthign these is to ban any connection with “http://code.google.com/appengine; appid:” in the user agent except for whitelisted apps. There are two whitelisted aps are pubsubhubbub and s~feedly-social. Many other app-ids that have hit have clearly been proxies.
But the purpose of this post isn’t to complain about those. I’ve blocked them. My purpose is to ask for help in figuring out which google aps I should whitelist. To that end, does anyone know:
- Is there any central archive where I can learn what service any particular google app provides?
- Failing that, do you know of good google apps that I should not be blocking?
These things are often free. New ones are sprouting each day. Clearly, the only way to approach it is to block an app until it is proven “useful to me and my readers“. Any help identifying those would be welcome.
Announcement 2: Blocking speed demons commenters.
Commenting now uses cookies to compute the wait time on a page. You must turn them on to comment. (They are not required if you read only.)
The cookies are not so much to prevent spam (which as you can tell already doesn’t appear) as to kick off the bots that trigger the spam filter and then just try to add another spammy message over and over. Along with these feature there are some secret hidden comment forms sprinkled around the blog. Bots that fill those out are getting banned– and quite a few are filling them out.
As for humans: The comment filter will block you if you submit a comment less than 3 seconds after the page loads. Although it is possible for a human to refresh the page, scroll down fast, enter a 1 letter comment and hit submit in under 3 seconds, I don’t think anyone will accidentally trigger this here. You will probably be able to just use your back browser and hit “submit” again. But these infractions accumulate– and of course, it’s a PITA to fish people out of the spam bin. So please do not experiment with this just to see how it goes, you could end up banned at Cloudflare. That said: If you see the comment warning, let me know. If it’s catching actual humans, I want to know that.
Announcement 3: Anonymous Proxies and Forwarding
ZBblock previously only monitored the final IP in a connection that showed a number of IPs in the X-Forward headers. I have been experimenting with a few extra checks on those connections with more than 1 IP in the X-Forward headers. This has “caught” a few innocent people– mostly because I was a bit too strict on people in “spammy” countries (e.g. China, Thailand, Brazil, Ukraine etc. Note: Australia sometimes gets ‘caught’ because some of you guys are identified as coming from various parts of Asia. ) I’ve backed off on some of my boneheaded mistakes with forwarding– but I am still checking those more intensively than non-forwarded connections. I’ve also discovered some people using proxies don’t know they are using proxies! (This can happen at work and so on.)
If you are using a proxy you might see a “Ban” message it may be that you are — knowingly or unknowingly– using a proxy. The ban may kick in either because of your IP or because the proxy IP appearing in the X-Forward headers is banned. I ban many proxy services– especially the cheap 6euro/month anonymous proxies; they are a constant source of hack attempts. I mean hack– not spam. I’m sure they spam too though. So, if you have been using those to read the blog and no longer can read, you will either need to use your real IP address or find a proxy that is not blocked.
Open thread– talk about anything including Climate Change. 🙂
UPDATE
The word ‘slimy’ has been removed from the referrer spam ‘bad words’ and I will fish out all the people who tried to visit by clicking a link over at Bishop Hill’s ‘slimy’ article.
Update 2
I’m turning AKISMET spam filtering off for an hour as an experiment.
Update: My blog requires appropriate ‘referrers’ to be passed when you comment. The “appropriate” referrer that should be passed when you hit “submit” for comments on a post is the uri for that post. So: if you comment in this post, the referrer should be “http://rankexploits.com/musings/2012/three-admin-announcements/”. Passing this referrer is the default behavior of most browsers. But some people read (often bad) advice by “privacy” advocates and turn referrers off on their browsers. (It’s easy to turn them off.)
That said: If you are not passing an appropriate referrer, and you submit a comment the spam filter will think you are a spam bot. To fix this turn off the ‘fake-privacy’ filter you are using and pass a referrer. Because if my blog thinks you are submitting comment without visiting a blog post, it’s going to think you are a spam bot programmed to “post” to a variety of addresses without ever visiting posts or reading them. It will think this because that’s what spam bots do and you have modified your browser behavior to make it look like a spam bot.
Also note: It’s not just my blog. If you use “privacy” filters of any type and find you are having difficulty commenting at other blogs, it could be your “privacy” filters are making you look like a spam bot. Because — depending on the choices– it might just do that.
Following you tweaking and managing this website/blog is almost as interesting as the climate topics covered.
Best wishes for a great Thanksgiving to you and your family.
Kind regards,
Thanks hunter.
I actually “deploy” a lot over at “the most boring blog in the world” first.
blog.bannasties.com
Fiddling there, I know it’s possible to load a page and enter a short comment in less than 5 seconds. ‘Cuz I banned myself.
I’m also testing out a ZB-block/comment WP plugin interface that will enter IPs for various types of comment spammers into the ZB_block log. (I then ban them at cloudflare.)
You’ve probably noticed comments auto close here. What you certainly don’t notice– because you don’t see the logs– is bots try to post comments at those posts quite regularly! There’s no comment form on those posts. So…. I figure there is a 100% chance all those IPs are ban-worthy. I just rigged that up and it’s functioning over on the “blog no one does or should read”.
Good luck Lucia ! And happy thanksgiving to everybody 🙂
I wonder if any IPv6 to IPv4 tunnel brokers or gateways will show up at some point. Some of those are sort of anonymous.
MrE–
I’ve seen 2 or 3 hits with the newer address forms. I sent in a ticket to Cloudflare asking them to get ready for that so we can ban them when we start to see those.
Currently though, I see things like “ipredator.se” or “anonine.com” etc. There are tons of services. If you monitor hack attempts at a blog, you don’t even need to hunt them down. The hackers get subscriptions and you see the services in your logs. 🙂
Not every using these services is a hacker. But I suspect the hacker/normal user ratio on anonymous proxies is higher than average. And I know all of these people could just use their normal IP to access if they wanted to do so.
Ok… but this is the sort of thing the “flash-gordon” screen is catching:
(Note: My “why blocked” are not well organized. They never will be. )
This thing loaded the page “http://blog.bannasties.com/controlling-exploitative-bots-aka-nasties/” scrolled down, and hit the “submit” button in 0 seconds. 0!!! There wasn’t even time for the click to click over to 1second!
BTW: Even with all spam filters turned off, no spam in hours. I started running the “Flash Gordon” spam filter last night, caught a whole bunch quickly, and now there is very little. ( Historically, I’ve noticed something works quickly…. then seems to be ‘victorious’ and then the spammers may adapt. It sort of depends on whether adapting is worth it. It’s often not so the absolute level of spam/hackery does drop. )
Nice tweaks – have a great Thanksgiving and long weekend!
Lucia,
Loading RSS feed with IE also seems to cause the banishment.
Phi–
Can you send me the information in the error log? I need to read it to see which of my “rules” is misguided. The most important thing for me is the the “Event ID” which will have a numerical value somewhere in the range bewteen 100000 and 200000. Otherwise just knowing that loading RSS feed with IE causes banishment is not enough for me to figure out what’s going on nor to fix the problem.
Lucia,
I just try again. No problem this time. It might be something else.
Eli does believe you are enjoying this.
The IPv6 address forms themselves will not be noticeable from IPv4 (old address system) since they are not compatible and need a translator or gateway. IPv6 can be tunnelled encapsulated in IPv4. Unless you are also concurrently running IPv6 which is possible but you would probably know it. This dual stack of running both protocols is becoming more common but still not wide spread and impossible if your ISP doesn’t support it – unless you set yourself up to use a IPv6 tunnel broker. Once you do use IPv6 there will be much less scanning seen since the addresses are 128 bit rather than 32 bit. However, you could see the common IPv4 address that are tunnel brokers or translators but I doubt they are very noticeable unless you search. Some of these are somewhat anonymous but the IPv4 address can be embedded in the IPv6 packet.
Cloudflare is growing quite fast and will certainly be on top of it too if it becomes an issue. It looks like they have an IPv6 gateway product and you can test your site for IPv6 compatibility
http://blog.cloudflare.com/introducing-cloudflares-automatic-ipv6-gatewa
Also the new DNS names then that would also something to be aware of for next year – if it actually happens.
Eli–
Now that the blog doesn’t crash, it’s more “good” than “bad”. I don’t like the fact that I occassionally ban people. But…. some people.. (I banned a woman who claims she was only trying to visit a “sockulator”. I looked into it and “somehow” she managed to request 20 blog posts in 4 seconds. How… dunno. Might be innocent. Might be “social engineering” trying to get around the bot blocks. Anyway, assuming innocent, I told her to turn off any pre-fetchers she might have. If she’s a bot-master, she’ll just have to learn to set her bot to *pause* between requests!!)
I admit I find it interesting to see what the “bots” try to do. The logs for the “blog.bannasties.com” blog are especially illuminating. When I started screening the X-Forward headers, I started to see all sorts of compensating stuff that clearly involves bot-masters who really, really, really wanting to see which IPs have been banned and for what reasons.
For example: IPs are trying to “see” the site by looking at Google Preview, google proxies, anonymous proxies and so on.
But I see similar attempts to “see” this blog– but I rarely see similar stuff at the knitting blog. Go. Figure.
(I’m keeping Google Preview operating here. I’m not letting it show the “bannasties” blog. )
I saw that at cloudflare. I didn’t experiment clicking it.
I admit to having no idea what’s going to happen as new DNS names kick in. I assume people like Zaphod who writes ZBblock will be monitoring which new DNS names are “stinky”– same as always. So, the list will change (or get longer. Sadly.)
The investment in a toplevel domain is sufficiently high that I assume spammers will not want to invest the ‘.spammer’ domain and sell domain names to all the spammers who want to be easily identified!
Currently, Cloudflare does not let me write a blog for an IPV6 type address. So… if I’m not mistaken (and I easily could be) there is little advantage for me to be able to “see” those. Or… at least I don’t want those values to be a way to sneak around Cloudflare. (Is that possible? Anyway, I told them when I saw a few showing up in logs. Later… that stopped.)
The computer and network card would need to have both protocols if you saw IPv6 since they(IPv4 and IPv6) are incompatible. Having both protocols is called dual stack and the ISP would have to have them both supported as well as IPv6 DNS and maybe you already know that. Dual stack or using both protocols will become more common as IPv4 addresses supplies have already been given out to ISPs. If that was the case, i.e. that you have dual stack, you probably could find out what IPv6 address they have assigned you.
You can tunnel one over the other but that would have to be in data part of the packets and then your computer needs to un-encapsulate the packet.
I don’t know if tunnelling around Cloudfare is possible (or relevant) because I am not sure how Cloudflare’s network works. Most content delivery networks(CDNs), two years ago at least, used to have server farms spread around the world and the biggest ones even deploy servers locally inside your local larger ISPs since it is mutually beneficial to do so with them. However, Cloud-fare is not a typical CDN and has not been around long. I see they took part in the IPv6 day this year so they must be at least planning on customer solutions for it. Many ISPs are using IPv6 internally but have not rolled out end customer solutions yet.
I used a free tunnel broker a couple years ago at home from sixxs.net, or somewhere like that, just to try IPv6 out. Maybe there are more current methods. Its was pretty easy to set up on a windows machine. Comcast is one of leaders for using IPv6 in the US if you use them at home. There are simple test-your-connection sites like this too http://test-ipv6.com
I hope you don’t mind but I plugged your DNS name. After which the site advertises Cloudflare can set you up with IPv6.
Results
IPv6 validation for rankexploits.com
Checking for AAAA DNS record : no AAAA record
Checking for IPv6 web server :
This website does not support IPv6 (yet).
IPv6 might be somewhat unnecessary right now since getting rid of IPv4 may take an extremely long time but adding it is getting to be more common and without too many problems that I have heard off.
MrE
It seems totally unnecessary for my site at present. Or am I missing something? What problems could not being IPv6 ready cause me at present?
MrE:
I suspect that is even an understatement.
lucia:
Not a one. The only reason you would need to be “IPv6 ready” is if IPv4 stopped being universally supported, or if you wanted a feature only available with IPv6. The former likely won’t happen for ages, and the latter would only happen if you chose for it to (I have no idea what feature you might want from it).
Brandon–
Thanks for the answer.
It seems to me that right now if I made things IPv6 ready, potentially, IPv6 addresses could be presented to my server. And right now I have access to extensive lists of IPv4 addresses that are “spammy” and “blockworthy”. I have no similar list for IPv6. So… if anything… I would open an avenue for spammers using IPv6 while providing no benefit to my own web site.
(Mind you, I understand the long term need for the switchto IPv6. I just don’t see much advantage to making my small hobby blog be in the front lines of IPv6 readiness. It seems wiser to me to switch when more ISPs offer IPv6 and more spam blocking tools have identified which IPv6 address blocks have been taken over by spammers/hackers etc.
lucia, that sounds about right. There’s pretty much no reason for most content providers to support IPv6. At some point it will likely become necessary, but as far as I can see, that’s nowhere in the foreseeable future.
By the way, happy Thanksgiving!
“nowhere in the foreseeable future” I can’t agree with that statement. Check this hockey stick out:
http://www.google.com/intl/en/ipv6/statistics.html
You could always turn it off it causes problems. It could be that since IPv6 has much smaller use and better security and privacy features that its less of a problem. Hobby blogs with smart operators like Lucia will probably the amongst the first to try it. I just think it’s likely inevitable and you could be ahead of the curve by trying it. On the Cloudflare site they say you can turn IPv6 with a few clicks.
The only way I can think of to refresh+comment within a few seconds would be if I was posting and the connection died or something and I used Lazarus (firefox add on) and submitted after it loaded… could happen, but honestly not enough people use it, though it’s fantastic.
As for IPv6, as the comment above says, it’s going to happen eventually, if you are able to move over to it now, there’s just no reason to not do it, ultimately.
Love the hidden comment bot traps btw, couldn’t see them in 3d mode, but I didn’t dig around too much.
MrE
Could you elaborate on “better privacy features”. Because users ‘privacy’ is one of the things hackers use to hack. If IPv6 gives users more anonymity, that’s a very good reason for a web site operator to defer switching until necessary.
Specifically: Hackers/ spammers and crackers want “privacy” a lot. They want it to be impossible to distinguish their connections from those by normal users. So giving them better privacy means I would need to figure out new other ways to protect my site from hacking attempts.
I don’t want to be a trogolodyte. But I see my possibly being unable to protect my site from being hacked as a good reason to wait before using IPv6.
It’s possible that there are devices involved in your hosting which have IPv6 active without it being obvious, as it is on by default in various devices and OSes.
The address protocol itself isn’t really what is attacked, though with IPv4 you could scan around for open ports, that isn’t really possible when there are quadrillions of possibilities.
On the other hand, IPv6 addresses can be device unique, which could be viewed as either a privacy issue or feature, depending on how you look at it.
In the situation where you might have IPv6 vulnerabilities but don’t know about them, due to your hosting possibly not being secure (or even aware of IPv6 being active) you definitely won’t be helping yourself by remaining “in the dark” so to speak.
If you’re concerned but not comfortable, I would set it up, learn a bit about it, then block it, so you know when you get hits, can get an idea of the traffic using it, but don’t feel like you’re hanging your rear out a window.
I agree that being cautious is the right way to go. I only bring it up since you seem to go more hands-on and under-the-hood with your blog. In response to some comments the main privacy improvements in IPv6 was designed more in the middle than on the ends. When a packet goes through the network it can pass through several companies, locations and countries and that is what IPv6 was supposed to help secure. They have added some of the same security options to IPv4 so its less of issue, or at least watered down. Every single packet has source and destination IP address. One touted benefit for the extra information(128 bits IPv6 vs older 32 bit IPv4) was to help make the ends more identifiable but optionally so because more bits means more info. The real reason to go to IPv6 is simply for more address space. The reason is very similar to extra numbers for the area code on the telephone. For someone who trials it would be a chance to cautiously try it out and that 1% using native IPv6 now might also have a better connection while you did.
Max’s tnemmoC(?) above about a trial is worth considering. I wouldn’t too spend much time on it or do turn it on and forget about it but it should be on a “to do” list for someone operating a long term website.
MrE:
I’m not sure how you think that calls what I said into question. First, a steep rise from less than one percent to one percent doesn’t indicate much about the future. I’d wager a large portion of that influx is due to smart phones with IPv6 enabled. A steep rise would be expected when those are initially introduced, but it wouldn’t be expected to continue for that long. That’s especially true since that rise would come with a rise in the number of devices, a deflatory effect.
Second, the discussion is not over the popularity of IPv6. It’s over the support of IPv4. Even if every device in the world supported IPv6, one would have no inherent need to switch to IPv6. As long as IPv4 was equally supported, and there was no benefit to switching to IPv6, lucia would have no reason to switch.
Until content providers find features of IPv6 they want to take advantage of, or until switching to IPv6 is beneficial to their users, they have no particular reason to support IPv6.
Brandon,
I don’t know what you are arguing exactly. A) The feature to take advantage of is serving a growing portion of the internet in their native protocol. B) Using both protocols is not a switch. C) It doesn’t matter if they are Smart phones or not.
MrE, I said:
You disagreed, pointing out the fact there has been a rise in devices using IPv6. My response was to show your point failed to rebut what I said. In other words, your disagreement was based upon an irrelevant point.
You offer no reason to believe this is “feature to take advantage of.” If devices are equally capable of using IPv6 and IPv4, but default to IPv6, there is no inherent advantage in serving them via one protocol rather than the other.
This is both wrong and a matter of minor semantics. It is not a “switch to IPv6,” but it is a “switch to supporting IPv6.” I don’t think the difference in wording is important, but I can change it if doing so will make things more clear.
I gave two clear reasons as to why it would matter. You ignored both then claimed it wouldn’t matter. That’s a little annoying, but I’ll reiterate the point. You argued a growth curve shows content providers will need to support IPv6 in the foreseeable future. I argued that growth curve wouldn’t continue so your argument is wrong.
Hmmm, you know that IPv4 addresses cap at a bit over 4 billion, right?
There was no reason to think IPv6 would take off before there was sort of a requirement for it to happen, but if you want to continue assigning addresses and adding devices to the internet, you sorta do have to get more addresses… which means you need to transition from IPv4. I would be defaulting to IPv6 right now if not for my router being outdated, hopefully I’ll be able to swap it for an 802.11n model that handles IPv6 natively soon.
Max, IPv4 has enough addresses to handle internet needs for decades to come. The problem is those addresses were not assigned in accordance with a long-term plan. Because of the non-structured and inefficient handling of IP address assignment, we’ll “run out” of addresses far sooner. Even so, nobody really knows for how long IPv4 would have been enough.
Max–For some reason, in some place the text of your comment read backwards. If you are doing something on purpose that causes that, please stop. Stripping out the subscript “TM” seems to have fixed that– and I’ll strip it in future if that’s what seems to cause it.
On the “switch to supporting IPv6 at my blog” it seems to me I can defer the decision for at least a year with no disadvantage a) to my blog or b) to my blog readers.
I understand that IPv6 is required at some point. But I see no advantage to switching right now. For the time being at least, 1% (and growing) of users who use IPv6 can connect if I offer only IPv4 connection. I don’t do ecommerce, so I can’t see how the improved “privacy” for their messages in transit matters with respect to my blog. (I’m not entirely sure they lose it if I use IPv4.) So I really don’t see the hurry.
As for the reasons why IPv4 is running out of addresses or when: I am neutral. The reasons “why” are not important to my decision whether or not to click the button at Cloudflare to see what happens either with people’s ability to access nor spammer/hackers ability to evade my spam/hack protection.
I’m really just not seeing the “upside” for me to spend even 30 seconds switching now rather than waiting a year.
Now: if someone who thinks I should switch over can tell me what advantage it would have for me operating this blog (rather than explaining the general advantages for the web at large), I might consider switching. To be specific about what advantage will not make me invest time in the experiment of offering IPv6 connection: the fact that I might gain advantages if my ISP permitted me IPv6 is not the same thing. I can see the advantage to my using an ISP that has IPv6– but that’s not the same as making my blog permit people whose ISPs simultaneously let them access using IPv6 or IPv4 use IPv6. Right now, they can use either, and my offering IPv4 seems fine for my blog.
Maybe there is something I am missing. But I really haven’t heard an advantage for the blackboard.
Early adoption is not for everyone and has its risks. I realize I am probably boring/annoying you with this and I accept you don’t want to do it but at this stage it’s not considered a “switch over”. It’s turning on something extra and testing the waters for you at this point. The goal of switch over is much further down the road and dependent on the dual stage. The advantage for the operation of your blog it is that you will be better prepared and be able to see if there is more or less spam or malicious hits than with IPv4, which would still be running.
You aren’t annoying or boring me. I know it’s just “turning on something extra and testing the waters”.
The difficulty I don’t see how doing this would make me better prepared for anything. I don’t even see how my watching to see whether the IPv6 connections have a disproportionate amount of spamming or hacking helps me. But I do see how “testing the waters” could create a big time sink for me– which is a disadvantage. So, as far as I can see, those people who have the goal of getting users to switch from IPv4 to IPv6 are going to have to spearhead the switch because I’m going to adopt after other people have tested the waters and started figuring out which IPv6 addresses were immediately taken over by spammers and hackers.