Sorry for the OT, but for reasons unknown, it turns out that myself and various other Norwegians independently of each other suddenly don’t get access to Lucia’s Blackboard. We get the message<\/p>\n
“Forbidden
\nYou don’t have permission to access \/musings\/ on this server.<\/p>\n
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.”<\/p>\n
So what’s up, did you block us Lucia… please don’t, we miss you…<\/p>\n
Bebben (and others), Bergen – Norway<\/p><\/blockquote>\n
A few other innocent people have been blocked. I thought I would explain why. Of course, it has to do with ‘bots. As I’ve mentioned before ‘bots rove around hitting my server. To people who don’t self host, the main problem with ‘bots is a ‘bot might:<\/p>\n
- \n
- Read your<\/i> email on a web page and send you<\/i> spam.<\/li>\n
- Break into my blog, insert malicious links and inject malware into your pc or<\/li>\n
- Leave spammy comments for “male enhancement products”<\/li>\n<\/ol>\n
But since I pay for hosting, I have another difficulty. While attempting to do the above, some stupidly programmed bots loads tons of pages at once and cause my server load to exceed the level of resources I pay for. As it happens, I pay enough to get Dreamhost to guarantee me 400 MB of memory. If the memory draw rises above that, my blog can go down. Some comments can get lost. Entire posts can get lost. Visitors get grumpy because pages don’t load, and so on.<\/p>\n
Notice in the graph below, the memory draw (red line) suddenly shot up around quarter to 7 (server time) today:<\/p>\n
![\"\"](\"http:\/\/rankexploits.com\/musings\/wp-content\/uploads\/2011\/11\/ResourcesUsed-500x265.jpg\" "\\"ResourcesUsed\\"")
<\/a><\/p>\nSome of you might wonder: What could have happened at quarter to seven? Turn’s out, that’s when a stupidly programmed ‘bot hit the blog:
\n![\"\"](\"http:\/\/rankexploits.com\/musings\/wp-content\/uploads\/2011\/11\/ResourceConsumingBot1-500x154.jpg\" "\\"ResourceConsumingBot\\"")
<\/a>
\nThat image represents a subset of the hits by this ‘bot.<\/p>\nI don’t know what this bot meant to do. But I do know it was loading so many pages it caused Dreamhost to send me this email:<\/p>\n
Hi lucia!<\/p>\n
Nobody likes to hear that they’ve packed on a few pounds, but we wanted you to hear it from a friend.<\/p>\n
Your DreamHost VPS, ps33830, has just exceeded the memory allocation that you’ve established. If left unchecked that behavior could begin to negatively impact the VPS services of every other customer on your server.<\/p>\n
As a result we’ve had to reboot your VPS, effectively restarting your virtual machine. You may experience several minutes of downtime while your services come back online.<\/p>\n
Based on the usage pattern that we’ve seen today, it’s clear that you’d be best served by either working to reduce your memory footprint or simply increasing the amount of memory available to your VPS.<\/em><\/p>\n
We’ve put together some documentation that can help you troubleshoot and even reduce your current memory usage:
\nhttp:\/\/wiki.dreamhost.com\/DreamHost_PS_Troubleshooting
\nhttp:\/\/wiki.dreamhost.com\/PS_Optimization<\/p>\nRemember, you can visit https:\/\/panel.dreamhost.com\/index.cgi?tree=vserver.usage to scale your memory allocation up (or down) whenever you’d like!<\/p>\n
Strictly speaking, you don’t HAVE to take any action right now, but if your memory usage continues to routinely exceed the allocation level you’ve specified, you may experience more brief periods of downtime as your services are restarted. You may also get sick of seeing this message in your inbox. Unfortunately there’s not much that we can do about either, so we hope you’ll consider upgrading!<\/p>\n
Thank you for choosing DreamHost!\n<\/p><\/blockquote>\n
Now, my difficulty is this: The resources are being consumed by a ‘bot. It’s quite<\/I> evident that if I increased resources to 500 MB, this bot would just load more and more pages making the spike higher. I could increase to 1000 MB, and the bot would just keep loading till it hit 1000 MB and the blog went down. I’m pretty sure there is practically no level of resources I could pay for that would make this bot stop. It only stops when the blog goes down. <\/p>\n
I think I have to figure out how to block the ‘bot and it’s ‘bot friends. Today, I blocked ‘f1.bb.sky.com’ by adding “deny from f1.bb.sky.com” to my .htaccess file. I’m also trying to block it by adding “Java” to my rewrites: <\/p>\n
RewriteEngine on
\nRewriteCond %{REQUEST_FILENAME} (.+)timthumb\\.php$ [nc,or]
\nRewriteCond %{HTTP_USER_AGENT} ^Jakarta\\ Commons-HttpClient [nc,or]
\nRewriteCond %{HTTP_USER_AGENT} ^Content\\ Crawler [nc,or]
\nRewriteCond %{HTTP_USER_AGENT} ^Java [nc,or]
\nRewriteCond %{HTTP_USER_AGENT} ^Mail.RU
\nRewriteRule .* http:\/\/%{REMOTE_ADDR} [r=301,l]<\/p>\nBut I’m not competent at .htaccess, and I’m not sure this works. Meanwhile, blocking “f1.bb.sky.com” at least solves todays problem.<\/p>\n
So, what does this have to do with Norway?<\/b>
\nUnfortunately, “f1.bb.sky.com” is not the first bot taking the blog down. I’ve been seeing huge numbers of ‘bots over the past month. I’ve been trying to block them. Naturally, to avoid getting blocked, the spammers who program the badly behaved bots take out accounts with ISP’s who also provide service to entirely innocent people. Evidently, in the process of blocking something<\/i> I blocked Bergen, Norway. (I still don’t know which command was blocking them. I solved the problem by removing a whole bunch of blocks last night. Was I worried the blog would overload while I slept? Yes. Did it happen? Yes.)<\/p>\nTo get to the point those: I apologize Bergen, Norway. <\/p>\n
Meanwhile to others: It’s possible I will accidentally block someone innocent in future. If this happens to you: Sorry. But the alternative is the blog going down 3 times a day. (Seriously.) If you get blocked, you will be sent to this page:<\/p>\n
http:\/\/rankexploits.com\/forbidden.html<\/a>
\nLet me know if you can’t<\/I> figure out my email based on that page. If not, we can work together on how to make my email more obvious to a human while not revealing it to a ‘bot.<\/p>\nOddly enough, you could be sent to the exact same page if the blog is overloaded. If you are sent to that page and you are not a ‘bot, you should be able to figure out my email by reading that page. That means you will be able to send me an email asking me to unblock you. However, when you email me, please help me by following the instructions to learn your IP and useragent. Then send me an email containing your IP and useragent string. Knowing these might help me figure out how to unblock you without<\/i> also letting in things like “f1.bb.sky.com” who will take down the blog. Also, if I can’t figure out the specific commands blocking you, I may also email you and ask you to help me unblock you while not letting “f1.bb.sky.com” (or his friends) swamp my blog. <\/p>\n
Oh. If anyone is an .htaccess whiz, let me know. There are somethings only<\/i> ‘bots request, and I’d like to use those requests to better identify the bots. Or if you know any tricks– tell me. I’d really like to get rid of these periodic spikes that take the blog down. <\/p>\n","protected":false},"excerpt":{"rendered":"
Yesterday, Bebben from Norway left this note at Jeff’s blog: Sorry for the OT, but for reasons unknown, it turns out that myself and various other Norwegians independently of each other suddenly don’t get access to Lucia’s Blackboard. We get the message “Forbidden You don’t have permission to access \/musings\/ on this server. Additionally, a … Continue reading Sorry Bergen Norway!<\/span>
![\"\"](\"http:\/\/rankexploits.com\/musings\/wp-content\/uploads\/2011\/11\/ResourcesUsed.jpg\" "\\"ResourcesUsed\\"")
<\/a><\/p>\n