Looking for some good online training?

Here I would keep on updating some of the good online & free training materials.

• http://www.codecademy.com/
• http://www.codeschool.com/ - Chrome Developer Tools - course is good. 

What to do with Bots??? Web crawlers???

Getting considerable visits to you page? Well soon you will have more visits from Bots that from the real users. It is a phenomenon we have to live with. But then we can make a few changes to how we deal with them based on their characteristics.

1. Good Bots : Some companies have a different way of handling good bots. You need to take good care of them as they always lead you to more and more traffic which may lead to higher conversion. One way I have found in use is to redirect them to a different origin. Why we need to do this is because, it then becomes easier for NOC to distinguish between real issues and noise from the bots. You can always use robots.txt to define rules to see if you can get some of the major bots to behave well. You can even specify a time for crawl. What I have seen is that googlebot causes more then 75% of the requests from bots.

2. Bad Bots: Consider blocking them at Akamai WAF level or at the web server level based on the User-Agent value. You may even have to use you secondary fiirewall to add in additional rules.


Providing below some know bots.

• *googlebot* 
• *msnbot* 
• *Yahoo!* 
• *jobot*
• *dotbot* 
• *Pipes* 
• *yacybot* 
• *Python* 
• *skJeeves* 
• *AdsBot-Google* 
• *YandexBot* 
• *bingbot* 
• *Ultraseek* 
• *rogerbot* 
• *youdaobot* 
• *Mechanize* 
• *Jakarta* 
• *decobot* 
• *curl* 
• *psbot* 
• *Slurp* 
• *discobot* 
• *MaxPointCrawler* 
• *sogou* 
• *ezooms* 
• *RedCarpet* 
• *bingshoppingbot* 
• *FatBot* 
• *spotbot* 

Also some times, bad bots will not have an easily identifiable User-Agent and so it may become necessary to block them by IP. For anonymous network routes likes Tors, it will be a different strategy. Will cover it some time in future. 

CloudTest Videos

You can find some good videos on SOASTA CloudTest here. Some of them could have been made better.

http://cloudlink.soasta.com/t5/CloudTest-Videos/bg-p/CloudTest_Videos