很多centos系统安装宝塔搭建网站的用户，负载经常100%，cpu经常100%原因是被很多蜘蛛爬网站造成的，那么就需要屏蔽一下这些乱七八糟的蜘蛛，或主要是GET|HEAD|POST类型的请求方式，请问这种攻击怎么防范呢？具体方式如下：

一，Nginx屏蔽蜘蛛

1.修改Nginx配置参数

需要在该目录下添加文件 /www/server/nginx/conf/agent_deny.conf

#禁止Scrapy等工具的抓取
if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {
return 403;
}
#禁止指定UA访问。UA为空的可以访问，比如火车头可以正常发布。
if  ($http_user_agent ~ "FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|YandexBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|Bytespider|Ezooms|JikeSpider|SemrushBot" ) {
return 403;             
}
#禁止非GET|HEAD|POST方式的抓取
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 403;
}
if ($http_rang ~ "\d{9,}") {
return 444;
}

 

文件添加以上内容并保存

2.修改网站配置文件

网站 ---> 设置 -----> 配置文件

root /www/wwwroot/ggplus.cn;

include agent_deny.conf;

#SSL-START SSL相关配置，请勿删除或修改下一行带注释的404规则

#error_page 404/404.html;

3.重启nginx 生效

 

二，屏蔽注入某些后缀文件

站点设置-访问限制-禁止访问-文件后缀 rar|zip|gz|sql|cgi|ini|docx|txt|doc

 

三，禁止蜘蛛索引图片等文件

1,

User-agent: *       //比如只禁止谷歌，则将*换成 Googlebot

Disallow: /*.jpg$

Disallow: /*.jpeg$

Disallow: /*.gif$

Disallow: /*.png$

Disallow: /*.bmp$

Disallow: /*.rar$

Disallow: /*.zip$

Disallow: /*.rm$

Disallow: /*.rmvb$

Disallow: /*.wma$

Disallow: /*.wmv$

Disallow: /*..mp3$

Disallow: /*.mp4$

Disallow: /*.mpeg$

Disallow: /*.avi$

Disallow: /*.exe$

Disallow: /*.msi$

Disallow: /*.iso$

其他蜘蛛：

Baiduspider

YisouSpider

360Spider

Sosospider

sogou spider

YodaoBot

Googlebot

bingbot

2, 在User-Agent过滤里添加 规则, UA的响应为444 (注：加入此规则，会出现屏蔽Baiduboxapp的UA）(FeedDemon|CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms)