使用Fail2ban防止Nginx被badbot爬取
1. 安装fail2ban
sudo apt install fail2ban2. 测试ban爬虫规则
正则表达式如下
<HOST> -.* "(GET|POST).*" (401|403|404) .* (Bytespider|AhrefsBot|Amazonbot|DataForSeoBot|MJ12bot|ds-robot|meta-externalagent|SemrushBot)从/var/log/nginx/access.log取一行日志用于测试
tail -n 1 /var/log/nginx/access.log输出如下
47.128.59.139 - - [06/Oct/2025:15:00:20 +0000] "GET /opensource/dashboard-icons/compare/renovate/react-monorepo...feat/pagination?style=unified&whitespace=ignore-change HTTP/2.0" 403 153 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"使用fail2ban-regex命令进行测试, 格式为 fail2ban-regex ‘log_line’ ‘regex’, 注意使用单引号’包裹
fail2ban-regex '47.128.59.139 - - [06/Oct/2025:15:00:20 +0000] "GET /opensource/dashboard-icons/compare/renovate/react-monorepo...feat/pagination?style=unified&whitespace=ignore-change HTTP/2.0" 403 153 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"' '<HOST> -.* "(GET|POST).*" (401|403|404) .* (Bytespider|AhrefsBot|Amazonbot|DataForSeoBot|MJ12bot|AhrefsBot|ds-robot)'输出如下
Running tests
=============
Use failregex line : <HOST> -.* "(GET|POST).*" (401|403|404) .* (Bytesp...
Use single line : 47.128.59.139 - - [06/Oct/2025:15:00:20 +0000] "GE...
Results
=======
Failregex: 1 total
|- #) [# of hits] regular expression
| 1) [1] <HOST> -.* "(GET|POST).*" (401|403|404) .* (Bytespider|AhrefsBot|Amazonbot|DataForSeoBot|MJ12bot|AhrefsBot|ds-robot)
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [1] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.03 sec]请求包含Bytespider， 1 matched, 符合预期
3. 编辑过滤器文件
vim /etc/fail2ban/filter.d/nginx-badbots.conf输入规则配置, 通过正则表达式匹配触发规则的请求
[Definition]
failregex = <HOST> -.* "(GET|POST).*" (401|403|404) .* (Bytespider|AhrefsBot|Amazonbot|DataForSeoBot|MJ12bot|AhrefsBot|ds-robot|meta-externalagent|SemrushBot)
ignoreregex =4. 编辑用户ban规则
我这里直接使用默认的/etc/fail2ban/jail.d/defaults-debian.conf
vim /etc/fail2ban/jail.d/defaults-debian.conf输入如下规则
[DEFAULT]
# 禁封时间，秒为单位 (600秒 = 10分钟)
bantime = 600000
# 尝试次数
maxretry = 5
# 日志文件格式 (自动检测)
logencoding = auto
# 忽略的IP列表 (例如你的本地IP地址)
ignoreip = 127.0.0.1/8 ::1 你的IP地址
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
[nginx-http-auth]
enabled = true
port = http,https
filter = nginx-http-auth
logpath = /var/log/nginx/error.log
maxretry = 5
[nginx-botsearch]
enabled = true
port = http,https
filter = nginx-botsearch
logpath = /var/log/nginx/access.log
maxretry = 6
bantime = 86400
# bot ban
[nginx-badbots]
enabled = true
port = http,https
filter = nginx-badbots
logpath = /var/log/nginx/access.log
maxretry = 1
bantime = 864005. 启动fail2ban
systemctl restart fail2ban6. 查看封禁的IP
fail2ban-client status nginx-badbots输出如下
Status for the jail: nginx-badbots
|- Filter
| |- Currently failed: 0
| |- Total failed: 500
| `- File list: /var/log/nginx/access.log
`- Actions
|- Currently banned: 514
|- Total banned: 514
`- Banned IP list: 100.28.204.82 107.20.255.194 184.73.195.18 3.219.81.66 34.204.150.196 34.236.135.14 44.195.145.102 47.128.24.0 47.128.28.26 47.128.36.8 47.128.52.74 50.16.72.185 54.225.81.20 54.84.102.81 54.210.152.179 47.128.117.189 52.204.89.12 47.128.29.198 47.128.113.233 44.205.74.196 47.128.112.51 47.128.116.2 18.209.137.234 47.128.43.95 3.222.85.38 47.128.97.244 47.128.56.142 47.128.97.208 34.231.118.144 54.225.199.17 3.214.176.44 47.128.28.100 47.128.29.69 98.83.72.38 47.128.33.63 47.128.57.254 52.71.46.142 34.192.125.239 47.128.27.152 47.128.27.121 52.0.63.151 100.29.63.24 18.213.27.222 34.202.88.37 3.215.221.125 47.128.99.180 34.239.85.139 107.22.208.39 47.128.23.6 23.21.179.27 47.128.113.13 34.196.114.170 3.94.40.182 52.203.68.145 3.93.253.174 34.194.233.48 100.27.153.9 3.235.215.92 47.128.32.17 54.84.250.51 3.213.106.226 47.128.35.93 44.218.6.93 44.215.210.112 3.225.45.252 3.211.181.86 18.214.251.19 47.128.21.91 34.231.77.232 54.84.93.8 34.233.219.155 47.128.33.53 3.227.180.70 44.207.252.58 54.91.122.193 34.192.67.98 47.128.34.73 3.209.174.110 44.210.204.255 54.235.158.162 44.194.139.149 47.128.60.166 52.1.106.130 98.80.130.239 3.223.134.5 47.128.38.229 184.72.95.195 44.215.61.66 3.94.156.104 3.208.146.193 52.6.232.201 3.93.98.99 47.128.115.18 54.84.147.79 52.6.97.88 52.207.47.227 100.29.34.97 35.171.117.160 3.94.199.128 52.45.92.83 47.128.127.191 184.73.47.24 52.200.142.199 3.230.224.6 44.218.170.184 47.128.29.96 54.164.106.236 3.229.95.193 18.215.112.101 23.23.214.190 54.167.32.123 47.128.127.241 34.234.206.30 52.54.249.218 44.209.35.147 44.223.116.180 54.197.114.76 34.225.24.180 23.20.178.124 47.128.19.66 34.230.124.21 44.216.172.204 34.195.60.66 18.214.124.6 44.207.69.106 47.128.41.25 35.174.141.243 34.199.252.22 52.71.218.25 18.232.12.157 54.197.102.71 44.205.120.22 47.128.19.53 54.90.8.255 52.44.148.203 47.128.123.236 98.83.178.66 54.147.238.89 50.19.102.70 54.157.99.244 47.128.19.28 52.73.6.26 54.166.126.132 52.204.81.148 98.83.177.42 18.205.91.101 52.2.4.213 47.128.28.187 34.236.185.101 44.217.255.167 34.224.132.215 34.238.45.183 23.23.99.55 47.128.28.180 44.193.102.198 54.83.56.1 52.204.71.8 3.217.171.106 47.128.31.180 54.235.125.129 52.5.242.243 54.88.84.219 47.128.63.41 18.204.89.56 54.152.163.42 3.216.86.144 100.24.167.60 18.209.201.119 44.221.37.41 47.128.51.220 34.205.163.103 35.174.253.85 44.221.105.234 52.22.87.224 47.128.26.39 44.213.36.21 184.72.84.154 98.82.59.253 54.221.203.24 47.128.24.243 44.217.177.142 3.208.156.9 34.195.248.30 44.215.231.15 47.128.61.153 34.225.87.80 52.204.253.129 47.128.120.8 3.231.193.38 35.170.205.140 52.2.83.227 47.128.99.41 54.84.161.62 23.21.119.232 107.20.25.33 47.128.121.42 18.210.58.238 23.21.228.180 47.128.54.213 52.202.233.37 34.235.239.240 54.147.182.90 47.128.47.40 52.23.112.144 54.235.172.108 35.153.86.200 47.128.28.131 100.29.155.89 44.210.213.220 34.225.243.131 35.173.38.202 34.227.234.246 47.128.54.32 52.70.209.13 47.128.54.17 52.205.222.214 47.128.22.83 23.22.105.143 23.21.175.228 107.20.181.148 3.218.35.239 47.128.30.89 44.205.192.249 52.3.155.146 54.162.69.192 44.223.232.55 47.128.42.11 34.196.6.199 3.221.50.71 3.226.34.98 47.128.96.43 52.0.41.164 100.28.118.16 18.207.89.138 54.85.109.140 35.173.18.61 47.128.48.239 23.21.179.120 18.205.127.11 47.128.116.74 44.194.134.53 52.71.216.196 98.82.38.120 47.128.25.208 35.169.102.85 52.204.174.139 44.209.187.99 34.206.249.188 47.128.25.182 34.197.28.78 54.197.178.107 18.215.24.66 3.81.253.213 54.87.95.7 47.128.55.27 54.85.7.119 23.23.180.225 44.213.202.136 44.214.19.8 44.215.235.20 34.193.2.57 47.128.110.124 34.206.212.24 47.128.32.63 52.1.157.90 47.128.98.0 54.225.98.148 47.128.30.225 52.71.203.206 47.128.119.1 34.231.181.240 54.156.248.117 54.80.73.122 52.203.237.170 47.128.40.17 52.70.138.176 18.214.186.220 54.86.59.155 47.128.116.60 18.215.49.176 47.128.16.230 52.21.62.139 47.128.17.223 23.21.227.240 52.200.54.136 47.128.63.191 3.210.223.61 44.221.227.90 54.166.104.83 52.204.37.237 47.128.121.238 44.206.93.215 47.128.28.201 54.204.62.163 3.222.190.107 52.205.141.124 47.128.61.201 44.197.76.210 3.213.213.161 3.94.157.25 54.159.18.27 47.128.25.138 100.24.149.244 47.128.50.85 52.44.229.124 47.128.28.178 47.128.34.192 44.205.180.155 44.206.65.8 54.89.90.224 47.128.113.50 3.221.244.28 3.229.2.217 47.128.113.137 3.219.80.71 44.207.207.36 52.4.238.8 3.217.82.254 47.128.42.239 3.213.46.222 44.223.116.149 47.128.116.37 47.128.19.16 47.128.16.220 47.128.23.76 3.212.205.90 3.232.102.111 47.128.124.227 52.54.157.23 47.128.57.206 52.2.58.41 47.128.47.236 18.235.158.19 47.128.33.244 23.22.59.87 44.195.50.71 47.128.23.11 34.194.165.45 47.128.120.108 98.82.214.73 47.128.62.14 35.171.141.42 54.163.136.244 47.128.113.90 54.80.185.200 47.128.62.114 47.128.115.25 47.128.47.14 47.128.60.159 47.128.41.152 47.128.25.195 47.128.41.45 47.128.43.89 47.128.22.58 47.128.26.25 47.128.127.214 47.128.115.161 47.128.32.150 47.128.97.229 47.128.55.74 47.128.33.24 47.128.110.244 47.128.54.236 47.128.47.33 47.128.44.41 47.128.40.97 47.128.127.148 47.128.121.21 47.128.44.149 47.128.51.227 47.128.121.211 47.128.44.142 47.128.61.214 47.128.111.74 47.128.57.212 47.128.36.101 47.128.58.224 52.45.15.233 34.205.170.13 47.128.124.235 85.208.96.200 47.128.124.64 23.21.204.95 47.128.40.195 98.82.39.241 3.224.215.150 23.23.104.107 47.128.57.22 54.204.12.115 47.128.36.7 34.194.95.99 34.234.200.207 47.128.28.161 47.128.51.198 47.128.113.88 3.232.39.98 54.225.148.123 18.235.81.246 47.128.44.164 98.84.184.80 3.216.227.216 44.196.118.6 44.193.115.232 52.7.33.248 52.0.218.219 23.21.250.48 3.211.105.134 54.83.180.239 34.239.197.197 100.28.57.133 3.210.29.96 3.93.211.16 18.213.102.186 54.156.124.2 23.23.103.31 98.83.226.125 34.194.226.74 23.21.225.190 100.29.128.75 23.23.212.212 34.233.114.237 52.200.251.20 18.213.70.100 54.144.185.255 98.82.107.102 34.194.14.255 54.156.55.147 18.233.24.238 100.29.107.38 184.73.68.20 18.211.39.188 44.220.2.97 3.230.69.161 35.168.238.50 54.83.23.103 52.3.156.186 47.128.50.104 52.45.29.57 47.128.38.137 52.5.232.250 54.145.82.217 47.128.17.214 54.197.82.195 47.128.38.1 47.128.56.209 47.128.116.75 52.45.194.165 47.128.42.142 47.128.97.220 47.128.59.106 52.2.191.202 47.128.51.222 47.128.98.249 47.128.56.43 47.128.62.222 44.208.193.63 47.128.61.60 47.128.118.177 54.235.191.179 47.128.124.206 34.203.111.15 47.128.47.17 47.128.52.90 52.4.229.9 52.0.105.244 52.4.76.156 35.172.125.172 47.128.122.84 47.128.37.67 47.128.115.182 47.128.43.177 47.128.28.185 107.20.224.184 47.128.52.96 47.128.99.63 47.128.18.177 47.128.33.140 47.128.18.250 47.128.61.29 54.92.171.106 47.128.126.110 47.128.25.217 47.128.19.231 47.128.27.133 47.128.119.230 47.128.112.110 47.128.59.135 47.128.61.202 47.128.59.148 52.73.142.41 47.128.59.139 47.128.48.73 47.128.16.205 47.128.37.16 47.128.22.245 47.128.124.254 47.128.124.229 47.128.58.220 34.206.193.60 47.128.47.23 100.29.164.178 47.128.19.60 47.128.44.92 47.128.35.101 47.128.119.62 47.128.18.139 52.3.102.51 47.128.127.31 47.128.115.124 47.128.125.159 47.128.115.207 47.128.124.183 47.128.33.188 185.191.171.14