-
-
Notifications
You must be signed in to change notification settings - Fork 534
Rate Limiting ‐ Caddy
The current design of Nitter, along with its methodology for accessing the Twitter service, necessitates heightened vigilance on the part of instance operators to manage unwarranted access by web scraping bots. This guide is intended to provide operators with essential information to effectively mitigate unauthorized web scraping bot traffic.
Before proceeding with the rate-limiting setup, ensure that you have:
- A functional Nitter installation.
- Caddy as your web server, with an installation of the custom module RussellLuo/caddy-ext/ratelimit. You can refer to the Caddy docs or its docker docs for instructions.
- In this guide, we will add rate-limiting rules to the server block as outlined in Caddy Configuration.
Caddyfile
nitter.example.com {
@notstatic {
not path /css/* /js/* /fonts/* /browserconfig.xml /android-chrome* /favicon* /logo* /lp.svg /robots.txt /safari* /site.webmanifest /pic/*
}
rate_limit @notstatic {remote.ip} 2r/s 60000 500
rate_limit @notstatic {remote.ip} 30r/m 300000 500
reverse_proxy localhost:8080 {
transport http {compression off}
}
header {
Strict-Transport-Security "max-age=63072000"
Referrer-Policy no-referrer
X-Permitted-Cross-Domain-Policies none
X-Content-Type-Options nosniff
Permissions-Policy "Permissions-Policy: accelerometer=(), ambient-light-sensor=(), autoplay=(self), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(self), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(self), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"
header Content-Security-Policy "default-src 'none'; script-src 'self' 'unsafe-inline'; img-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; object-src 'none'; media-src 'self' blob:; worker-src 'self' blob:; base-uri 'self'; form-action 'self'; frame-ancestors 'self'; connect-src 'self' https://*.twimg.com; manifest-src 'self'"
}
# Optional: discard logs completely
log {
output discard
}
}
The @notstatic named matcher ensure that normal usage, such as serving images, videos, and site data, does not trigger rate limiting.
The rate_limit directives limits users to 2 request per second and 30 requests per minute, a natural browsing rate for the site.
Over this, you need to setup fail2ban to persist the bans. Follow https://muetsch.io/how-to-integrate-caddy-with-fail2ban.html and the Fail2Ban instructions under https://github.com/zedeus/nitter/wiki/Rate-Limiting-%E2%80%90-Nginx#fail2ban to do so. (Do note you need logs enabled for fail2ban)
You may also use the module mholt/caddy-ratelimit to manage unwarranted access.