Page Comparison

...

Verify blocking rules

To check whether the a bot was blocked successfully, is blocked or is still allowed you can use curl to access your website with a custom user agent. See how to do this in the following exampleexamples:

Blocked user agent:

curl -I https://www.domain.tld -A "SemrushBot -BA" 

HTTP/2 403 
date: Tue, 01 Sep 2020 15:03:24 GMT
content-type: text/html
content-length: 162
server: nginx

...