A file at the root of a website that tells crawlers which areas they may visit. It can also allow or block individual AI crawlers such as OAI-SearchBot or PerplexityBot.
Glossary
robots.txt
Work with us
Ready for your next stage of growth?
Tell us where you want to take your brand. Our team will come back within one working day with next steps.