robots.txt AI crawler rules
ai-001a ·
AI & Automation · Level 1 ·
Automated (browser)
robots.txt contains rules for known AI crawlers
Results across 110 scanned sites
| Outcome | Count |
|---|---|
| Passed | 104 |
| Failed | 6 |
| Not yet scanned for this check | 6 |
Failing (6)
Passing (104)
- ASOS · retail
- Amnesty International UK · not_for_profit
- Arcadia Fund · not_for_profit
- Argos · retail
- BBC · media
- Barnardo's · not_for_profit
- Bloomsbury Publishing · publishers
- Bodleian Library · libraries
- Boots · retail
- British Heart Foundation · not_for_profit
- British Library · libraries
- British Museum · archives
- British Red Cross · not_for_profit
- Cambridge University Library · libraries
- Cambridge University Press · publishers
- Cancer Research UK · not_for_profit
- Care Quality Commission · government
- Channel 4 · media
- Citizens Advice · not_for_profit
- Cory Doctorow · media
- Currys · retail
- Durham University · higher_education
- Evening Standard · media
- Faber & Faber · publishers
- Financial Conduct Authority · government
- Financial Times · media
- GOV.UK · government
- Hachette UK · publishers
- HarperCollins UK · publishers
- Imperial College London · higher_education
- Imperial War Museums · archives
- Indigo Trust · not_for_profit
- Information Commissioner's Office · government
- King's College London · higher_education
- Leeds Libraries · libraries
- Library of Birmingham · libraries
- London School of Economics · higher_education
- Macmillan Cancer Support · not_for_profit
- Marks & Spencer · retail
- Mind · not_for_profit
- NHS · government
- National Gallery · archives
- National Library of Scotland · libraries
- National Library of Wales · libraries
- National Museums Scotland · archives
- National Records of Scotland · archives
- National Trust · not_for_profit
- Natural History Museum · archives
- New Statesman · media
- Next · retail
- Northern Ireland Direct · government
- Ocado · retail
- Ofcom · government
- Office for National Statistics · government
- Oxfam GB · not_for_profit
- Oxford University Press · publishers
- Pan Macmillan · publishers
- Pearson UK · publishers
- Penguin Random House UK · publishers
- RSPB · not_for_profit
- RSPCA · not_for_profit
- Royal Museums Greenwich · archives
- Sainsbury's · retail
- Save the Children UK · not_for_profit
- Scholastic UK · publishers
- Science Museum · archives
- Scope · not_for_profit
- Scottish Government · government
- Sense About Science · not_for_profit
- Shelter · not_for_profit
- Sky News · media
- Tate · archives
- Taylor & Francis · publishers
- Tesco · retail
- The Economist · media
- The Guardian · media
- The Independent · media
- The London Library · libraries
- The National Archives · archives
- The Telegraph · media
- The Times · media
- Thinking Way Outside The Box · not_for_profit
- UK Legislation · government
- UK Parliament · government
- University College London · higher_education
- University of Birmingham · higher_education
- University of Bristol · higher_education
- University of Cambridge · higher_education
- University of Edinburgh · higher_education
- University of Glasgow · higher_education
- University of Leeds · higher_education
- University of Manchester · higher_education
- University of Nottingham · higher_education
- University of Oxford · higher_education
- University of St Andrews · higher_education
- University of Warwick · higher_education
- Verso Books · publishers
- Victoria and Albert Museum · archives
- Waterstones · retail
- Wellcome Collection · libraries
- Welsh Government · government
- White Label · not_for_profit
- eff.org
- eve.gd