User-agent: * Crawl-Delay: 10 Disallow: /articles/* Disallow: /lp/ Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir* User-agent: Slurp Disallow: /articles/* Disallow: /lp/ Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir* User-agent: bingbot Disallow: /articles/* Disallow: /lp/ Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir* User-agent: dotbot Disallow: /articles/* Disallow: /lp/ Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir* User-agent: AhrefsBot Disallow: /articles/* Disallow: /lp/ Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir* User-agent: SemrushBot Disallow: /articles/* Disallow: /lp/ Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir* User-agent: Googlebot Disallow: /articles/* Disallow: /lp/ Disallow: /*/page/1 Disallow: /redirect.php Disallow: /redirect* Disallow: /redir.php Disallow: /redir*