Hey there!

Scraping websites is great fun, and can be extremely useful.

However, many websites are made only for humans (and not programs or robots) to use. Frequently it can be expensive for these websites if they have a lot of robots using them, because they can't make money (such as from advertising).

Therefore, often websites will ask you that you don't scrape them. They do this by showing you the pages they don't want you to scrape in a file called robots.txt.

If you navigate to any popular website, such as https://google.com/robots.txt, you'll see that there is a list of pages that you can and can't scrape.

Note that robots.txt is not a protection mechanism. It's more of a polite notice. If you ignore it, sites may still have other protection mechanisms in place to prevent your scraper from accessing the website, such as captchas.

For more information on robots.txt: http://www.robotstxt.org/

Happy scraping!

—Jose and the Teclado team