Legal issues

Before you start scraping a particular website make sure you're not breaking any laws.

  • If the website has an API for getting needed information use it!
  • Read the website's Terms of Use.
  • Do not harvest email addresses, personal phone numbers etc.
  • Respect robots.txt
  • Use a readable User-Agent string with your contacts (e.g. a website address).
  • Make sure you do not create a significant load on the website. Make pauses between the requests.
  • Read other articles and/or books about web scraping legal issues