Web crawling and data protection: CNIL has issued a 180,000 EUR fine against a provider of automobile insurance policies for failure to adequately protect data in violation of GDPR, specifically citing disallowing web crawling as a way to protect personal data from wrongful access.

In particular the company :

1. sent usernames and passwords in cleartext
2. allowed users to access other users accounts
3. allowed users’ accounts to be accessible by the general public when entering a URL or changing the last numbers in a URL

The compromised information included copies of driver’s licenses, registration cards, bank identification records and documents to determine whether a person had been subject to a license withdrawal or hit-and-run.

Key takeaways:

• Don’t send passwords in cleartext.
• Adopt a strong password policy.
• Ensure access controls to information are limited and accurate.
• Use a “robot.txt” or other means to disallow SEO and crawling by search engines of internal web pages containing sensitive information.

[View source.]