Parse /robots.txt file
