--parserobots doesn't seem to work #93
Description
If I use the --parserobots argument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives.
You can try it on https://google.com to see the behaviour.