Parses body of a robots.txt and emits parse callbacks. This will accept typical typos found in robots.txt, such as 'disalow'.
Note, this function will accept all kind of input but will skip everything that does not look like a robots directive.
See Implementation
Parses body of a robots.txt and emits parse callbacks. This will accept typical typos found in robots.txt, such as 'disalow'.
Note, this function will accept all kind of input but will skip everything that does not look like a robots directive.