unrobotstxt ~master (2020-07-03T06:00:54Z)
Dub
Repo
RobotsMatcher.disallow
unrobotstxt
RobotsMatcher
Returns true if we are disallowed from crawling a matching URI.
class
RobotsMatcher
const
bool
disallow
(
)
Meta
Source
See Implementation
unrobotstxt
RobotsMatcher
constructors
this
functions
AllowedByRobots
HandleAllow
HandleDisallow
HandleRobotsEnd
HandleRobotsStart
HandleSitemap
HandleUnknownAction
HandleUserAgent
InitUserAgentsAndPath
OneAgentAllowedByRobots
disallow
disallow_ignore_global
ever_seen_specific_agent
matching_line
seen_any_agent
static functions
IsValidUserAgentToObey
structs
Match
MatchHierarchy
variables
allow_
disallow_
ever_seen_specific_agent_
match_strategy_
path_
seen_global_agent_
seen_separator_
seen_specific_agent_
user_agents_
Returns true if we are disallowed from crawling a matching URI.