HTML Author's Guide to the Robots META tagThe Web Robots Pages
HTML Author's Guide
to the Robots META tag.
The Robots META tag is a simple mechanism to indicate to visiting Web Robots if
a page should be indexed, or links on the page should be followed.
It differs from the Protocol for Robots Exclusion in that you need no effort or
permission from your Web Server Administrator.
Note: Currently only few robots support this tag!
Where to put the Robots META tag
Like any META tag it should be placed in the HEAD section of an HTML page:
...
...
What to put into the Robots META tag
The content of the Robots META tag contains directives separated by commas. The
currently defined directives are [NO]INDEX and [NO]FOLLOW. The INDEX directive
specifies if an indexing robot should index the page. The FOLLOW directive
specifies if a robot is to follow links on the page. The defaults are INDEX and
FOLLOW. The values ALL and NONE set all directives on or off: ALL=INDEX,FOLLOW
and NONE=NOINDEX,NOFOLLOW.
Some examples:
Note the "robots" name of the tag and the content are case insensitive.
You obviously should not specify conflicting or repeating directives such as:
A formal syntax for the Robots META tag content is:
content = all | none | directives
all = "ALL"
none = "NONE"
directives = directive ["," directives]
directive = index | follow
index = "INDEX" | "NOINDEX"
follow = "FOLLOW" | "NOFOLLOW"
The Web Robots Pages