No Index No Follow Htaccess
The cool thing about this is that you can do it for an entire site. If your site is running on Apache, and modheaders is enabled (it usually is), you could add the following single line to your.htaccess file: Header set X-Robots-Tag 'noindex, nofollow' And this would have the effect that that entire site can be indexed. But would never be.
About the Robots <META> tag
In a nutshell
You can use a special HTML <META> tag to tell robots not to indexthe content of a page, and/or not scan it for links to follow.
For example:
There are two important considerations when using the robots <META> tag:
- robots can ignore your <META> tag. Especially malware robots that scan theweb for security vulnerabilities, and email address harvesters used by spammerswill pay no attention.
- the NOFOLLOW directive only applies to links on this page. It'sentirely likely that a robot might find the same links on some otherpage without a NOFOLLOW (perhaps on some other site), and so stillarrives at your undesired page.
No Index No Follow Htaccess File
Don't confuse this NOFOLLOW with the rel='nofollow' link attribute.
The details
Like the /robots.txt, the robots METAtag is a de-facto standard. It originated from a 'birds of a feather' meeting at a 1996distributed indexing workshop, and was described in meeting notes.
The META tag is also described in the HTML4.01 specification, Appendix B.4.1.
The rest of this page gives an overview of how to use the robots<META> tags in your pages, with some simple recipes.To learn more see also the FAQ.
How to write a Robots Meta Tag
Where to put it
Like any <META> tag it should be placed in the HEAD section of an HTMLpage, as in the example above. You should put it in every page on yoursite, because a robot can encounter a deep link to anypage on your site.
What to put into it
No Index No Follow Htaccess Number
The 'NAME' attribute must be 'ROBOTS'.
Valid values for the 'CONTENT' attribute are:'INDEX', 'NOINDEX', 'FOLLOW','NOFOLLOW'. Multiple comma-separated values are allowed, butobviously only some combinations make sense. If there is norobots <META> tag, the default is 'INDEX,FOLLOW',so there's no need to spell that out. That leaves: