subfolder - Disallow subdirectory in robots.txt with language code -
we have website translated 15 languages, more comming in regulary our editors.
our url built follows:
sitename.company.com/{lcid}/folder1/folder2/
our company want's hide subfolder/page search engines.
lcid can en-us zh-hans. , looking solution don't have update robots file every time new language added.
can make wildcard entry or filter out single foldername regardless of it's path?
ie: companyinfo.nokia.com/en-us/aboutus/darknews/ want google crawl darknews , down (so /darknews/elopsellsout or /darknews/3210revival)
is possible?
you can wildcards:
user-agent: * disallow: /*/darknews/
just aware not robots support wildcards. work googlebot , bingbot, not work many others.
Comments
Post a Comment