Robots.txt Blocking Subdomain From Root

[zompus]

I know it's possible to block subdomains using a separate robots.txt file located inside the actual subdomain folder, however because of the subdomain is a rewrite, I cannot, nor do i want to create a new directory just to place 1 robots.txt file inside. I also know placing a meta noindex within the download pages will accomplish this.

However, here's my question:

Disallow: http://download.domain.com/

Will this disallow all URLs under the download subdomain or just "download.domain.com"?

Thanks in advance, big hugs. :xmas-smiley-010:

 

[kblessinggr]

Well no because they don't understand that kind of command. It's based on the server that the robot.txt is found and all paths under it are relative to the robot.txt

For example

User-agent: *
Disallow: /

placed inside the root of your subdomain would prevent any user agent from indexing anything from the root onward.

But if you place a robot.txt into www.domain.com it only applies to "www.domain.com"

Keep in mind that disallowing won't prevent spiders from getting in anyways, its more of a suggestion of telling search engines and bots "Please don't view this content". If you have a directory listing enabled, I'd recommend putting a blank index.html in the root of the subdomain if you do not intend on having a page there at all.

Also when the bots come to your normal www.domain.com and see a link to download.domain.com it'll look for a robots.txt there to define the parameters, not in www.domain.com, so my suggestion is, if its important to you, either put a robots.txt inside of download.domain.com (which I don't know why you don't want to). Or make it into a subdirectory redirect.