web-dev-qa-db-ja.com

.htaccess経由でGoogleクローラーを別のrobots.txtにリダイレクトする

終日グーグルで答えを探しましたが、まだ答えが見つかりませんでした。

www.static.example.comのミラーサイトである仮想サブドメインwww.example.comがあります。これは、サブドメインとドメインのルートフォルダーが1つしかないことを意味します。

クローラーを別のrobots.txtファイルにリダイレクトしたい-robots_static.txtコマンドによるインデックス作成を禁止するURLに.staticが表示されたら/disallow Googleの検索結果にコンテンツが重複しているため、これを実行したいと思います。サブドメインは、メインドメインとまったく同じコンテンツを表示しています。

クローラーがrobots_static.txtの代わりにrobots.txtを認識できるようにするにはどうすればよいでしょうか？

私がこれまでに見つけたのはこれです：

RewriteCond %{HTTP_Host} ^www.static.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots_static.txt [NC,L]

ただし、ウェブマスターツールをチェックインしても、robots.txtではなくrobots_static.txtがロボットファイルとして表示されるため、すべてをクロールしてインデックスを2回作成します。

私は何を間違えましたか？ありがとう

編集：これは私の.htaccessファイルです

##
# @package      Joomla
# @copyright    Copyright (C) 2005 - 2013 Open Source Matters. All rights reserved.
# @license      GNU General Public License version 2 or later; see LICENSE.txt
##

##
# READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE!
#
# The line just below this section: 'Options +FollowSymLinks' may cause problems
# with some server configurations.  It is required for use of mod_rewrite, but may already
# be set by your server administrator in a way that dissallows changing it in
# your .htaccess file.  If using it causes your server to error out, comment it out (add # to
# beginning of line), reload your site in your browser and test your sef url's.  If they work,
# it has been set by your server administrator and you do not need it set here.
##

## Can be commented out if causes errors, see notes above.
Options +FollowSymLinks

## Mod_rewrite in use.

RewriteEngine On

RewriteEngine On
RewriteCond %{HTTP_Host} !^www\.
RewriteRule ^(.*)$ http://www.%{HTTP_Host}/$1 [R=301,L]




RewriteCond %{HTTP_Host} ^www.static.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots_static.txt [NC,L]


## Begin - Rewrite rules to block out some common exploits.
# If you experience problems on your site block out the operations listed below
# This attempts to block the most common type of exploit `attempts` to Joomla!
#
# Block out any script trying to base64_encode data within the URL.
RewriteCond %{QUERY_STRING} base64_encode[^(]*\([^)]*\) [OR]
# Block out any script that includes a <script> tag in URL.
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
# Block out any script trying to set a PHP GLOBALS variable via URL.
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
# Block out any script trying to modify a _REQUEST variable via URL.
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
# Return 403 Forbidden header and show the content of the root homepage
RewriteRule .* index.php [F]
#
## End - Rewrite rules to block out some common exploits.

## Begin - Custom redirects
#
# If you need to redirect some pages, or set a canonical non-www to
# www redirect (or vice versa), place that code here. Ensure those
# redirects use the correct RewriteRule syntax and the [R=301,L] flags.
#
## End - Custom redirects

##
# Uncomment following line if your webserver's URL
# is not directly related to physical file paths.
# Update Your Joomla! Directory (just / for root).
##

# RewriteBase /

RewriteCond %{THE_REQUEST} ^GET.*index\.php [NC]
RewriteCond %{THE_REQUEST} !/system/.*
RewriteRule (.*?)index\.php/*(.*) /$1$2 [R=301,L]
RewriteCond %{THE_REQUEST} ^GET

## Begin - Joomla! core SEF Section.
#
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
#
# If the requested path and file is not /index.php and the request
# has not already been internally rewritten to the index.php script
RewriteCond %{REQUEST_URI} !^/index\.php
# and the request is for something within the component folder,
# or for the site root, or for an extensionless URL, or the
# requested URL ends with one of the listed extensions
RewriteCond %{REQUEST_URI} /component/|(/[^.]*|\.(php|html?|feed|pdf|vcf|raw))$ [NC]
# and the requested path and file doesn't directly match a physical file
RewriteCond %{REQUEST_FILENAME} !-f
# and the requested path and file doesn't directly match a physical folder
RewriteCond %{REQUEST_FILENAME} !-d
# internally rewrite the request to the index.php script
RewriteRule .* index.php [L]
#
## End - Joomla! core SEF Section.

<FilesMatch "\.(ico|pdf|flv|jpg|ttf|jpg|jpeg|png|gif|js|css|swf)$">
Header set Expires "Wed, 15 Apr 2020 20:00:00 GMT"
Header set Cache-Control "public"
</FilesMatch>

<ifModule mod_headers.c>
    Header set Connection keep-alive
</ifModule>

########## Begin - Remove Etags
    #
    FileETag none
    #
    ########## End - Remove Etags
htaccesssubdomainweb-crawlersrobots.txtduplicate-content
3
user3474818

Googleのボットは、/ robots_static.txtではなく、サブドメインから/robots.txtを要求しますが、これは意味がありません。

RewriteCond %{HTTP_Host} ^www\.static\..*$ [NC]
RewriteRule ^/robots\.txt$  /robots_static.txt [L]

Www.staticドメインから/robots.txtのリクエストが行われると、/ robots_static.txtファイルは/robots.txtであるかのように提供されます

7
Dave Lozier