seo and htaccess question

[Anonymous]

So, I made htaccess file to redirect anyone who enters my site from www.domain.com to www.domain.com/en/ (I have a multilanguage site). So, I submitted my robots.txt file to google webmasters. I have only this inside:

User-agent: *
Allow: /
Disallow: /admin

But google webmasters say that it can't reach robots.txt and none of my 5600 pages are indexed... Does it have something to do with the redirect? It's not showing any other errors... I'll provide more info if you need. Thanks

 

[Anonymous]

That will be probably htaccess fault.

If you're storing your robots.txt on www.domain.com, then when trying to access file you would get redirected to www.domain.com/en/robots.txt

If you're using htaccess to redirect to en folder, you would only need to move copy robots.txt.
If you're rewriting URL, you would need to add exceptions for your mod_rewrite

 

[Anonymous]

I tried adding the exceptions like this:

RewriteCond %{HTTP_HOST} ^domain\.com$ [NC]
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^(.*)$ http://domain.com/en/ [R=301,L]
RewriteRule ^(.*)/$ /index.php?lang=$1
RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteCond %{REQUEST_URI} !^/sitemap.xml$

hope it works... I think I need to wait google a couple of days...