| |
 |
|
 |
 |
|
 |
|
|
|
| Author |
Message |
Roy Schestowitz
Guest
|
 Posted: Sun Aug 28, 2005 6:31 am Post subject: robots.txt Variations? |
 |
|
I am hoping that someone in this group can help me out. For the past few
months I have been spotting errors for odd variations of the file
robots.txt. (among others)
Putting mistaken bots aside, there maybe would be one error for every ~100
visits, so I still have a very frequent look at the error logs (trying to
identify internal broken link), but I sometimes get unexplained errors,
e.g. so far this month:
/robots1.txt 8 times this month
/zzrobots.txt 4
The rest might be human errors:
/robots.tx 1
/robotsxx.txt 1
Is it possible that some crawlers 'extended' this type of protocol?
Even /sitemap.rdf has been requested twice even though I haven't signed up
with Google Site Maps. Can all of the above just be visitors that temper
with the server? They seem to come from addresses that do not contain
numbers, but still have obscure domains.
Many thanks in advance,
Roy
|
|
| Back to top |
|
|
|
|
|
|
|
| Author |
Message |
John Bokma
Guest
|
| Posted: Sun Aug 28, 2005 8:04 am Post subject: Re: robots.txt Variations? |
|
|
Roy Schestowitz <newsgroups@schestowitz.com> wrote:
| Quote: | I am hoping that someone in this group can help me out. For the past
few months I have been spotting errors for odd variations of the file
robots.txt. (among others)
Putting mistaken bots aside, there maybe would be one error for every
~100 visits, so I still have a very frequent look at the error logs
(trying to identify internal broken link), but I sometimes get
unexplained errors, e.g. so far this month:
/robots1.txt 8 times this month
/zzrobots.txt 4
The rest might be human errors:
/robots.tx 1
/robotsxx.txt 1
|
I'll check my error log...
| Quote: | Is it possible that some crawlers 'extended' this type of protocol?
Even /sitemap.rdf has been requested twice even though I haven't
signed up with Google Site Maps. Can all of the above just be visitors
that temper with the server? They seem to come from addresses that do
not contain numbers, but still have obscure domains.
|
[192.55.214.54] zzrobots.txt
[205.236.116.250] robots1.txt
And several requests for sitemap.rdf
--
John Perl SEO tools: http://johnbokma.com/perl/
Experienced (web) developer: http://castleamber.com/
Get a SEO report of your site for just 100 USD:
http://johnbokma.com/websitedesign/seo-expert-help.html
|
|
| Back to top |
|
|
|
|
|
|
|
| Author |
Message |
Roy Schestowitz
Guest
|
| Posted: Sun Aug 28, 2005 10:13 am Post subject: Re: robots.txt Variations? |
|
|
__/ On Sunday 28 August 2005 10:04, [John Bokma] wrote : \__
| Quote: | Roy Schestowitz <newsgroups@schestowitz.com> wrote:
I am hoping that someone in this group can help me out. For the past
few months I have been spotting errors for odd variations of the file
robots.txt. (among others)
Putting mistaken bots aside, there maybe would be one error for every
~100 visits, so I still have a very frequent look at the error logs
(trying to identify internal broken link), but I sometimes get
unexplained errors, e.g. so far this month:
/robots1.txt 8 times this month
/zzrobots.txt 4
The rest might be human errors:
/robots.tx 1
/robotsxx.txt 1
I'll check my error log...
Is it possible that some crawlers 'extended' this type of protocol?
Even /sitemap.rdf has been requested twice even though I haven't
signed up with Google Site Maps. Can all of the above just be visitors
that temper with the server? They seem to come from addresses that do
not contain numbers, but still have obscure domains.
[192.55.214.54] zzrobots.txt
[205.236.116.250] robots1.txt
And several requests for sitemap.rdf
|
[Sun Aug 28 07:21:56 2005] [error] [client 205.236.116.250] File does not
exist: /home/schestow/public_html/robots1.txt
which is a match (the latest error) - reverse DNS comes up with:
master.carrefourinternet.com
I have checked some of the other IP's in the past, but they appeared to have
come from completely different sources. Would inclusion in the IP deny list
be worthwhile? It's a recurring theme, but maybe a request for robots.txt
is subsequently made... and if so, why?!?!
Roy
|
|
| Back to top |
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(../../ad_network_ads_380.txt) is not within the allowed path(s): (/var/www/vhosts/jamroll.net/httpdocs:/tmp) in /var/www/vhosts/jamroll.net/httpdocs/support/ad_network_380.php on line 38
Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(../../../ad_network_ads_380.txt) is not within the allowed path(s): (/var/www/vhosts/jamroll.net/httpdocs:/tmp) in /var/www/vhosts/jamroll.net/httpdocs/support/ad_network_380.php on line 38
Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(../../../../ad_network_ads_380.txt) is not within the allowed path(s): (/var/www/vhosts/jamroll.net/httpdocs:/tmp) in /var/www/vhosts/jamroll.net/httpdocs/support/ad_network_380.php on line 38
Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(../../../../../ad_network_ads_380.txt) is not within the allowed path(s): (/var/www/vhosts/jamroll.net/httpdocs:/tmp) in /var/www/vhosts/jamroll.net/httpdocs/support/ad_network_380.php on line 38
Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(../../../../../../../ad_network_ads_380.txt) is not within the allowed path(s): (/var/www/vhosts/jamroll.net/httpdocs:/tmp) in /var/www/vhosts/jamroll.net/httpdocs/support/ad_network_380.php on line 38
Warning: file_exists() [function.file-exists]: open_basedir restriction in effect. File(../../../../../../../../../ad_network_ads_380.txt) is not within the allowed path(s): (/var/www/vhosts/jamroll.net/httpdocs:/tmp) in /var/www/vhosts/jamroll.net/httpdocs/support/ad_network_380.php on line 38
Powered by phpBB © 2001, 2002 phpBB Group
|
|
|
