PHOEN$X said:I caught some spiders browsing Prime! I wonder if they are responsible for some of the performance problems we've had recently.
I can't think of a reason not to use a robots.txt file, and I would think Lud uses one. As far as bandwidth use by the spiders, I would make sure that all the uploaded images, videos and any other large file source wouild be stored in a directory that was prohibited by the robots.txt file. That way, only text is indexed, which is what I would want for SE placement purposes.PHOEN$X said:Gene, I was also surprised that the spider was so readily identifiable. I doubt it had to register first (I don't see it in the Members List. I had assumed that Lud would've implemented a robots.txt file to disallow indexing of these message boards, since it could dramatically increase traffic and bandwidth usage.
KGP said:Neo, remember when you posted about spiders indexing our profiles? I just checked to see what Jeeves was up to. Yep, viewing (indexing) user profiles.
KGP said:I can't think of a reason not to use a robots.txt file
well, first, i think i'd want to ask, is there any benefit to lud/users in letting spiders index prime?
if so, what's the problem with letting spiders index prime compared to the benefit of them doing so?
performance? relatively intermittent after the intial indexing, i'd bet.
it's the best nsx site on the internet and it's FREE, where else are we *really* going to invest our nsx-related time?
otoh, perhaps there's an economic advantage (to lud *and* users) in letting askjeeves/similar ilk collect data from this site. can't say/don't know, but if that were the case, who could blame lud from letting the system pay for itself/his years of hard work?
not i, as this may also relieve us of paying membership fees, april fools day jokes notwithstanding
as a matter of fact, although i don't really care for them much, i can't help but notice that prime has advertising, a la google, in various locations... and i think it's a good thing for him - cash in the pocket to offset his efforts/costs or DARE I SAY IT - allow prime to provide cash flow in return for his great work.
so, if there's some economic benefit (long or short term) in letting spiders crawl here, it makes perfect sense to me.
how 'bout we ask lud?
lud - what's the spider story here at prime?