Help!! Spiderbots crawling on my Threads!!!

[guygee]

It used to be that lossless music sites blocked spiders for purposes of privacy, but I notice that more recently several sites have made the dubious decision to allow them.

For example, as I write this, TTD has these "honored guests" online:
snbot(2) , google , baiduspider , yahoo! slurp(13) , gigabot(2).

This may be partly driven by a desire to popularize the particular lossless music site in question, and there may be more noble motives...such as showcasing the many informative and erudite discussions taking place on said lossless music site that will undoubtedly be of great educational value to the general public.

If only that were true in practice! :lol

Instead, try googling this name of a Miles Davis bootleg and see the steaming pile'o'pages that tops the list:
"Sing In Singen". I ask anyone to read though the first entry that comes up and let me know if

1)If there is any useful information about the bootleg "Sing In Singen"
2) If the entire entry and how it comes up in the search is anything but an embarrassment to this proud site.

Let's try another search at random...oh, I don't know, how about this one:
"Led Zeppelin" "Freezer's Revenge"

Yes, the first entry in the search leads to the thread that will live in infamy...this is the thread that people refer to when disdainfully and dismissively putting down this great lossless music site, The Trader's Den.

On the other hand, there are many threads on this site that may be of great value to the reputation of this site and to the public at large, especially in the technobabble archives, and in the inactive archives as well. It is a pity that some selectivity cannot be applied to which threads get searched, and which are given their rightful place in the Recycle Bin of history.

I have not even brought up the issue that many people would probably prefer to have their activities here remain private.

Do not be fooled that your goofy "handle" will hide your identity from the likes of Choicepoint; companies that will dig up every bit of information on you...your political beliefs...your medical history...your "recreational proclivities"...any information that you may foolishly volunteer, plus almost all electronic information that you thought was private such as your credit history, your bank accounts, and your medical history..all to be made available to anyone who can pay the money for a "License" to know-all-about-you.

http://www.choicepoint.com/business/pre_employ/pre_employ_1.html

Next time you go to a job interview, consider that your prospective new employer may well be holding ALL of that information in his hand...feel better now??

SO I bring this whole issue up to this site and I hope a few people will give it some thought and reply to this question:

Do YOU want spiderbots crawling all over your threads?

[saltman]

I personally don't have anything to hide from the spiders. But I don't really like them being here, either. The threads that they are allowed to search are governed by the traditional way all websites limit their behavior (assuming they follow the instructions). They are not allowed access to all of the threads.

[guygee]

I personally don't have anything to hide from the cops, but that doesn't mean I let them in my house. The Bill of Rights gives me the right to stop the blue guys with guns at my doorstep, and I protect my rights by asserting them. I do this specifically to protect these same right for the generations that follow. If everyone just ignores their rights, then social convention soon changes, and the right is lost forever, or at least until another cycle of DarkAges/Renaissance/Enlightenment occurs (the last such cycle took about 2000 years).

I am not trying to insinuate that you are saying anything different saltman, but it is possible to lock out the spiders if TTD wants to. Maybe people just do not care, but I think they should.

[saltman]

I am not trying to insinuate that you are saying anything different saltman, but it is possible to lock out the spiders if TTD wants to. Maybe people just do not care, but I think they should. I am merely an underling here.... I am sure the super duper administrator that we have will chime in soon.

I think if the goal is to keep out prying eyes that it is unachievable. Any website that is on the web can be snooped at anytime. You can't keep out spiders that don't obey the robots.txt and that is only one manner of snooping. If the goal is to keep out the casual public that may be achievable. Well you can you can provide login access to restrict spiders to areas that people without permissions can't see... but if someone wants to snoop it's easy that's the nature of the world wide web. You already commented on the way to have this not be a problem.... don't post anything in public that you don't want the public to read.

What areas of the site do you think are inappropriate to show up in a search query?

[guygee]

saltman - If I say "George W. Bush sucks donkey balls" in public, I will never get a job in my chosen profession...
(too late).

Actually, my internet footprint extends back well before the WWW, back to the usenet. Anyone that wants to can get a full read on my political beliefs. These days, you cannot work for the government if you do not have complete loyalty to the current administration. Ask all of those DOJ U.S. attorneys that were fired for purely political reasons.

Ask the United States Secretary of Housing and Urban Development, the 'honorable' Alphonso Jackson, after he fired a government contractor for not being loyal to the Chimperor:
"He had made every effort to get a contract with HUD for 10 years," Jackson said of the prospective contractor. "He made a heck of a proposal and was on the (General Services Administration) list, so we selected him. He came to see me and thank me for selecting him. Then he said something ... he said, 'I have a problem with your president.'
"I said, 'What do you mean?' He said, 'I don't like President Bush.' I thought to myself, 'Brother, you have a disconnect -- the president is elected, I was selected. You wouldn't be getting the contract unless I was sitting here. If you have a problem with the president, don't tell the secretary.'

"He didn't get the contract," Jackson continued. "Why should I reward someone who doesn't like the president, so they can use funds to try to campaign against the president? Logic says they don't get the contract. That's the way I believe." Source: Dallas Business Journal (http://dallas.bizjournals.com/dallas/stories/2006/05/08/story1.html)

So the problem is that almost anything you say can get you excluded from the economy...no job...blacklisted. Like Hollywood in the McCarthy era, but for everyone.

You tell me what should be kept private? I have felt this injustice in my own life, and I do not wish it on anyone else.

[U2Lynne]

I have thought about not having the Torrent forums 'public' and thus keep out the spiders, but when it's been brought up before, no one has really had a strong opinion about 'closing' them and so I've just left them alone. It's simple for me to make them not 'spiderable', but really, if people are worried about what will show up on a search engine, I don't think it is the torrent threads they should worry about, it would be The Lounge/Politics threads.

[direwolf-pgh]

they tap the phones - they scan the websites
par for the course these days

[guygee]

I have thought about not having the Torrent forums 'public' and thus keep out the spiders, but when it's been brought up before, no one has really had a strong opinion about 'closing' them and so I've just left them alone. It's simple for me to make them not 'spiderable', but really, if people are worried about what will show up on a search engine, I don't think it is the torrent threads they should worry about, it would be The Lounge/Politics threads.

Lynne - Priobably most torrent threads are fine, same goes for the technobabble threads. There are those exceptions, two of which I specifically pointed out. I have no idea if there is any way to lower the google priority once they are out there (a totally different issue, I imagine).

Personally, I vote on having the The Lounge/Politics threads being non-spiderable, but I leave that out there for discussion.

But consider this...one of the last calls for my professional research services was from somebody unofficially shilling for the NSA...calling for "advanced techniques for rapid data mining in constructing social network representations". Please chew on that one for awhile :disbelief

[saltman]

Does this mean I should be worried about my myspace page.... :lol I jest.

We should make this a poll. I'm curious what the rest of the users would think. I agree that the forums mentioned seem logical. And I'm sure if there were certain threads in the forums that were acceptable that they could be actively hidden in the same manner. I'm not sure who determines whether certain ones are ok or not....

Can you edit your first post to make this a poll?

[dementrium]

I don't like spider bots either. But, I admit they helped me, back when starting this torrent madness, to discover this one of a kind site. (I remember googling for Zep boots at that time :disbelief )

companies that will dig up every bit of information on you...your political beliefs...your medical history...your "recreational proclivities"...any information

On the other hand it's quite true what you put there. No wonder I can't make a con t ract with this fu kci ng letf win gar go v' t. :lol

May I ask if that data mining systems can sort out typos and spacements?

[direwolf-pgh]

sounds like you already have been checked then guygee. Me too - couple times. no big deal.

except the one time they took all my trash & replaced it - that was creepy.

[guygee]

On the other hand it's quite true what you put there. No wonder I can't make a con t ract with this fu kci ng letf win gar go v' t. :lol

May I ask if that data mining systems can sort out typos and spacements?

You bring up two good points at once...

Data mining can randomly choose a sample of anything "out of the ordinary", such as non-dictionary words in a text file. Undoubtedly it is trivial to bring up a small sampling of these instances so they can be chosen for closer review. But these are too common for anything but a very low-probability hit.

More likely, effective data mining works best on specific keywords in combination with "suspect" origins and/or destinations ("suspect" possibly including your peacenik Unitarian grandmother).

Data mining is so easily defeated by the technological knowledgeable and the determined that it is obvious its only utility and its only intentional purpose is in commonplace domestic spying, like COINTELPRO on super-steriods.

If you chewed on the other tidbit, then bite on that one. :down:

Working on information at least two years old and at least thrice removed from the classified world, my source told me that the NSA has (had) the resources to store two days worth of all U.S. and domestic/international communications. That is the window that they have(had) for winnowing the set down to "communications of interest".

I am sure all of these resources are there just to protect us from terrorists...well...maybe a little political blackmail...and corporate espionage...and why not profit on the stock market to support covert ops for "national security...and we always hated those Quakers anyways...

Those with the guns rule the world...those with the information manipulate those with the guns...

[direwolf-pgh]

having configured sniffers & data collectors for businesses...

it would be easier for the gov. to simply collect data as it travels 'through the pipes' - rather than having spiders run around arbitrarily gathering (thats for public/private companies).

long story short - its collected before it hits the sites - they control/own the pipes.

[guygee]

having configured sniffers & data collectors for businesses...

it would be easier for the gov. to simply collect data as it travels 'through the pipes' - rather than having spiders run around arbitrarily gathering (thats for public/private companies).

lost story short - its collected before it hits the sites - they control/own the pipes.

Yes, true...we definitely strayed off-topic.

But there is still the issue of companies serving as data-storage proxies for the US government, companies who will research your "internet footprint" among many other data sources. Spiders play an important role in this. Consider, in the Clinton years and before you could never get a Top Secret clearance if you were a "commie". Now the target has shifted...you need to be a "Bushie" to get that clearance.

Any disagreements with that observation?

[direwolf-pgh]

nope. :)

im just in a chatty mood & there is nothing going on in the lounge today.

[saltman]

having configured sniffers & data collectors for businesses...

it would be easier for the gov. to simply collect data as it travels 'through the pipes' - rather than having spiders run around arbitrarily gathering (thats for public/private companies).

long story short - its collected before it hits the sites - they control/own the pipes.I used to be really into this kind of stuff many years ago. What's carnivore's replacement called.... That's some interesting reading there. ;)

[guygee]

nope. :)

im just in a chatty mood & there is nothing going on in the lounge today.
:lol

Well you sure got me to "tell all". :lol

Luckily I have my headphones on and I am listening to gd1973-05-26_Kezar-WBOTB...one of the greatest shows of all time:rockin:

But those spiders are still crawlin' on my threads... :disbelief

Wonder if any one else cares? :mad:

P.S. This third bloody mary is going down nicely as well, especially after last nights brawl where this old dude got beat up...funny how you never feel it until later :D

[direwolf-pgh]

had to remove it.. :)

love those '73 big 'eyes jams :thumbsup

74 - with just billy is a fav - not unlike 72 - one drummer is a good thing

[dementrium]

Great topic, indeed.
This thing is deep into my head since we have a brnad neu grate ufckin' letf w eengie guv' nt mustly inspired on such good ancient cummies' herita-ge.
Good thing is
I'm pretty confident on their mee diocre cy & lass-iness to implemnet such sophie-sticated data minnin' systems.

Well, one never knows til' it's too late. :hmm:

[guygee]

dementrium - Never underestimate the gravity of "guilt by association" :D

[direwolf-pgh]

weird part is.. lets say a spider grabs some info & that info is read incorrectly/taken out of context..

if the 'powers that be' wish to - they can take everything. home, possessions.. everything. No questions, no warning. accidents do happen. Its enough to make one paranoid for no reason.

so another vote for no spiders :lol:

[dementrium]

dementrium - Never underestimate the gravity of "guilt by association" :D

That's why I'm hanging a lot a TTD. Just to be tagged as "yet another junkie boot grabber freak". :clap:



Note to the mgmt: it's a joke, i love this place. :D

[guygee]

weird part is.. lets say a spider grabs some info & that info is read incorrectly/taken out of context..

if the 'powers that be' wish to - they can take everything. home, possessions.. everything. No questions, no warning. accidents do happen. Its enough to make one paranoid for no reason.

so another vote for no spiders :lol:

That is no joke...I have somebody who is very close to me that got convicted on "conspriracy to ...bla blah blaugh" under the RICO statutes. This person had no contraband at all, nor did he commit a crime...all that happened is that the Federal government threatened others who did commit crimes to testify against him, saying that he "conspired" to commit a crime. The weasel lying snitches got off light, the crooked cops that set him up are now in jail serving long sentences for other crimes, and the innocent "stand-up" guy got @% 25 fucking years. That is American justice for you...it is no joke.

Yes, he did lose his home, all of his possessions, everything; plus his parents lost almost everything too.

It is no joke.

[dementrium]

That is some fucked up shit. If that is north american justice, guess what you have at the very south.
Well, I know I'm going pretty out of topic.

Hope some others throw their points of view on this interesting matter.

[direwolf-pgh]

:( ouch.

[freezer]

weird part is.. lets say a spider grabs some info & that info is read incorrectly/taken out of context...........



Info taken out of context or read incorrectly........

:rolleyes:

Doesn't have to be a spider to do that.....humans do it well.....Then want to argue about it.

:lol

[pmonk]

nope. :)

im just in a chatty mood & there is nothing going on in the lounge today.

:hmm:

[guygee]

So the obvious is finally sinking in, like when the girl told me for the 12th time, "we are good friends...". Nobody much cares about the spiders.

Well people at least be aware of them!

Lynn, leave them spidergates wide open! Ride them spiders high, yeehaw! :rolleyes:

[guygee]

OK, poll added, typos and all. Can't figure out how to edit it to remove the typos and awkward phrasing.

Open for 14 days. I guess a lack of response to this poll is the same as a "I don't care" vote (or it could mean "I cannot read" :D ).

The point being, "This poll is not scientific and reflects the opinions of only those who have chosen to participate. Any sponsor of this poll is not responsible for the content or opinions expressed. This poll is provided for informational purposes only. No individual data will be retained."

[direwolf-pgh]

"This poll is not scientific and reflects the opinions of only those who have chosen to participate. Any sponsor of this poll is not responsible for the content or opinions expressed. This poll is provided for informational purposes only. No individual data will be retained."
:lol: :lol: :lol:

[U2Lynne]

So the obvious is finally sinking in, like when the girl told me for the 12th time, "we are good friends...". Nobody much cares about the spiders.

Well people at least be aware of them!

Lynn, leave them spidergates wide open! Ride them spiders high, yeehaw! :rolleyes:
Actually, I never just *do* something, I always try to discuss it with Staff first. So, I brought up the Spiders with them yesterday and this morning, every mod who responded agreed that we should close the doors to the Spiders in the Where We Go To Play forums. So, if you log out right now, you will not be able to see those forums.

[saltman]

Actually, I never just *do* something, I always try to discuss it with Staff first. So, I brought up the Spiders with them yesterday and this morning, every mod who responded agreed that we should close the doors to the Spiders in the Where We Go To Play forums. So, if you log out right now, you will not be able to see those forums.

How's that for customer service..... Thanks Lynne :thumbsup You rock, as always. :banger2:


mmmm... Spider sandwiches

http://i51.photobucket.com/albums/f384/saltman_was/03735b2t.jpg

[jdawson]

Thanks for the info. I've been curious about them for a while but didn't want to bother Lynne with a "what are these things?"

[guygee]

Actually, I never just *do* something, I always try to discuss it with Staff first. So, I brought up the Spiders with them yesterday and this morning, every mod who responded agreed that we should close the doors to the Spiders in the Where We Go To Play forums. So, if you log out right now, you will not be able to see those forums.

My apologies Lynn...my "sense of humor" is very strange (actually I am not sure if it can be called a "sense" at all, "nonsense" may be a better description.). I most admire the kind of humor displayed by the recently deceased great American humorist Kurt Vonnegut, who in turn was especially influenced by the late French author Ferdinand Céline (who in his turn aspired to write his dark humorist prose "on the cusp of the plane of delirium").

So please excuse my excess in the name of humor (exaggeration being a pillar of humor)...I want to emphasize that I never once doubted your good intentions, your judgment or your sincerity.

[U2Lynne]

All is good, Guy. Sometimes I post just to make a point to others who are reading a thread. Sometimes people seem to think that I 'rule' this site, but I try hard not to make that so. Sure, I'm the one who knows how the software works and all that, but I try to always get input from Staff before I 'do' anything.

edit: And you just reminded me that I want to steal that Vonnegut book from my son!

[dementrium]

I went with option #2. (Couldn't understand option #1).

One more time I say this is a very interesting topic that may affect everyone living in this planets, despite our different geographical locations and/or cultural gaps. :hmm:

So, here's to the management' quick reaction!

http://i160.photobucket.com/albums/t164/isby_2007/172.gif

[Chachi420]

I vote that the only page the spiders should be able to see is the homepage.

[guygee]

I vote that the only page the spiders should be able to see is the homepage.
If you check the wayback machine that is exactly the approach taken by "Sharing the Groove"

[guygee]

Thanks for the info. I've been curious about them for a while but didn't want to bother Lynne with a "what are these things?"
__________________


Originally Posted by Zepenita
Where in the hell is Trumbull County?!?

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

You are quite welcome :)

I also like your Zepenita quote. Here is my own answer,

"Trumbull County: Area of Ohio partially containing the former Congressional District of the Honorable James Traficant, Congressman for Life until the FBI finally took him down. Free James Traficant! (http://www.freetraficant.com/) Now Beam Me Up! (http://www.traficant.com/) :clap:

[KoolKat]

If I say "George W. Bush sucks donkey balls" in public, I will never get a job in my chosen profession...


:hmm: have i got this right guy..you mean he took what you wanted to do??? :lol :lol

K_K

PS: I say make it private whether there is something to hide or not.
It's a club that is members only.OUR club
What is typed is typed for members only
All torrents are private passkey for members only.


That says that it should be all private for members only to me

I dont remember seeing "Itsy Bitsy" as a member so no goodies..or baddies should be revealed until you are a member i feel.

Just my 2 cents

K_K

[guygee]

:hmm: have i got this right guy..you mean he took what you wanted to do??? :lol :lol

K_K

No K_K, you are wrong...good "SWAG" though ("Stupid Wild-Assed Guess", in your case) :D

BTW, what is your middle name? Does it begin with a K? Does rhyme with "Queunt"?

Well no matter, KKK wannabe fondler of Bush's Ball's, whatever gets you though your night.

(Sigh...another typical TTD thread...ends in senseless insults...me as guilty as "KKK"...what else is new...vote in the poll, say goodbye to this stupid thread, I sure as hell will)

[KoolKat]

Hahhahaha...you took it as an insult..plonker.
It was a play on words joke share of what you wrote.
Now YOU insult me

But if you wanna insult me go ahead chap...it shows your IQ.

I did add a PS with my thourghts on the spider issue so the reply was a members opinion on your thread question.

Read before you crack ,

K_K

[guygee]

Hahhahaha...you took it as an insult..plonker.
It was a play on words joke share of what you wrote.
Now YOU insult me

But if you wanna insult me go ahead chap...it shows your IQ.

I did add a PS with my thourghts on the spider issue so the reply was a members opinion on your thread question.

Read before you crack ,

K_K

Exactly my point K_K, it seems impossible to have a conversation here without it turning into some form of "oneupsmanship". OK, good one...I am the fool here...why does that make you feel better?

[KoolKat]

It wasn't a case of "oneupmanship" chap.
I gave my thourghts to your thread subject and joshed with a sentence that you quoted like a buddy drinking with you in a bar that came back with a one liner.

I am only sorry that typed words do not show expression or tone of how it would be said well enough.

Mis-interpretation is all that is at fault here.
;)

K_K

[guygee]

Ha! I seem to be getting oversensitive in my old age, like some kind of damned grandmother! (Now I am waiting for an army of grandmothers to attack me...Good! It will be more female attention than I've had in ages :lol: )

[guygee]

Not exactly "Breaking News", but still valid...I just did notice this news item regarding bt.etree:


9/22/2006
Ongoing problems with bittorrent include degraded performance thanks to torrent indexing sites spidering our goods. Media Tracker, a company hired by the RIAA is deliberately poisoning our torrents. Links between db & bt are no longer active.