The Traders' Den  

  The Traders' Den > Where we go to learn ..... > Site Announcements & Suggestions
 

Notices

Site Announcements & Suggestions This is where you should make your suggestions to us on how to improve your experience here and where to post about site problems/issues.
Moderators

View Poll Results: Should TTD allow spiderbots free access to allow for public searching of TTD threads?
No. The Trader's Den is private. What happens here, STAYS here!. 10 52.63%
Ban spidebots only from "non-informational" forums like the Lounge and Politics 7 36.84%
Let spiderbots crawl over ALL of our threads. We have nothing to hide! 1 5.26%
Spiders are yummy. Google "Spider sandwiches" for free recipes. 1 5.26%
Voters: 19. You may not vote on this poll

Reply
 
Thread Tools
  #1  
Old 2007-05-09, 08:25 AM
guygee's Avatar
guygee guygee is offline
Non Serviam
3.27 TB/25.56 TB/7.81
 
Join Date: Feb 2005
Location: Astro Intergalactic Infinity
Help!! Spiderbots crawling on my Threads!!!

It used to be that lossless music sites blocked spiders for purposes of privacy, but I notice that more recently several sites have made the dubious decision to allow them.

For example, as I write this, TTD has these "honored guests" online:
snbot(2) , google , baiduspider , yahoo! slurp(13) , gigabot(2).

This may be partly driven by a desire to popularize the particular lossless music site in question, and there may be more noble motives...such as showcasing the many informative and erudite discussions taking place on said lossless music site that will undoubtedly be of great educational value to the general public.

If only that were true in practice!

Instead, try googling this name of a Miles Davis bootleg and see the steaming pile'o'pages that tops the list:
"Sing In Singen". I ask anyone to read though the first entry that comes up and let me know if

1)If there is any useful information about the bootleg "Sing In Singen"
2) If the entire entry and how it comes up in the search is anything but an embarrassment to this proud site.

Let's try another search at random...oh, I don't know, how about this one:
"Led Zeppelin" "Freezer's Revenge"

Yes, the first entry in the search leads to the thread that will live in infamy...this is the thread that people refer to when disdainfully and dismissively putting down this great lossless music site, The Trader's Den.

On the other hand, there are many threads on this site that may be of great value to the reputation of this site and to the public at large, especially in the technobabble archives, and in the inactive archives as well. It is a pity that some selectivity cannot be applied to which threads get searched, and which are given their rightful place in the Recycle Bin of history.

I have not even brought up the issue that many people would probably prefer to have their activities here remain private.

Do not be fooled that your goofy "handle" will hide your identity from the likes of Choicepoint; companies that will dig up every bit of information on you...your political beliefs...your medical history...your "recreational proclivities"...any information that you may foolishly volunteer, plus almost all electronic information that you thought was private such as your credit history, your bank accounts, and your medical history..all to be made available to anyone who can pay the money for a "License" to know-all-about-you.

http://www.choicepoint.com/business/..._employ_1.html

Next time you go to a job interview, consider that your prospective new employer may well be holding ALL of that information in his hand...feel better now??

SO I bring this whole issue up to this site and I hope a few people will give it some thought and reply to this question:

Do YOU want spiderbots crawling all over your threads?
__________________

Constantly choosing the lesser of two evils is still choosing evil. --Jerry Garcia

Last edited by guygee; 2007-05-10 at 08:12 AM.
Reply With Quote Reply with Nested Quotes
  #2  
Old 2007-05-09, 09:20 AM
saltman's Avatar
saltman saltman is offline
Shareblue Platinum Member
471.23 GB/591.81 GB/1.26
 
Join Date: Dec 2004
Re: Help!! Spiderbots crawling on my Threads!!!

I personally don't have anything to hide from the spiders. But I don't really like them being here, either. The threads that they are allowed to search are governed by the traditional way all websites limit their behavior (assuming they follow the instructions). They are not allowed access to all of the threads.
__________________
68 Stat. 775, 50 U.S.C. 841-844
Reply With Quote Reply with Nested Quotes
  #3  
Old 2007-05-09, 10:31 AM
guygee's Avatar
guygee guygee is offline
Non Serviam
3.27 TB/25.56 TB/7.81
 
Join Date: Feb 2005
Location: Astro Intergalactic Infinity
Re: Help!! Spiderbots crawling on my Threads!!!

I personally don't have anything to hide from the cops, but that doesn't mean I let them in my house. The Bill of Rights gives me the right to stop the blue guys with guns at my doorstep, and I protect my rights by asserting them. I do this specifically to protect these same right for the generations that follow. If everyone just ignores their rights, then social convention soon changes, and the right is lost forever, or at least until another cycle of DarkAges/Renaissance/Enlightenment occurs (the last such cycle took about 2000 years).

I am not trying to insinuate that you are saying anything different saltman, but it is possible to lock out the spiders if TTD wants to. Maybe people just do not care, but I think they should.
__________________

Constantly choosing the lesser of two evils is still choosing evil. --Jerry Garcia
Reply With Quote Reply with Nested Quotes
  #4  
Old 2007-05-09, 10:49 AM
saltman's Avatar
saltman saltman is offline
Shareblue Platinum Member
471.23 GB/591.81 GB/1.26
 
Join Date: Dec 2004
Re: Help!! Spiderbots crawling on my Threads!!!

Quote:
Originally Posted by guygee
I am not trying to insinuate that you are saying anything different saltman, but it is possible to lock out the spiders if TTD wants to. Maybe people just do not care, but I think they should.
I am merely an underling here.... I am sure the super duper administrator that we have will chime in soon.

I think if the goal is to keep out prying eyes that it is unachievable. Any website that is on the web can be snooped at anytime. You can't keep out spiders that don't obey the robots.txt and that is only one manner of snooping. If the goal is to keep out the casual public that may be achievable. Well you can you can provide login access to restrict spiders to areas that people without permissions can't see... but if someone wants to snoop it's easy that's the nature of the world wide web. You already commented on the way to have this not be a problem.... don't post anything in public that you don't want the public to read.

What areas of the site do you think are inappropriate to show up in a search query?
__________________
68 Stat. 775, 50 U.S.C. 841-844
Reply With Quote Reply with Nested Quotes
  #5  
Old 2007-05-09, 11:11 AM
guygee's Avatar
guygee guygee is offline
Non Serviam
3.27 TB/25.56 TB/7.81
 
Join Date: Feb 2005
Location: Astro Intergalactic Infinity
Re: Help!! Spiderbots crawling on my Threads!!!

saltman - If I say "George W. Bush sucks donkey balls" in public, I will never get a job in my chosen profession...
(too late).

Actually, my internet footprint extends back well before the WWW, back to the usenet. Anyone that wants to can get a full read on my political beliefs. These days, you cannot work for the government if you do not have complete loyalty to the current administration. Ask all of those DOJ U.S. attorneys that were fired for purely political reasons.

Ask the United States Secretary of Housing and Urban Development, the 'honorable' Alphonso Jackson, after he fired a government contractor for not being loyal to the Chimperor:
Quote:
"He had made every effort to get a contract with HUD for 10 years," Jackson said of the prospective contractor. "He made a heck of a proposal and was on the (General Services Administration) list, so we selected him. He came to see me and thank me for selecting him. Then he said something ... he said, 'I have a problem with your president.'
"I said, 'What do you mean?' He said, 'I don't like President Bush.' I thought to myself, 'Brother, you have a disconnect -- the president is elected, I was selected. You wouldn't be getting the contract unless I was sitting here. If you have a problem with the president, don't tell the secretary.'

"He didn't get the contract," Jackson continued. "Why should I reward someone who doesn't like the president, so they can use funds to try to campaign against the president? Logic says they don't get the contract. That's the way I believe."
Source: Dallas Business Journal

So the problem is that almost anything you say can get you excluded from the economy...no job...blacklisted. Like Hollywood in the McCarthy era, but for everyone.

You tell me what should be kept private? I have felt this injustice in my own life, and I do not wish it on anyone else.
__________________

Constantly choosing the lesser of two evils is still choosing evil. --Jerry Garcia

Last edited by guygee; 2007-05-09 at 11:16 AM.
Reply With Quote Reply with Nested Quotes
  #6  
Old 2007-05-09, 11:29 AM
U2Lynne's Avatar
U2Lynne U2Lynne is offline
TTD Staff
474.39 GB/2.01 TB/4.34
 
Join Date: Oct 2004
Location: California
Re: Help!! Spiderbots crawling on my Threads!!!

I have thought about not having the Torrent forums 'public' and thus keep out the spiders, but when it's been brought up before, no one has really had a strong opinion about 'closing' them and so I've just left them alone. It's simple for me to make them not 'spiderable', but really, if people are worried about what will show up on a search engine, I don't think it is the torrent threads they should worry about, it would be The Lounge/Politics threads.
__________________
Five's Checksums Demystified - everything and anything you want to know about checksums
On a Mac? Get XLD to rip your CDs. Please see this guide - X Lossless Decoder (XLD): How to create flawless CD rips on Mac OS X


Reply With Quote Reply with Nested Quotes
  #7  
Old 2007-05-09, 11:36 AM
direwolf-pgh's Avatar
direwolf-pgh direwolf-pgh is offline
On the Beach
666.18 GB/1.29 TB/1.99
 
Join Date: Dec 2005
Location: down in the basement
Re: Help!! Spiderbots crawling on my Threads!!!

they tap the phones - they scan the websites
par for the course these days
Reply With Quote Reply with Nested Quotes
  #8  
Old 2007-05-09, 11:55 AM
guygee's Avatar
guygee guygee is offline
Non Serviam
3.27 TB/25.56 TB/7.81
 
Join Date: Feb 2005
Location: Astro Intergalactic Infinity
Re: Help!! Spiderbots crawling on my Threads!!!

Quote:
Originally Posted by U2Lynne
I have thought about not having the Torrent forums 'public' and thus keep out the spiders, but when it's been brought up before, no one has really had a strong opinion about 'closing' them and so I've just left them alone. It's simple for me to make them not 'spiderable', but really, if people are worried about what will show up on a search engine, I don't think it is the torrent threads they should worry about, it would be The Lounge/Politics threads.
Lynne - Priobably most torrent threads are fine, same goes for the technobabble threads. There are those exceptions, two of which I specifically pointed out. I have no idea if there is any way to lower the google priority once they are out there (a totally different issue, I imagine).

Personally, I vote on having the The Lounge/Politics threads being non-spiderable, but I leave that out there for discussion.

But consider this...one of the last calls for my professional research services was from somebody unofficially shilling for the NSA...calling for "advanced techniques for rapid data mining in constructing social network representations". Please chew on that one for awhile
__________________

Constantly choosing the lesser of two evils is still choosing evil. --Jerry Garcia
Reply With Quote Reply with Nested Quotes
  #9  
Old 2007-05-09, 12:00 PM
saltman's Avatar
saltman saltman is offline
Shareblue Platinum Member
471.23 GB/591.81 GB/1.26
 
Join Date: Dec 2004
Re: Help!! Spiderbots crawling on my Threads!!!

Does this mean I should be worried about my myspace page.... I jest.

We should make this a poll. I'm curious what the rest of the users would think. I agree that the forums mentioned seem logical. And I'm sure if there were certain threads in the forums that were acceptable that they could be actively hidden in the same manner. I'm not sure who determines whether certain ones are ok or not....

Can you edit your first post to make this a poll?
__________________
68 Stat. 775, 50 U.S.C. 841-844
Reply With Quote Reply with Nested Quotes
  #10  
Old 2007-05-09, 12:03 PM
dementrium's Avatar
dementrium dementrium is offline
433.71 GB/433.00 GB/1.00
 
Join Date: Dec 2005
Re: Help!! Spiderbots crawling on my Threads!!!

I don't like spider bots either. But, I admit they helped me, back when starting this torrent madness, to discover this one of a kind site. (I remember googling for Zep boots at that time )

Quote:
Originally Posted by guygee
companies that will dig up every bit of information on you...your political beliefs...your medical history...your "recreational proclivities"...any information
On the other hand it's quite true what you put there. No wonder I can't make a con t ract with this fu kci ng letf win gar go v' t.

May I ask if that data mining systems can sort out typos and spacements?
Reply With Quote Reply with Nested Quotes
  #11  
Old 2007-05-09, 12:06 PM
direwolf-pgh's Avatar
direwolf-pgh direwolf-pgh is offline
On the Beach
666.18 GB/1.29 TB/1.99
 
Join Date: Dec 2005
Location: down in the basement
Re: Help!! Spiderbots crawling on my Threads!!!

sounds like you already have been checked then guygee. Me too - couple times. no big deal.

except the one time they took all my trash & replaced it - that was creepy.
Reply With Quote Reply with Nested Quotes
  #12  
Old 2007-05-09, 12:50 PM
guygee's Avatar
guygee guygee is offline
Non Serviam
3.27 TB/25.56 TB/7.81
 
Join Date: Feb 2005
Location: Astro Intergalactic Infinity
Re: Help!! Spiderbots crawling on my Threads!!!

Quote:
Originally Posted by dementrium
On the other hand it's quite true what you put there. No wonder I can't make a con t ract with this fu kci ng letf win gar go v' t.

May I ask if that data mining systems can sort out typos and spacements?
You bring up two good points at once...

Data mining can randomly choose a sample of anything "out of the ordinary", such as non-dictionary words in a text file. Undoubtedly it is trivial to bring up a small sampling of these instances so they can be chosen for closer review. But these are too common for anything but a very low-probability hit.

More likely, effective data mining works best on specific keywords in combination with "suspect" origins and/or destinations ("suspect" possibly including your peacenik Unitarian grandmother).

Data mining is so easily defeated by the technological knowledgeable and the determined that it is obvious its only utility and its only intentional purpose is in commonplace domestic spying, like COINTELPRO on super-steriods.

If you chewed on the other tidbit, then bite on that one.

Working on information at least two years old and at least thrice removed from the classified world, my source told me that the NSA has (had) the resources to store two days worth of all U.S. and domestic/international communications. That is the window that they have(had) for winnowing the set down to "communications of interest".

I am sure all of these resources are there just to protect us from terrorists...well...maybe a little political blackmail...and corporate espionage...and why not profit on the stock market to support covert ops for "national security...and we always hated those Quakers anyways...

Those with the guns rule the world...those with the information manipulate those with the guns...
__________________

Constantly choosing the lesser of two evils is still choosing evil. --Jerry Garcia

Last edited by guygee; 2007-05-09 at 12:58 PM.
Reply With Quote Reply with Nested Quotes
  #13  
Old 2007-05-09, 12:59 PM
direwolf-pgh's Avatar
direwolf-pgh direwolf-pgh is offline
On the Beach
666.18 GB/1.29 TB/1.99
 
Join Date: Dec 2005
Location: down in the basement
Re: Help!! Spiderbots crawling on my Threads!!!

having configured sniffers & data collectors for businesses...

it would be easier for the gov. to simply collect data as it travels 'through the pipes' - rather than having spiders run around arbitrarily gathering (thats for public/private companies).

long story short - its collected before it hits the sites - they control/own the pipes.
Reply With Quote Reply with Nested Quotes
  #14  
Old 2007-05-09, 01:05 PM
guygee's Avatar
guygee guygee is offline
Non Serviam
3.27 TB/25.56 TB/7.81
 
Join Date: Feb 2005
Location: Astro Intergalactic Infinity
Re: Help!! Spiderbots crawling on my Threads!!!

Quote:
Originally Posted by direwolf-pgh
having configured sniffers & data collectors for businesses...

it would be easier for the gov. to simply collect data as it travels 'through the pipes' - rather than having spiders run around arbitrarily gathering (thats for public/private companies).

lost story short - its collected before it hits the sites - they control/own the pipes.
Yes, true...we definitely strayed off-topic.

But there is still the issue of companies serving as data-storage proxies for the US government, companies who will research your "internet footprint" among many other data sources. Spiders play an important role in this. Consider, in the Clinton years and before you could never get a Top Secret clearance if you were a "commie". Now the target has shifted...you need to be a "Bushie" to get that clearance.

Any disagreements with that observation?
__________________

Constantly choosing the lesser of two evils is still choosing evil. --Jerry Garcia
Reply With Quote Reply with Nested Quotes
  #15  
Old 2007-05-09, 01:11 PM
direwolf-pgh's Avatar
direwolf-pgh direwolf-pgh is offline
On the Beach
666.18 GB/1.29 TB/1.99
 
Join Date: Dec 2005
Location: down in the basement
Re: Help!! Spiderbots crawling on my Threads!!!

nope.

im just in a chatty mood & there is nothing going on in the lounge today.
Reply With Quote Reply with Nested Quotes
Reply

The Traders' Den > Where we go to learn ..... > Site Announcements & Suggestions


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forums


All times are GMT -5. The time now is 08:46 AM.


Powered by: vBulletin, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004 - , TheTradersDen.org - All Rights Reserved - Hosted at QuickPacket