Webmastersite.net
Register Log In

Adding a Spider to a directory?

Comments on Adding a Spider to a directory?

URL Direct
Member

Usergroup: Customer
Joined: Jun 24, 2005

Total Topics: 12
Total Comments: 26
Posted Aug 11, 2005 - 1:50 AM:

I was thinking about adding a spider/bot to my directory where it will just spider the internal pages of websites that I have accepted in to my directory.

I was hoping there is a free or very cheap script I could use which would work with WSN Links to achieve this.

I have no clue how to do this, can anyone help me? Thanks.
URL Direct
Member

Usergroup: Customer
Joined: Jun 24, 2005

Total Topics: 12
Total Comments: 26
Posted Aug 13, 2005 - 9:50 AM:

bump sad If Paul can't help me than I have no hope..... *begs*
Paul
developer

Usergroup: Administrator
Joined: Dec 20, 2001
Location: Diamond Springs, California

Total Topics: 61
Total Comments: 7868
Paul
Posted Aug 14, 2005 - 6:21 AM:

phpdig is free, but neither it nor anything will work with WSN Links to do anything (though I won't rule out phpdig integration as an add-on in the future). For one thing you haven't specified what you want it to do. Full searches of their site content? Checking for recip links?
URL Direct
Member

Usergroup: Customer
Joined: Jun 24, 2005

Total Topics: 12
Total Comments: 26
Posted Aug 15, 2005 - 12:37 AM:

I just want it to automatically submit internal pages of websites that I have accepted into the directory. Also I want it to gather the page's Title name and meta tag description or some random text from the page. Than I want the search feature in WSN Links to be able to list these pages just like it does with any other site submitted in the directory. I was thinking of having the crawled internal pages all go into one category and than users will see them when they perform a search.
mrowton
Forum Regular

Usergroup: Customer
Joined: Feb 19, 2004
Location: Michigan

Total Topics: 57
Total Comments: 185
mrowton
Posted Aug 15, 2005 - 2:55 PM:

I've used sphider in the past, seems ok.

http://www.cs.ioc.ee/~ando/sphider/
URL Direct
Member

Usergroup: Customer
Joined: Jun 24, 2005

Total Topics: 12
Total Comments: 26
Posted Aug 15, 2005 - 9:13 PM:

Thanks mrowton, Iv'e sent you a PMsmiling face
Search thread for
Download thread as
  • 0/5
  • 1
  • 2
  • 3
  • 4
  • 5



This thread is closed, so you cannot post a reply.