I was thinking about adding a spider/bot to my directory where it will just spider the internal pages of websites that I have accepted in to my directory.
I was hoping there is a free or very cheap script I could use which would work with WSN Links to achieve this.
I have no clue how to do this, can anyone help me? Thanks.
phpdig is free, but neither it nor anything will work with WSN Links to do anything (though I won't rule out phpdig integration as an add-on in the future). For one thing you haven't specified what you want it to do. Full searches of their site content? Checking for recip links?
I just want it to automatically submit internal pages of websites that I have accepted into the directory. Also I want it to gather the page's Title name and meta tag description or some random text from the page. Than I want the search feature in WSN Links to be able to list these pages just like it does with any other site submitted in the directory. I was thinking of having the crawled internal pages all go into one category and than users will see them when they perform a search.
Comments on Adding a Spider to a directory?
Member
Usergroup: Customer
Joined: Jun 23, 2005
Total Topics: 12
Total Comments: 26
I was thinking about adding a spider/bot to my directory where it will just spider the internal pages of websites that I have accepted in to my directory.
I was hoping there is a free or very cheap script I could use which would work with WSN Links to achieve this.
I have no clue how to do this, can anyone help me? Thanks.
Member
Usergroup: Customer
Joined: Jun 23, 2005
Total Topics: 12
Total Comments: 26
bump
developer
Usergroup: Administrator
Joined: Dec 20, 2001
Location: Diamond Springs, California
Total Topics: 61
Total Comments: 7868
phpdig is free, but neither it nor anything will work with WSN Links to do anything (though I won't rule out phpdig integration as an add-on in the future). For one thing you haven't specified what you want it to do. Full searches of their site content? Checking for recip links?
Member
Usergroup: Customer
Joined: Jun 23, 2005
Total Topics: 12
Total Comments: 26
I just want it to automatically submit internal pages of websites that I have accepted into the directory. Also I want it to gather the page's Title name and meta tag description or some random text from the page. Than I want the search feature in WSN Links to be able to list these pages just like it does with any other site submitted in the directory. I was thinking of having the crawled internal pages all go into one category and than users will see them when they perform a search.
Forum Regular
Usergroup: Customer
Joined: Feb 19, 2004
Location: Michigan
Total Topics: 57
Total Comments: 185
I've used sphider in the past, seems ok.
http://www.cs.ioc.ee/~ando/sphider/
Member
Usergroup: Customer
Joined: Jun 23, 2005
Total Topics: 12
Total Comments: 26
Thanks mrowton, Iv'e sent you a PM