+- Sinisterly (https://sinister.ly)
+-- Forum: General (https://sinister.ly/Forum-General)
+--- Forum: The Lounge (https://sinister.ly/Forum-The-Lounge)
+--- Thread: Bronze Search engine for crawling? (/Thread-Bronze-Search-engine-for-crawling)
Search engine for crawling? - Cyb3rNuX - 09-06-2017
I'm just curious, does search engine for crawling exist?
Basically a search engine (or api at least) that can do unlimited searches (without proxy etc) without blocking/blacklisting my IP?
As far as i know, google has search limits, so other "popular" search engines.
Any suggestions?
RE: Search engine for crawling? - Slacker - 09-06-2017
Google is one hell of a web site brohamski... Just throwing that out there lol
https://www.webcrawler.com/
https://www.deepcrawl.com
https://scrapy.org/
monstercrawler.com
nutch.apache.org/
or develop your own:
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&cad=rja&uact=8&ved=0ahUKEwjs8LKIlJHWAhUJ_4MKHTXqA70QFgh3MAk&url=https%3A%2F%2Fmedium.com%2Fpython-pandemonium%2Fdevelop-your-first-web-crawler-in-python-scrapy-6b2ee4baf954&usg=AFQjCNHB0lOakIfP5I8dLj9b6hW9uXKGpg
RE: Search engine for crawling? - Cyb3rNuX - 09-06-2017
(09-06-2017, 07:11 PM)Slacker Wrote: Google is one hell of a web site brohamski... Just throwing that out there lol
https://www.webcrawler.com/
https://www.deepcrawl.com
https://scrapy.org/
monstercrawler.com
nutch.apache.org/
or develop your own:
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&cad=rja&uact=8&ved=0ahUKEwjs8LKIlJHWAhUJ_4MKHTXqA70QFgh3MAk&url=https%3A%2F%2Fmedium.com%2Fpython-pandemonium%2Fdevelop-your-first-web-crawler-in-python-scrapy-6b2ee4baf954&usg=AFQjCNHB0lOakIfP5I8dLj9b6hW9uXKGpg
Thanks, i'll test those
