chevron_left chevron_right
Login Register invert_colors photo_library
Stay updated and chat with others! - Join the Discord!
Thread Rating:
  • 2 Vote(s) - 2 Average


Simple Python Proxy Scraper filter_list
Author
Message
RE: Simple Python Proxy Scraper #5
My multithreading proxy scraper:
Code:
# -*- coding: utf-8 -*-
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool
import re, os
from urllib2 import urlopen

Path = os.path.dirname(os.path.realpath(__file__))

with open(Path+'\\url.txt', 'r') as file:
    urls = file.readlines()
    file.close()

def parseproxy(url):
    try:
        source = urlopen(url).read()
    except:
        return None

    proxies = re.findall( r'[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\:[\d]{1,6}', source[5:], re.M|re.I)

    with open(Path+'\\proxy.txt', "a") as file:
        for proxy in proxies:
            file.write(proxy+'\n')    
        file.close()

    print '[PARSED] - ', url.strip(), '['+str(len(proxies))+']'

pool = ThreadPool(100)
results = pool.map(parseproxy, urls)

pool.close()

pool.join()
example file url.txt:
Code:
http://best-proxy-list-ips.blogspot.com/feeds/posts/default?alt=rss
http://bestpremiumproxylist.blogspot.ru/feeds/posts/default?alt=rss

Reply




Messages In This Thread
Simple Python Proxy Scraper - by 720 - 06-03-2016, 04:31 PM
RE: Simple Python Proxy Scraper - by insidious - 06-03-2016, 05:23 PM
RE: Simple Python Proxy Scraper - by 720 - 06-04-2016, 11:50 AM
RE: Simple Python Proxy Scraper - by _t_ - 06-05-2016, 02:30 AM
RE: Simple Python Proxy Scraper - by BadSnow - 01-06-2017, 09:15 PM



Users browsing this thread: 1 Guest(s)