chevron_left chevron_right
Login Register invert_colors photo_library


Stay updated and chat with others! - Join the Discord!
Thread Rating:
  • 1 Vote(s) - 5 Average


How does google know my bot is a bot? filter_list
Author
Message
RE: How does google know my bot is a bot? #8
(06-25-2016, 02:40 PM)m0dem Wrote: Selenium and mechanize are the real deal.  I was programming a web bot and I went through the old-fashioned HTTP modules (urllib...), but I kept getting blocked by anti-bot stuff until I used Selenium webdriver (and some mechanize).
Thanks, this sounds like a really good suggestion.

(06-25-2016, 06:01 PM)meow Wrote: Most (if not all) major search engines have IP blacklists, so your proxies could be the problem. Try doing whatever you're doing without a proxy and see if it still gives you that mobile verification page. If it still does it, keep reading.

Try adding some extra headers to make it more believable that you're an actual browser. I conducted a test where I sent a request to a website with my normal browser and then sent a request with mechanize (the only header I changed was the user-agent). The top is the mechanize request, the bottom is my browser's request.

[Image: 85ab308ae4f44726852798caf48e635d.png]

Also, does the browser object accept and store cookies and send them with every request? You might want to implement that if you haven't already, unless it's automatic. I wouldn't know since I use the requests module for all of my shit.

Cheers meow. I'll try out your suggestions. In terms of cookies, I think mechanize does already store them so I doub't that the issue.
(This post was last modified: 06-26-2016, 02:35 AM by superMAUS.)

Reply




Messages In This Thread
RE: How does google know my bot is a bot? - by superMAUS - 06-26-2016, 02:33 AM



Users browsing this thread: 1 Guest(s)