chevron_left chevron_right
Login Register invert_colors photo_library


Stay updated and chat with others! - Join the Discord!
Thread Rating:
  • 1 Vote(s) - 5 Average


How does google know my bot is a bot? filter_list
Author
Message
RE: How does google know my bot is a bot? #21
(07-22-2016, 10:19 PM)0M4R Wrote: It could be the web client, i'm not sure if this is the same with python as i have very little knowledge in that. But with vb if you use the internet explorer client it asks you for the verification however with google chrome it does not.

Stop.

[+] 1 user Likes meow's post
Reply

RE: How does google know my bot is a bot? #22
(07-22-2016, 11:55 PM)meow Wrote:
(07-22-2016, 10:19 PM)0M4R Wrote: It could be the web client, i'm not sure if this is the same with python as i have very little knowledge in that. But with vb if you use the internet explorer client it asks you for the verification however with google chrome it does not.

Stop.

What do you mean?
Donate to be in my siggy!
Spoiler: Donators
@Ender ♦ @Customer-chan ♦ @Diego ♦ @Logo ♦ @Bish0pQ ♦ @Altair ♦ @Botanicals

Reply

RE: How does google know my bot is a bot? #23
it detect the user-agent , and for those who said selenium..it is detectable for some web applications like twitter , for example I have create a bot for register new user in twitter with browser module which is based on selenuim .. and I got blocked from twitter ...and I guess twitter knows based on the time spend to fill the registration form which is second or two...so I did sleep 2,3 or 4 second every time filling a field ..so it's working ......and for google use list of user-agents and time sleep. It's working for me.

Reply

RE: How does google know my bot is a bot? #24
(08-16-2016, 11:08 PM)0x6f776c Wrote: it detect the user-agent

Every HTTP software detects the user-agent, however that is not the problem here. Please read the thread before taking a swing at the problem.

Reply

RE: How does google know my bot is a bot? #25
Just to be clear, Selenium solved this issue. Definetly recommend it, although the browser window pop is somewhat annoying (but AMAZING for debugging).

Reply

RE: How does google know my bot is a bot? #26
If you want something that is headless (ie. no browser window) check out PhantomJS.

PhantomJS is what I use at work, we make microservices that scrape different things and then have other main applications that interact with the microservice APIs. You can then run many hundreds of scrapers on a singe server.
(This post was last modified: 01-30-2017, 06:15 PM by Hoss.)
Hoss#2951

Reply






Users browsing this thread: 1 Guest(s)