+- Sinisterly (https://sinister.ly)
+-- Forum: Coding (https://sinister.ly/Forum-Coding)
+--- Forum: Python (https://sinister.ly/Forum-Python)
+--- Thread: Ajax submitted Web Form data extraction using urllib2 (/Thread-Ajax-submitted-Web-Form-data-extraction-using-urllib2)
Ajax submitted Web Form data extraction using urllib2 - Psycho_Coder - 04-26-2014
Hello Guys,
I am relatively new to python and hasn't got much to do around it. I read a few things about the urllib and urllib2 library of python. Actually I was inspired by @Ligeti 's project and thought if making something useful.
Now there is a site suppose :-
http://www.somesite.com/login.php
Now there is a login form and when I fill up the form and submit it a new table is generated via the Ajax request and we can see the data. But we are not redirected to other page but same page. I tried to scrape the data but I am not getting it. Even after the login when I see the page source it still contains the login form code. Then in what manner can I extract the data after the new data are set by the Ajax call.
I did something like this to get the whole of the html source after the form is submitted but it only gives me the code of the login page :-
Code:
import mechanize
url = "http://www.somesite.com/login.php"
br = mechanize.Browser()
br.set_handle_robots(False) # ignore robots
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:20.0) Gecko/20100101 Firefox/20.0')]
br.open(url)
br.select_form(name="frmdetails")
br["Username"] = "Test"
br["Password"] = "1234"
res = br.submit()
content = res.read()
print contentCan you tell me if there is a way to do so ?
Ajax submitted Web Form data extraction using urllib2 - Psycho_Coder - 04-26-2014
Hello Guys,
I am relatively new to python and hasn't got much to do around it. I read a few things about the urllib and urllib2 library of python. Actually I was inspired by @Ligeti 's project and thought if making something useful.
Now there is a site suppose :-
http://www.somesite.com/login.php
Now there is a login form and when I fill up the form and submit it a new table is generated via the Ajax request and we can see the data. But we are not redirected to other page but same page. I tried to scrape the data but I am not getting it. Even after the login when I see the page source it still contains the login form code. Then in what manner can I extract the data after the new data are set by the Ajax call.
I did something like this to get the whole of the html source after the form is submitted but it only gives me the code of the login page :-
Code:
import mechanize
url = "http://www.somesite.com/login.php"
br = mechanize.Browser()
br.set_handle_robots(False) # ignore robots
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:20.0) Gecko/20100101 Firefox/20.0')]
br.open(url)
br.select_form(name="frmdetails")
br["Username"] = "Test"
br["Password"] = "1234"
res = br.submit()
content = res.read()
print contentCan you tell me if there is a way to do so ?
RE: Ajax submitted Web Form data extraction using urllib2 - Psycho_Coder - 04-28-2014
Can someone help me a bit regarding this ? its been two days since I posted this.
Any help would be appreciated!
RE: Ajax submitted Web Form data extraction using urllib2 - Psycho_Coder - 04-30-2014
Can I have some help here ?