Hacking HC GUI (basics) 04-17-2014, 09:55 AM
#1
Hello
(this is not a tutorial... it is a project)
Introduction
If you didn't read my thread about "Maybe you want to read this before programming" (I know... bad title), then you maybe should, because this work I am presenting here is nothing but an echo for that thread!
This project was born in "Open projects (Workshop)", where I suggested making a new GUI for HC as a practice, where you (members of HC) get a chance to work, study and research/investigate with me the following:
There is more to this list, but for now I am reporting phase one only!
So the idea is:
The Project
As for scraping HTML I showed some examples in my "HTML Scraping Project" So, baybe you should read that for more information... the only thing I changed is the use of "urllib" library, because of the security check we have in the forum, the server doesn't allow bots to connect to the forum, so this following code will not get the page:
Output:
So I had to use mechanize library which is a "Stateful programmatic web browsing in Python, after Andy Lester’s Perl module", anyway with mechanize library you can make a browser and set the user-agent of that browser! And this way the server will not suspect that you are a bot/script (in theory... read the note at the end).
OK, enough talking, here is the scraper code:
(The code need clean up, I will do it today and update)
So ... how does it work?
The following code is the one responsible of downloading the web page, we make a browser br and makes it ignore robot check (robot.txt), and after that we set the user-agent, which in my case is 'User-agent', 'Mozilla/5.0 (X11; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0'
After that we open the web-page (index.php) and read it: htmltext = htmlpage.read()
OK, now we get the page we will need to scrape it, I noticed that MyBB for some reason are presenting the forum (sections and links) using a table, which is for me personally a bad practice, because I would use CSS to style the forum! Anyway...
So, if you check the source of the index.php page you'll find that each section is represented by a table! Which makes it easy to scrape, I first used BeautifulSoup, but things got complicated (I hated it) and I went back to Regex, my favorite method anyway!
So, size of tables now should be 10 and not 9 because the last table/section is the status of the forum... which is not of our interest, so now we have the tables, we loop to get the title and sub-sections of each table/section.
One think to mention here, is that I didn't go deep into the forum (I didn't scrape the threads of each section), for that I would need to do web-crawling and I wanted to keep things simple (so far).
The code needs more work but I am tired now (I will clean it later):
As you can see we will store the information in a list forum, this forum we will use later to build our dictionary object, the function "ListToDict" was inspired/found here
The rest of the code is clear I guess, so I will not go through that now, if you have questions please comment and I will try my best to help!
The Interface
And here is the final result:
![[Image: tqZe503.png]](http://i.imgur.com/tqZe503.png)
Next is to scrape the links (href of each section) and make the title click-able (Will redirect the user to the forum)
Conclusion
This is just the beginning of the project, there is a second phase where I should make this GUI functional and acceptable (add animation, background, colors... etc.), so you guys still have a chance to join me, or to join one of my other projects!
This is all for the moment... please leave your comments!
Thanks
Ligeti
[note] @bluedog.tar.gz (the admin) is making some changes in the forum, so if you'll face issues using mechanize (because I did before) you can download the page manually and open it as a file!
(this is not a tutorial... it is a project)
Introduction
If you didn't read my thread about "Maybe you want to read this before programming" (I know... bad title), then you maybe should, because this work I am presenting here is nothing but an echo for that thread!
This project was born in "Open projects (Workshop)", where I suggested making a new GUI for HC as a practice, where you (members of HC) get a chance to work, study and research/investigate with me the following:
- Python: HTML Scraping
- Python: working with JSON
- JS: working with D3JS
There is more to this list, but for now I am reporting phase one only!
So the idea is:
- Scrape information form the forum
- Convert the data/information to JSON format
- Save the JSON format in a file
- Load the data using JS/D3JS
- Graph the information using JS/D3JS/SVG
The Project
As for scraping HTML I showed some examples in my "HTML Scraping Project" So, baybe you should read that for more information... the only thing I changed is the use of "urllib" library, because of the security check we have in the forum, the server doesn't allow bots to connect to the forum, so this following code will not get the page:
Spoiler: Testing urllib
Code:
htmlpage = urllib.urlopen('http://hackcommunity.com')
htmltext = htmlpage.read()
regex = '<title>(.+?)</title>'
pattern = re.compile(regex)
result = re.findall(pattern, htmltext)
print resultCode:
['HackCommunity.com - ERROR']So I had to use mechanize library which is a "Stateful programmatic web browsing in Python, after Andy Lester’s Perl module", anyway with mechanize library you can make a browser and set the user-agent of that browser! And this way the server will not suspect that you are a bot/script (in theory... read the note at the end).
OK, enough talking, here is the scraper code:
Spoiler: HC HTML Scraper
Code:
from mechanize import Browser
import re
import json
def listToDict(input):
root = {}
lookup = {}
for parent_id, id, name, attr in input:
if parent_id == -1:
root['name'] = name;
lookup[id] = root
else:
node = {'name': name}
lookup[parent_id].setdefault('children', []).append(node)
lookup[id] = node
return root
br = Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0')]
url = "http://hackcommunity.com/index.php"
htmlpage = br.open(url)
htmltext = htmlpage.read()
regex = '<table border="0" cellspacing="0" cellpadding="0" class="tborder">(.+?)\s*</table>'
pattern = re.compile(regex, re.DOTALL)
tables = re.findall(pattern, htmltext)
forum = []
forum.append([-1,0,"HckCommunity", 1000])
z = 1
for i in range(0,len(tables)):
regex = '<div><strong><a href=".*">(.+?)</a></strong><br /><div class="smalltext"></div></div>'
pattern = re.compile(regex)
title = re.findall(pattern, str(tables[i]))
title = filter(None, title)
if (title):
pIndex = z
forum.append([0, pIndex, str(title[0]), i] )(The code need clean up, I will do it today and update)
So ... how does it work?
The following code is the one responsible of downloading the web page, we make a browser br and makes it ignore robot check (robot.txt), and after that we set the user-agent, which in my case is 'User-agent', 'Mozilla/5.0 (X11; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0'
Code:
br = Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0')]
url = "http://hackcommunity.com/index.php"
htmlpage = br.open(url)
htmltext = htmlpage.read()After that we open the web-page (index.php) and read it: htmltext = htmlpage.read()
OK, now we get the page we will need to scrape it, I noticed that MyBB for some reason are presenting the forum (sections and links) using a table, which is for me personally a bad practice, because I would use CSS to style the forum! Anyway...
So, if you check the source of the index.php page you'll find that each section is represented by a table! Which makes it easy to scrape, I first used BeautifulSoup, but things got complicated (I hated it) and I went back to Regex, my favorite method anyway!
Code:
regex = '<table border="0" cellspacing="0" cellpadding="0" class="tborder">(.+?)\s*</table>'
pattern = re.compile(regex, re.DOTALL)
tables = re.findall(pattern, htmltext)So, size of tables now should be 10 and not 9 because the last table/section is the status of the forum... which is not of our interest, so now we have the tables, we loop to get the title and sub-sections of each table/section.
Quote:HeadquartersGeneral
- Forum Announcements and Feedback
- HackCommunity News
- Update log
And so on...
- Introductions
- Contests
- Open Discussion
One think to mention here, is that I didn't go deep into the forum (I didn't scrape the threads of each section), for that I would need to do web-crawling and I wanted to keep things simple (so far).
The code needs more work but I am tired now (I will clean it later):
Code:
forum = []
forum.append([-1,0,"HckCommunity", 1000])
z = 1
for i in range(0,len(tables)):
regex = '<div><strong><a href=".*">(.+?)</a></strong><br /><div class="smalltext"></div></div>'
pattern = re.compile(regex)
title = re.findall(pattern, str(tables[i]))
title = filter(None, title)
if (title):
pIndex = z
forum.append([0, pIndex, str(title[0]), i] )
z += 1
regex = '<td class="trow\d" valign="top">\s*<a href=".*">(.+?)</a>'
pattern = re.compile(regex)
sections = re.findall(pattern, str(tables[i]))
sections = filter(None, sections)
for j in range(0, len(sections)):
forum.append([pIndex, z, str(sections[j]), j])
z += 1As you can see we will store the information in a list forum, this forum we will use later to build our dictionary object, the function "ListToDict" was inspired/found here
The rest of the code is clear I guess, so I will not go through that now, if you have questions please comment and I will try my best to help!
The Interface
Spoiler: The Gui Code
[
Code:
jscript]
<!DOCTYPE html>
<meta charset="utf-8">
<style>
.node circle {
fill: #fff;
stroke: steelblue;
stroke-width: 1.5px;
}
.node {
font: 10px sans-serif;
}
.link {
fill: none;
stroke: #ccc;
stroke-width: 1.5px;
}
</style>
<body>
<script src="http://d3js.org/d3.v3.min.js"></script>
<script>
var diameter = 1024;
var tree = d3.layout.tree()
.size([360, diameter / 2 - 120])
.separation(function(a, b) { return (a.parent == b.parent ? 1 : 2) / a.depth; });
var diagonal = d3.svg.diagonal.radial()
.projection(function(d) { return [d.y, d.x / 180 * Math.PI]; });
var svg = d3.select("body").append("svg")
.attr("width", diameter)
.attr("height", diameter)
.append("g")
.attr("transform", "translate(" + diameter / 2 + "," + diameter / 2 + ")");
d3.json("HCgui.json", function(error, root) {
var nodes = tree.nodes(root),
links = tree.links(nodes);
var link = svg.selectAll(".link")
.data(links)
.enter().append("path")
.attr("class", "link")
.attr("d", diagonal);
var node = svg.selectAll(".node")
.data(nodes)
.enter().append("g")
.attr("class", "node")
.attr("transform", function(d) { return "rotate(" + (d.x - 90) + ")translate(" + d.y + ")"; })
node.append("circle")
.attr("r", 4.5);
node.append("text")
.attr("dy", ".31em")
.attr("text-anchor", function(d) { return d.x < 180 ? "start" : "end"; })
.attr("transform", function(d) { return d.x < 180 ? "translate(8)" : "rotate(180)translate(-8)"; })
.text(function(d) { return d.name; });
});
d3.select(self.frameElement).style("height", diameter - 150 + "px");
</script>And here is the final result:
Next is to scrape the links (href of each section) and make the title click-able (Will redirect the user to the forum)
Conclusion
This is just the beginning of the project, there is a second phase where I should make this GUI functional and acceptable (add animation, background, colors... etc.), so you guys still have a chance to join me, or to join one of my other projects!
This is all for the moment... please leave your comments!
Thanks
Ligeti
[note] @bluedog.tar.gz (the admin) is making some changes in the forum, so if you'll face issues using mechanize (because I did before) you can download the page manually and open it as a file!
![[Image: wvBFmA5.png]](http://i.imgur.com/wvBFmA5.png)
![[+]](https://sinister.ly/images/modern/collapse_collapsed.png)
![[Image: OilyCostlyEwe.gif]](http://fat.gfycat.com/OilyCostlyEwe.gif)
