Sinisterly
VB.Net Download full webpage HTML - Printable Version

+- Sinisterly (https://sinister.ly)
+-- Forum: Coding (https://sinister.ly/Forum-Coding)
+--- Forum: Visual Basic & .NET Framework (https://sinister.ly/Forum-Visual-Basic-NET-Framework)
+--- Thread: VB.Net Download full webpage HTML (/Thread-VB-Net-Download-full-webpage-HTML)



VB.Net Download full webpage HTML - Cyb3rNuX - 07-06-2017

You know when you save page html on chrome (or any other browser) and there's field where you select Webpage, complete, like in the pic?

[Image: ZFxRuwc.png]

I want to do same thing in VB.Net.

So far i can only download webpage html using code below:

Code:
Dim dlhtml As New System.Net.WebClient() dlhtml.Encoding = System.Text.Encoding.UTF8 Dim myhtml As String = dlhtml.DownloadString("url")



RE: VB.Net Download full webpage HTML - ClawsMissingBall - 07-06-2017

You will want to scrape all of the relative links and save them. When you get the text look for href=" or src=" in the string. Then from that point find the ending quote and extract the link. If its an absolute link throw it out the window like a 300 lbs hooker. If its relative, download the file and save it with the exact same path.


RE: VB.Net Download full webpage HTML - Cyb3rNuX - 07-06-2017

(07-06-2017, 03:21 PM)ClawsMissingBall Wrote: You will want to scrape all of the relative links and save them.  When you get the text look for href=" or src=" in the string.  Then from that point find the ending quote and extract the link.  If its an absolute link throw it out the window like a 300 lbs hooker.  If its relative, download the file and save it with the exact same path.

Yeah, i already know i should loop through all links and get them, BUT....

On spotify login page, if you save whole document (with chrome browser) it will save one .html file with bunch of code AND line of code i want. If you save just that page, it will get small html file with 2 urls which DOESN'T contain any sources with line of code i want.

How is this possible :|


RE: VB.Net Download full webpage HTML - ClawsMissingBall - 07-06-2017

(07-06-2017, 04:23 PM)Cyb3rNuX Wrote:
(07-06-2017, 03:21 PM)ClawsMissingBall Wrote: You will want to scrape all of the relative links and save them.  When you get the text look for href=" or src=" in the string.  Then from that point find the ending quote and extract the link.  If its an absolute link throw it out the window like a 300 lbs hooker.  If its relative, download the file and save it with the exact same path.

Yeah, i already know i should loop through all links and get them, BUT....

On spotify login page, if you save whole document (with chrome browser) it will save one .html file with bunch of code AND line of code i want. If you save just that page, it will get small html file with 2 urls which DOESN'T contain any sources with line of code i want.

How is this possible :|

Don't understand exactly what you mean. Can you show me a comparison here.


RE: VB.Net Download full webpage HTML - Cyb3rNuX - 07-07-2017

(07-06-2017, 04:45 PM)ClawsMissingBall Wrote:
(07-06-2017, 04:23 PM)Cyb3rNuX Wrote:
(07-06-2017, 03:21 PM)ClawsMissingBall Wrote: You will want to scrape all of the relative links and save them.  When you get the text look for href=" or src=" in the string.  Then from that point find the ending quote and extract the link.  If its an absolute link throw it out the window like a 300 lbs hooker.  If its relative, download the file and save it with the exact same path.

Yeah, i already know i should loop through all links and get them, BUT....

On spotify login page, if you save whole document (with chrome browser) it will save one .html file with bunch of code AND line of code i want. If you save just that page, it will get small html file with 2 urls which DOESN'T contain any sources with line of code i want.

How is this possible :|

Don't understand exactly what you mean.  Can you show me a comparison here.

Go on this url and try to download webpage both ways (html only, complete) and you'll see difference. In my case, with html only it downloads small html file, but in complete document it downloads huge html file that contains state=AUTHTOKENHERE, which i need to get.

I hope you understand it now


RE: VB.Net Download full webpage HTML - titbang - 07-19-2017

In C, using gumbo and libcurl you could do something like this to clone the website. Pardon the code as it is quite sloppy and will need modified for PHP and other resources.

Code:
#include <iostream> #include <cstring> #include <curl/curl.h> #include <vector> #include <sys/types.h> #include <sys/stat.h> #include "gumbo.h" using namespace std; CURL *curl; CURLcode res; int i = 1; string data; vector<string> links; static size_t callback(void *data, size_t size, size_t nmemb, void *pointer) { ( (string*)pointer)->append((char*)data, size * nmemb); return size * nmemb; } string ltrim(string s, const char* t = " \t\n\r\f\v"){ s.erase(0, s.find_first_not_of(t)); return s; } string rtrim(string s, const char* t = " \t\n\r\f\v"){ s.erase(s.find_last_not_of(t) + 1); return s; } string trim(string s, const char* t = " \t\n\r\f\v"){ return ltrim(rtrim(s, t), t); } void extract_links( GumboNode* node ) { GumboAttribute* detail; if (node->type != GUMBO_NODE_ELEMENT) { return; } if( node->v.element.tag == GUMBO_TAG_A && (detail = gumbo_get_attribute(&node->v.element.attributes, "href"))){ links.push_back( trim( detail->value ) ); } GumboVector* children = &node->v.element.children; for (unsigned int i = 0; i < children->length; ++i) { extract_links(static_cast<GumboNode*>(children->data[i])); } } int initialize_curl( const char* url ) { curl_global_init(CURL_GLOBAL_ALL); curl = curl_easy_init(); if(curl) { curl_easy_setopt(curl, CURLOPT_URL, url); curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, callback); curl_easy_setopt(curl, CURLOPT_WRITEDATA, &data); curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"); curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt"); curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt"); res = curl_easy_perform(curl); if(res != CURLE_OK) { curl_easy_strerror(res); return 1; } curl_easy_cleanup(curl); } return 0; } void write_to_html_out( const char* input, const char* filename ) { string core = "/home/user/Downloads"; char buffer[260]; FILE *pFile; if(strstr( filename, "http") == NULL ) { if( strstr( filename, "html") == NULL ) { string tempfile = filename; string outfile = tempfile.substr(0, tempfile.find_last_of("/") ); outfile = core + outfile; mkdir(outfile.c_str(), S_IRWXU | S_IRWXG | S_IROTH | S_IXOTH); } } sprintf( buffer, "%s%s", core.c_str(), filename ); if ( ( pFile = fopen(buffer, "w+") ) != NULL) { fprintf(pFile, "%s", input ); fclose(pFile); } } string replace_all(string str, const string& from, const string& to) { size_t start_pos = 0; while((start_pos = str.find(from, start_pos)) != std::string::npos) { str.replace(start_pos, from.length(), to); start_pos += to.length(); } return str; } void cleanup() { data = ""; } int main (int argc, char *argv[]) { string root_url = "http://www.thedraculasociety.org.uk"; if( initialize_curl( root_url.c_str() ) == 0 ) { GumboOutput* output = gumbo_parse(data.c_str()); extract_links(output->root); write_to_html_out( data.c_str(), "/index.html" ); for(vector<int>::size_type i = 0; i != links.size(); i++) { string url = root_url + links[i].c_str(); cleanup( ); printf("Grabbing link: %s\r\n", links[i].c_str() ); if( initialize_curl( url.c_str() ) == 0 ) { GumboOutput* output = gumbo_parse(data.c_str()); write_to_html_out( data.c_str(), links[i].c_str() ); cleanup( ); } } getchar( ); } return 0; }