chevron_left chevron_right
Login Register invert_colors photo_library


Stay updated and chat with others! - Join the Discord!
Thread Rating:
  • 0 Vote(s) - 0 Average


Split large text files into smaller files filter_list
Author
Message
Split large text files into smaller files #1
I made this script because I have some very large wordlists, one with over 14 million lines. So I wanted to split this into smaller files.

Syntax
Code:
python splitfile.py [file] [chunk size]

Example
Code:
python splitfile.py rockyou.txt 500000
This will generate files with 500000 lines in each

Code:
#!/usr/bin/python

import sys
import math
import re

def makeFilename(number, chunks):
    length = str(len(str(chunks)))
    format = 'output-%0' + length + 'd'
    return format % (number)

if len(sys.argv) != 3:
    sys.exit('Syntax error: ./splitfile.py <filename> [lines per file]')
else:
    filename = sys.argv[1]
    if not re.match('^\d+$', sys.argv[2]):
        sys.exit('Chunk size must be a number')
    try:
        with open(filename) as f:
            lines = f.readlines()
        total = len(lines)
            chunk_size = int(sys.argv[2])
        chunks = (total / chunk_size) + 1
            
            i = 0
            j = 1
            fout = open(makeFilename(j, chunks), 'wb')
            print 'Writing file #' + str(j) + ' of ' + str(chunks)
            for line in lines:
                fout.write(line)
                if i % chunk_size == 0:
                    fout.close()
                    fout = open(makeFilename(j, chunks), 'wb')
                    print 'Writing file #' + str(j) + ' of ' + str(chunks)
                    j += 1
                i += 1
            fout.close()


    except IOError:
        sys.exit('IOError: Unable to read file')

Like always, any feedback and suggestions for improvement is very much appreciated Smile
"SQL Injection-a-holic"

Twitter | Security Sucks | My Blog

Reply

Split large text files into smaller files #2
I made this script because I have some very large wordlists, one with over 14 million lines. So I wanted to split this into smaller files.

Syntax
Code:
python splitfile.py [file] [chunk size]

Example
Code:
python splitfile.py rockyou.txt 500000
This will generate files with 500000 lines in each

Code:
#!/usr/bin/python

import sys
import math
import re

def makeFilename(number, chunks):
    length = str(len(str(chunks)))
    format = 'output-%0' + length + 'd'
    return format % (number)

if len(sys.argv) != 3:
    sys.exit('Syntax error: ./splitfile.py <filename> [lines per file]')
else:
    filename = sys.argv[1]
    if not re.match('^\d+$', sys.argv[2]):
        sys.exit('Chunk size must be a number')
    try:
        with open(filename) as f:
            lines = f.readlines()
        total = len(lines)
            chunk_size = int(sys.argv[2])
        chunks = (total / chunk_size) + 1
            
            i = 0
            j = 1
            fout = open(makeFilename(j, chunks), 'wb')
            print 'Writing file #' + str(j) + ' of ' + str(chunks)
            for line in lines:
                fout.write(line)
                if i % chunk_size == 0:
                    fout.close()
                    fout = open(makeFilename(j, chunks), 'wb')
                    print 'Writing file #' + str(j) + ' of ' + str(chunks)
                    j += 1
                i += 1
            fout.close()


    except IOError:
        sys.exit('IOError: Unable to read file')

Like always, any feedback and suggestions for improvement is very much appreciated Smile
"SQL Injection-a-holic"

Twitter | Security Sucks | My Blog

Reply

RE: Split large text files into smaller files #3
You could have used itertools http://docs.python.org/2/library/itertools.html#recipes It makes your work a little bit easier Smile
My Blog: http://www.procurity.wordpress.com
Donations: 1HLjiSbnWMpeQU46eUVCrYdbkrtduX7snG

Reply

RE: Split large text files into smaller files #4
You could have used itertools http://docs.python.org/2/library/itertools.html#recipes It makes your work a little bit easier Smile
My Blog: http://www.procurity.wordpress.com
Donations: 1HLjiSbnWMpeQU46eUVCrYdbkrtduX7snG

Reply

RE: Split large text files into smaller files #5
Thanks @Ex094 Smile I'll have look into that
"SQL Injection-a-holic"

Twitter | Security Sucks | My Blog

Reply

RE: Split large text files into smaller files #6
Thanks @Ex094 Smile I'll have look into that
"SQL Injection-a-holic"

Twitter | Security Sucks | My Blog

Reply

RE: Split large text files into smaller files #7
Nice dude Biggrin
I will add this to index.

Reply

RE: Split large text files into smaller files #8
Nice dude Biggrin
I will add this to index.

Reply

RE: Split large text files into smaller files #9
(05-17-2013, 05:32 PM)MrGeek Wrote: Nice dude Biggrin
I will add this to index.

Thanks @MrGeek Smile I'm feel honored being on that list Cool
"SQL Injection-a-holic"

Twitter | Security Sucks | My Blog

Reply

RE: Split large text files into smaller files #10
(05-17-2013, 05:32 PM)MrGeek Wrote: Nice dude Biggrin
I will add this to index.

Thanks @MrGeek Smile I'm feel honored being on that list Cool
"SQL Injection-a-holic"

Twitter | Security Sucks | My Blog

Reply






Users browsing this thread: 1 Guest(s)