CSV splitter (divide large CSV files)

[gazuz]

Hello everyone,

Well scorpion provided me with a 2Gb file of tick data of EUR/USD sinc 1998 I believe. I know that MetaTrader and my computer itself cannot support that much memory (I have 1GB of RAM). So I was wondering if there was any way to chop up the file to get different files to be able to import into MetaTrader or at least test them one by one.

Thanks for your time everyone,
Gazuz

[SJR3t2]

I know MS VS .NET 2003 can handle files that big and split them. Then you can import them into MT one slice at a time. PS: You may need to use some Find&Replace, as the CSV may not be in the same format as MT. Another oppion is write a program to do it.

Steven

[gazuz]

Originally Posted by SJR3t7

I know MS VS .NET 2003 can handle files that big and split them. Then you can import them into MT one slice at a time. PS: You may need to use some Find&Replace, as the CSV may not be in the same format as MT. Another oppion is write a program to do it.

Steven

Maybe a program already exists, I know file splitters exist but they end up with .01 and .02 files If anyone finds program to divide a .csv file per x lines then that would be awesome to use.

By the way I have 0 experience in Java, VB, C, C++ I just know the basic html and some php but even that most of the time i look up online.

Any help is greatly appreciated

[scorpion]

I've just written the CSV Splitter for you, as I think it might come in handy someday.

Download: csv-splitter-1.1.zip
Downloaded: 1398 times (Oct 18 2007)

I've attached the compiled exe along with source code. To run the exe, you will need .Net Framework Version 1.1 installed in prior.

Instructions:

1. Fill in necessary information:
CSV file: the path to the CSV that you wanted to split.
Number of lines: the maximum number of lines/rows in each splitted piece.
Max Pieces: limit the number of output files. '0' is unlimited.

2. Click "Split Now!", and that's it. You can find the splitted pieces in the a new folder of the same directory of the CSV file.

[gazuz]

Thanks scorpion, I'll try it tomorow when I have time, it's really incredible how quick one can program such a useful tool.

[scorpion]

You're welcome!

Now I've just fixed a few minor bugs in version 1 and added Abort Splitting feature. You can download CSV Splitter version 1.1 from my first post. Enjoy!

[HallowShal]

Hey Scorpion i just wanted to tell you thank you for the contribution of the CSV splitter I have been looking for a program like this for along time and while doing a good search found this application in your post. And its exactly what i needed!!! THANKS FOR THE APP!!!

Best,

Jay

[scorpion]

Hey Hallo. You're welcome!

Drop me a line if you have any problem with the app.

[mydm]

Hi all, I'm new here and couldn't find the Chit Chat forum to say hi , so I figured let me do so here and simultaneously say thank you to scorpion for the cvs splitter file. Very very helpful to me.

So hi all , and thanks scorpion .

[ashburton]

Hi,

I just wanted to say thanks for the CSV splitter solution, I may even take the source code and try and work out what's going on - I'd love to be able to write little VB apps like this myself.

Thanks, Ash.