Forex Forum > Strategies Zone > Tools / Utilities >

CSV splitter (divide large CSV files)

Forum

Like Tree16Likes

 
LinkBack Thread Tools Search this Thread Display Modes

CSV splitter (divide large CSV files)
  #1 (permalink)  
Old 02-15-2005, 04:52 PM
gazuz's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: California
Posts: 752
gazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to all
Hello everyone,

Well scorpion provided me with a 2Gb file of tick data of EUR/USD sinc 1998 I believe. I know that MetaTrader and my computer itself cannot support that much memory (I have 1GB of RAM). So I was wondering if there was any way to chop up the file to get different files to be able to import into MetaTrader or at least test them one by one.

Thanks for your time everyone,
Gazuz
__________________
____________________________\¦/
___________________________(ò ó)
______________________o0o___(_)___o0o__
___¦_____¦_____¦_____¦_____¦_____¦_____¦
Quote
  #2 (permalink)  
Old 02-15-2005, 06:28 PM
SJR3t2's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: SLC, UT, USA
Posts: 166
SJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of light
Send a message via ICQ to SJR3t2 Send a message via AIM to SJR3t2 Send a message via MSN to SJR3t2 Send a message via Yahoo to SJR3t2
I know MS VS .NET 2003 can handle files that big and split them. Then you can import them into MT one slice at a time. PS: You may need to use some Find&Replace, as the CSV may not be in the same format as MT. Another oppion is write a program to do it.

Steven
Quote
  #3 (permalink)  
Old 02-15-2005, 11:32 PM
gazuz's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: California
Posts: 752
gazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to all
Quote:
Originally Posted by SJR3t7
I know MS VS .NET 2003 can handle files that big and split them. Then you can import them into MT one slice at a time. PS: You may need to use some Find&Replace, as the CSV may not be in the same format as MT. Another oppion is write a program to do it.

Steven
Maybe a program already exists, I know file splitters exist but they end up with .01 and .02 files If anyone finds program to divide a .csv file per x lines then that would be awesome to use.

By the way I have 0 experience in Java, VB, C, C++ I just know the basic html and some php but even that most of the time i look up online.

Any help is greatly appreciated
__________________
____________________________\¦/
___________________________(ò ó)
______________________o0o___(_)___o0o__
___¦_____¦_____¦_____¦_____¦_____¦_____¦
Quote
Here's you go
  #4 (permalink)  
Old 02-16-2005, 03:39 AM
scorpion's Avatar
Forex Guru
 
Join Date: Aug 2004
Location: Behind You
Posts: 2,104
scorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to behold
I've just written the CSV Splitter for you, as I think it might come in handy someday.

Download: csv-splitter-1.1.zip
Downloaded: 1398 times (Oct 18 2007)

I've attached the compiled exe along with source code. To run the exe, you will need .Net Framework Version 1.1 installed in prior.

Instructions:

1. Fill in necessary information:
CSV file: the path to the CSV that you wanted to split.
Number of lines: the maximum number of lines/rows in each splitted piece.
Max Pieces: limit the number of output files. '0' is unlimited.

2. Click "Split Now!", and that's it. You can find the splitted pieces in the a new folder of the same directory of the CSV file.
gazuz, forexspot, mydm and 12 others like this.
__________________
Make easy pips with the Advanced Economic Calendar for Forex Trading.

Last edited by scorpion; 10-19-2007 at 02:11 AM. Reason: Update to version 1.1
Quote
  #5 (permalink)  
Old 02-16-2005, 06:24 AM
gazuz's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: California
Posts: 752
gazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to all
Thanks scorpion, I'll try it tomorow when I have time, it's really incredible how quick one can program such a useful tool.
__________________
____________________________\¦/
___________________________(ò ó)
______________________o0o___(_)___o0o__
___¦_____¦_____¦_____¦_____¦_____¦_____¦

Last edited by gazuz; 02-16-2005 at 06:36 AM.
Quote
  #6 (permalink)  
Old 02-16-2005, 08:03 AM
scorpion's Avatar
Forex Guru
 
Join Date: Aug 2004
Location: Behind You
Posts: 2,104
scorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to behold
You're welcome!

Now I've just fixed a few minor bugs in version 1 and added Abort Splitting feature. You can download CSV Splitter version 1.1 from my first post. Enjoy!
patgio likes this.
__________________
Make easy pips with the Advanced Economic Calendar for Forex Trading.
Quote
Helped!!
  #7 (permalink)  
Old 07-01-2006, 10:25 PM
Currency Trader
 
Join Date: Jul 2006
Posts: 1
HallowShal is on a distinguished road
Hey Scorpion i just wanted to tell you thank you for the contribution of the CSV splitter I have been looking for a program like this for along time and while doing a good search found this application in your post. And its exactly what i needed!!! THANKS FOR THE APP!!!

Best,

Jay
Quote
  #8 (permalink)  
Old 07-02-2006, 12:47 AM
scorpion's Avatar
Forex Guru
 
Join Date: Aug 2004
Location: Behind You
Posts: 2,104
scorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to behold
Hey Hallo. You're welcome!

Drop me a line if you have any problem with the app.
__________________
Make easy pips with the Advanced Economic Calendar for Forex Trading.
Quote
  #9 (permalink)  
Old 06-25-2007, 05:22 AM
Currency Trader
 
Join Date: Jun 2007
Posts: 1
mydm is on a distinguished road
Hi all, I'm new here and couldn't find the Chit Chat forum to say hi , so I figured let me do so here and simultaneously say thank you to scorpion for the cvs splitter file. Very very helpful to me.

So hi all , and thanks scorpion .
Quote
  #10 (permalink)  
Old 07-16-2007, 09:58 AM
Currency Trader
 
Join Date: Jul 2007
Posts: 1
ashburton is on a distinguished road
Hi,

I just wanted to say thanks for the CSV splitter solution, I may even take the source code and try and work out what's going on - I'd love to be able to write little VB apps like this myself.

Thanks, Ash.
Quote