CSV splitter (divide large CSV files)
  #1 (permalink)  
Old 02-15-2005, 04:52 PM
gazuz's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: California
Posts: 752
gazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to all
Default CSV splitter (divide large CSV files)

Hello everyone,

Well scorpion provided me with a 2Gb file of tick data of EUR/USD sinc 1998 I believe. I know that MetaTrader and my computer itself cannot support that much memory (I have 1GB of RAM). So I was wondering if there was any way to chop up the file to get different files to be able to import into MetaTrader or at least test them one by one.

Thanks for your time everyone,
Gazuz
__________________
____________________________\/
___________________________( )
______________________o0o___(_)___o0o__
_________________________________
Reply With Quote
  #2 (permalink)  
Old 02-15-2005, 06:28 PM
SJR3t2's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: SLC, UT, USA
Posts: 166
SJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of lightSJR3t2 is a glorious beacon of light
Send a message via ICQ to SJR3t2 Send a message via AIM to SJR3t2 Send a message via MSN to SJR3t2 Send a message via Yahoo to SJR3t2
Default

I know MS VS .NET 2003 can handle files that big and split them. Then you can import them into MT one slice at a time. PS: You may need to use some Find&Replace, as the CSV may not be in the same format as MT. Another oppion is write a program to do it.

Steven
Reply With Quote
  #3 (permalink)  
Old 02-15-2005, 11:32 PM
gazuz's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: California
Posts: 752
gazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to all
Default

Quote:
Originally Posted by SJR3t7
I know MS VS .NET 2003 can handle files that big and split them. Then you can import them into MT one slice at a time. PS: You may need to use some Find&Replace, as the CSV may not be in the same format as MT. Another oppion is write a program to do it.

Steven
Maybe a program already exists, I know file splitters exist but they end up with .01 and .02 files If anyone finds program to divide a .csv file per x lines then that would be awesome to use.

By the way I have 0 experience in Java, VB, C, C++ I just know the basic html and some php but even that most of the time i look up online.

Any help is greatly appreciated
__________________
____________________________\/
___________________________( )
______________________o0o___(_)___o0o__
_________________________________
Reply With Quote
Here's you go
  #4 (permalink)  
Old 02-16-2005, 03:39 AM
scorpion's Avatar
Forex Guru
 
Join Date: Aug 2004
Location: Behind You
Posts: 2,103
scorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to behold
Default Here's you go

I've just written the CSV Splitter for you, as I think it might come in handy someday.

Download: csv-splitter-1.1.zip
Downloaded: 1398 times (Oct 18 2007)

I've attached the compiled exe along with source code. To run the exe, you will need .Net Framework Version 1.1 installed in prior.

Instructions:

1. Fill in necessary information:
CSV file: the path to the CSV that you wanted to split.
Number of lines: the maximum number of lines/rows in each splitted piece.
Max Pieces: limit the number of output files. '0' is unlimited.

2. Click "Split Now!", and that's it. You can find the splitted pieces in the a new folder of the same directory of the CSV file.
gazuz, forexspot, mydm and 12 others like this.
__________________
Make easy pips with the Advanced Economic Calendar for Forex Trading.

Last edited by scorpion; 10-19-2007 at 01:11 AM. Reason: Update to version 1.1
Reply With Quote
  #5 (permalink)  
Old 02-16-2005, 06:24 AM
gazuz's Avatar
Forex Honorary Moderator
 
Join Date: Aug 2004
Location: California
Posts: 752
gazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to allgazuz is a name known to all
Default

Thanks scorpion, I'll try it tomorow when I have time, it's really incredible how quick one can program such a useful tool.
__________________
____________________________\/
___________________________( )
______________________o0o___(_)___o0o__
_________________________________

Last edited by gazuz; 02-16-2005 at 06:36 AM.
Reply With Quote
  #6 (permalink)  
Old 02-16-2005, 08:03 AM
scorpion's Avatar
Forex Guru
 
Join Date: Aug 2004
Location: Behind You
Posts: 2,103
scorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to behold
Default

You're welcome!

Now I've just fixed a few minor bugs in version 1 and added Abort Splitting feature. You can download CSV Splitter version 1.1 from my first post. Enjoy!
patgio likes this.
__________________
Make easy pips with the Advanced Economic Calendar for Forex Trading.
Reply With Quote
Helped!!
  #7 (permalink)  
Old 07-01-2006, 09:25 PM
Currency Trader
 
Join Date: Jul 2006
Posts: 1
HallowShal is on a distinguished road
Default Helped!!

Hey Scorpion i just wanted to tell you thank you for the contribution of the CSV splitter I have been looking for a program like this for along time and while doing a good search found this application in your post. And its exactly what i needed!!! THANKS FOR THE APP!!!

Best,

Jay
Reply With Quote
  #8 (permalink)  
Old 07-01-2006, 11:47 PM
scorpion's Avatar
Forex Guru
 
Join Date: Aug 2004
Location: Behind You
Posts: 2,103
scorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to beholdscorpion is a splendid one to behold
Default

Hey Hallo. You're welcome!

Drop me a line if you have any problem with the app.
__________________
Make easy pips with the Advanced Economic Calendar for Forex Trading.
Reply With Quote
  #9 (permalink)  
Old 06-25-2007, 04:22 AM
Currency Trader
 
Join Date: Jun 2007
Posts: 1
mydm is on a distinguished road
Default

Hi all, I'm new here and couldn't find the Chit Chat forum to say hi , so I figured let me do so here and simultaneously say thank you to scorpion for the cvs splitter file. Very very helpful to me.

So hi all , and thanks scorpion .
Reply With Quote
  #10 (permalink)  
Old 07-16-2007, 08:58 AM
Currency Trader
 
Join Date: Jul 2007
Posts: 1
ashburton is on a distinguished road
Default

Hi,

I just wanted to say thanks for the CSV splitter solution, I may even take the source code and try and work out what's going on - I'd love to be able to write little VB apps like this myself.

Thanks, Ash.
Reply With Quote
Reply

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT. The time now is 07:02 AM.

Registered members gain free access to online FOREX currency trading tools, foreign exchange software, Metatrader MT4/MT5 expert advisors, MT4 indicators and EAs. Register now