Parsing very big files around >20 MB each and around 500-700 files
I have a need to get some 4000-20000 strings like 2345577 as list and there are 500 - 700 files which has from 15000-60000 lines have the ids like above (2345577). I need to keep the lines with the ids from the database and delete the lines which does not have the ids. What is the best way to solve this. What I did is that I made a list of ids from db. Then I am looping through each file and then get the id from the line, then I say if the id from the line is contained in the list then add that to stringbuffer otherwise continue.
Is there a better way to do this?