Sorting large data sets
I'm working on something which needs to able to sort variable amounts of records. As I'm reading this data from a file, I'm looking at how using a TreeMap would perform, inserting each row as it's read - then iterating through the resultant map to write out the results.
My concern is that as I scale up the number of records - the performance will degrade as memory runs short. To try and get round this I'm planning on splitting the process, building maps to a certain point and then write it out to a temporary file. Repeating this for as many times as needed and then merging these resultant files into a final sorted result.
Can anyone see any problems with this or can anyone come up with alternatives.