Passing around a file between JVMs severly affects writing performance
Abstract: Passing around a file between multiple JVM severly affects writing performance to be a factor of 50 to 100 slower.
- One JVM creates a file, writes to it using FileChannel.write(), .force() and .close(). Filesize is about 300KB.
- The first JVM starts a second JVM which again writes to the file, partlially overwriting it. The file is closed again. The second JVM stops. Filesize is still about 300KB.
- The first JVM starts a third JVM which again opens the file and writes to it, until file size is now about 50MB.
The problem (Windows 7, Java 6u267 & 7u2, all 64 bit) is that the writing in the third JVM gets extremely slow in most cases:
Normal step 3 should take about 2 second, but in >95% of cases it takes over one minute. The process spends most of the time in the write methods:
7.0% 0 + 76 sun.nio.ch.FileChannelImpl.force0 7.1% 0 + 20 sun.nio.ch.FileDispatcher.pwrite0 [..]
Interpreted + native Method 48.1% 0 + 3587 sun.nio.ch.FileDispatcher.pwrite0 39.1% 0 + 2913 sun.nio.ch.FileChannelImpl.force0 [..] Stub + native Method 10.1% 0 + 751 sun.nio.ch.FileDispatcher.pwrite0 [..]
- Introducing a delay of 2 seconds before starting the 2nd and 3rd JVM increases the probablility of a fast run from <5% to about 50%.
- The code is either very slow (60secs) or fast (3sec), never anything in-between.
- During each JVM run, significant parts of the file are overwritten, FileChannel.force(...) is called probably 50-100 times in the third run.
- Locking the file with FileChannel.lock() does not change anything.
- Running the same code in a single JVM is always fast.
- The problem occurs on several machines, using different JVM versions. It does not appear on Linux.
- I'm not using a mapped file, but only direct ByteBuffers and FileChannel methods to write to the file.
- Calling System.gc() or System.runFinalizers() before starting a new JVM does not seem to improve things.
- The code is pure Java SE code, not native libraries, third party libraries or com.* (or similar) are used.
I have a somewhat esoteric theory about this: Java may not release file handles right away, so the next JVM may get a secondary file handle and the JVM or Windows installs some kind of concurrency handler, which slows things down.
Has anyone any idea what could be the problem and how it could be solved?