Skip to main content

Best method for reading and writing files

9 replies [Last post]
byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

I have an application that maintains a file consisting of a fixed size header and variable size data section with a FileChannel and MappedByteBuffer. To read and write to the header I maintain a header byte buffer via:

FileChannel ioc = new RandomAccessFile(file, "rw").getChannel();
MappedByteBuffer ioh = ioc.map(FileChannel.MapMode.READ_WRITE, 0, HEADER_SIZE);

The header holds a mximum of 8192 entries, each of which consists of the offset and length a data chunk, which can be up to 64K in size. To read and write the header I use:

ioh.getLong(position) and ioh.putLong(position, new_value).

To access the data I use the file channel's read and write:

ioc.read(output_buffer, ioh.getLong(position));

and

ioc.position(begin);
numBytes = ioc.write(dataBuffer);

I wish to have the best possible access performance for both the header (which indicates if a particular chunk is available), and the data. Can anyone say whether my choice of methods for reading and writing is the best.

John

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
briand
Offline
Joined: 2005-07-11
Points: 0

I think that much of the decision depends on how large the data
sections are (and, similarly, how large the individual I/O operations
are), and what you subsequently do with that data.

One advantage of MappedByteBuffer that you are giving up by
using read() and write() is the zero-copy advantage that they provide.
To make use of this capability, map the whole file into memory and
carve out view buffers (ByteBuffer.slice() and ByteBuffer.dupicate())
into the original MappedByteBuffer. View objects are inexpensive to
create and collect, which is something that cannot be said about
creating direct or mapped ByteBuffer objects. Then, use the get/put
operations to modify the data directly in the MappedByteBuffer.

There are potential consequences of this design. The get/put operations
are tedious to use and if you need too many of them to modify a view,
then the overhead of all those calls may outweigh other approaches.
Creating String objects from characters in a ByteBuffer is also a bit more
expensive, as there's a lot of copying involved. Also, if the mapped file
is rather large in size, then mapping the whole file into memory might
be problematic - you only have so much virtual address space in the
process and the java heap, the mapped file, and other components
of the JVM and application will compete for that space, forcing you to
make choices you might not want or be able to make. You might be
able to manage this by selectively mapping in sections of the file when
needed instead of mapping in the whole file, but the frequency of such
mapping operations better be low, as mapping has creation and more
importantly collection (and collection related address space) performance
implications. Furthermore, if you use get/put to operated on the view
buffers, you'd likely need to make sure that only one thread has a
reference to one section of the file, or provide some other form of
synchronization to that section of the file. Finally, when you want the in
memory changes to be synced with the disk, call FileChannel.force()
(though on most modern OS's, the OS itself is maintaining this
synchronization transparently, but the Java spec doesn't guarantee
this behavior, so you shouldn't rely on that side effect).

One alternative to the read and write operations you are currently
using is to use transferFrom/transferTo. For larger buffer sizes, these
methods use block copy routines that generally perform better, but
there's a trade-off relative to the buffer size for which these methods
are advantageous. You'll have to determine that by experimentation,
preferably with your application rather than some test program that
ends up turning in to a bad microbenchmark.

Brian

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

> I think that much of the decision depends on how large the data
> sections are (and, similarly, how large the individual I/O operations
> are), and what you subsequently do with that data.
The sections hold images, that I either read from the file or write to the file.

> One advantage of MappedByteBuffer that you are giving up by
> using read() and write() is the zero-copy advantage that they provide.
> To make use of this capability, map the whole file into memory and
> carve out view buffers (ByteBuffer.slice() and ByteBuffer.dupicate())
> into the original MappedByteBuffer. View objects are
> inexpensive to create and collect, which is something that cannot be
> said about creating direct or mapped ByteBuffer objects. Then,
> use the get/put operations to modify the data directly in the MappedByteBuffer.
If I have the file fully populated It will be about 500MB so I don't know if this is practical.

> You might be able to manage this by selectively mapping in
> sections of the file when needed instead of mapping in the whole file, but the
> frequency of such mapping operations better be low, as mapping has
> creation and more importantly collection (and collection related
> address space) performance implications. Furthermore, if you use get/put to
> operated on the view buffers, you'd likely need to make sure that only one
> thread has a reference to one section of the file, or provide some
> other form of synchronization to that section of the file.
I was thinking of this reading your reponse, such that depending on the particular
chunk request I would map into memory a certain number of chunks so that any future
requests would use the memory ones first.

> One alternative to the read and write operations you are currently
> using is to use transferFrom/transferTo. For larger buffer sizes, these
> methods use block copy routines that generally perform better, but
> there's a trade-off relative to the buffer size for which these methods
> are advantageous. You'll have to determine that by experimentation,
> preferably with your application rather than some test program that
> ends up turning in to a bad microbenchmark.
When I look at the javadocs for transferFrom/transferTo it seem that these
require a ByteChannel which is linked to a file. If I am reading from the
file and placing in a ByteBuffer and writing from the ByteBuffer to the file
how can I associate a ByteChannel with the ByteBuffer.

John

briand
Offline
Joined: 2005-07-11
Points: 0

> > One advantage of MappedByteBuffer that you aregiving up by
> > using read() and write() is the zero-copy advantage that they provide.
> > To make use of this capability, map the whole file into memory and
> > carve out view buffers (ByteBuffer.slice() and ByteBuffer.dupicate())
> > into the original MappedByteBuffer. View objects are
> > inexpensive to create and collect, which is something that cannot be
> > said about creating direct or mapped ByteBuffer objects. Then,
> > use the get/put operations to modify the data directly in the MappedByteBuffer.
> If I have the file fully populated It will be about 500MB so I don't know if this is practical.

It depends on what OS you are running on. With an OS that supports a
4GB virtual address space per process, 500MB might not be too bad.
But it depends on the java heap size your app needs, and potentially
other aspects of the application.

> > You might be able to manage this by selectively mapping in
> > sections of the file when needed instead of mapping in the whole file, but the
> > frequency of such mapping operations better be low, as mapping has
> > creation and more importantly collection (and collection related
> > address space) performance implications. Furthermore, if you use get/put to
> > operated on the view buffers, you'd likely need to make sure that only one
> > thread has a reference to one section of the file, or provide some
> > other form of synchronization to that section of the file.
> I was thinking of this reading your reponse, such that depending on the particular
> chunk request I would map into memory a certain number of chunks so that any future
> requests would use the memory ones first.

I'm not quite sure what you are proposing here, but sounds a bit like
a cache. You should be able to implement a cache like data structure
using java.lang.ref Reference objects.

>
> > One alternative to the read and write operations you are currently
> > using is to use transferFrom/transferTo. For larger buffer sizes, these
> > methods use block copy routines that generally perform better, but
> > there's a trade-off relative to the buffer size for which these methods
> > are advantageous. You'll have to determine that by experimentation,
> > preferably with your application rather than some test program that
> > ends up turning in to a bad microbenchmark.
> When I look at the javadocs for transferFrom/transferTo it seem that these
> require a ByteChannel which is linked to a file. If I am reading from the
> file and placing in a ByteBuffer and writing from the ByteBuffer to the file
> how can I associate a ByteChannel with the ByteBuffer.

My mistake - these are channel to channel operations, not ByteBuffer to
channel operations. Sorry about that.

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

> It depends on what OS you are running on. With an OS that supports a
> 4GB virtual address space per process, 500MB might not be too bad.
> But it depends on the java heap size your app needs, and potentially
> other aspects of the application.
I have a number of these files open at any point of the application,
each file being between 10's to 100's of mega bytes. Thus I don't want to
overload the base memory requirement of my application. I want to
have a fixed overhead for memory associated with the file reading/writing.

> I'm not quite sure what you are proposing here, but sounds a bit like
> a cache. You should be able to implement a cache like data structure
> using java.lang.ref Reference objects.
Actually I was thinking of a buffered reader where chunk requests load into memory
the next n chunks so that future requests are probably already in memory.
It would probably be a mapping into a bytebuffer with a size of n chunks
the file. Then if the next request fell with the data already in memory it would be
fast. If not then the area mapped is changed to the new position
and mapping applied again for n chunks.

It would be nice if I could create a mapping of the whole space and then
slice it up into smaller spaces which only get loaded if I request it, then
I cache these in a LRU cache to fix the memory overhead.

You mentioned using java.lang.ref Reference objects. How would you use these to
create a cache?

John

briand
Offline
Joined: 2005-07-11
Points: 0

> > It depends on what OS you are running on. With an OS that supports a
> > 4GB virtual address space per process, 500MB might not be too bad.
> > But it depends on the java heap size your app needs, and potentially
> > other aspects of the application.
> I have a number of these files open at any point of the application,
> each file being between 10's to 100's of mega bytes. Thus I don't want to
> overload the base memory requirement of my application. I want to
> have a fixed overhead for memory associated with the file reading/writing.

That could certainly be problematic. A 64-bit process on a system with
a large amount of RAM might help, but that has other potential issues
(cost of the RAM, reduced performance in a 64-bit address space, for
example).

>
> > I'm not quite sure what you are proposing here, but sounds a bit like
> > a cache. You should be able to implement a cache like data structure
> > using java.lang.ref Reference objects.
> Actually I was thinking of a buffered reader where chunk requests load into memory
> the next n chunks so that future requests are probably already in memory.
> It would probably be a mapping into a bytebuffer with a size of n chunks
> he file. Then if the next request fell with the data already in memory it would be
> fast. If not then the area mapped is changed to the new position
> and mapping applied again for n chunks.

You could take that approach, but I think you'd be likely to get into the
scenario where you are creating and collecting many mapped byte buffer
objects. The creation overhead is costly, and the collection overhead is
also high. The biggest issue in this scenario is the latency from the point in
time that the mapped byte buffer becomes unreferenceable to the time
that the underling mapped file region is actually released (thus freeing
that virtual address space region for other uses). The longer this period
of time, the more virtual address space pressure the process experiences
and the higher the probability of an OOM error when creating a new
mapped byte buffer.

>
> It would be nice if I could create a mapping of the whole space and then
> slice it up into smaller spaces which only get loaded if I request it, then
> I cache these in a LRU cache to fix the memory overhead.

That's actually what does happen. Real memory isn't allocated for the
mapped byte buffer object until a page in the file is actually touched
(read or written). Mapped byte buffers use mmap() on Solaris and Linux,
and the actual behavior is subject to OS policies. For example, it's possible
the OS may start prefetching some pages of the file in anticipation of
future use. Still, those pages (loaded into the file system cache) won't be
charged against your process real memory usage until the page is actually
touched and the virtual memory resources are allocated.

The real issue here is the virtual address space consumption that happens
when the file is mapped. The virtual address space is reserved when the
mapping operation occurs. With as many large files as you have, you'll consume
the virtual address space of a 32-bit process rather quickly, even though your
Resident Set Size (Working Set Size in windows) doesn't necessarily increase
much.

>
> You mentioned using java.lang.ref Reference objects.
> How would you use these to create a cache?
>

Most likely using SoftReference objects:

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ref/SoftReference.html

Note that using reference objects won't really solve the latency issue I
mentioned above, as the mapped byte buffer objects will still consume
virtual address space until the garbage collector discovers the softly
reachable object and then arranges to have it freed and collected.
You'll have to do some google searches to learn how to use SoftReference
object for implementing a cache.

The latency issue is the most problematic issue when using mapped
byte buffer objects, particularly when you are mapping and releasing
many of these objects. If you just map one such object, then this issue
is far less severe. That's one reason why we suggest that mapped byte
buffer objects are really more for long lived file mappings, not short lived
mappings.

Brian

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

> > It would be nice if I could create a mapping of the whole space and then
> > slice it up into smaller spaces which only get loaded if I request it, then
> > I cache these in a LRU cache to fix the memory overhead.
> That's actually what does happen. Real memory isn't allocated for the
> mapped byte buffer object until a page in the file is actually touched
> (read or written). Mapped byte buffers use mmap() on Solaris and Linux,
> and the actual behavior is subject to OS policies.
> For example, it's possible the OS may start prefetching some pages of the file
> in anticipation of future use. Still, those pages (loaded into the file
> system cache) won't be charged against your process real memory usage until
> the page is actually touched and the virtual memory resources are
> allocated.
>
> The real issue here is the virtual address space consumption that happens
> when the file is mapped. The virtual address space is reserved when the
> mapping operation occurs. With as many large files as you have, you'll consume
> the virtual address space of a 32-bit process rather quickly, even though your
> Resident Set Size (Working Set Size in windows) doesn't necessarily increase
> much.

So am I reading right when I say that if I'm gonna use a mapping of the full file (500MB say),
then slice it up into smaller buffers which come and go, I only have to worry about how many
full file mappings exists, each of which will eat up the address space, as these base mappings
will exist for the duration of the application.

Or if I'm gonna use a number of smaller mappings for blocks of data, that I will create when
necessary, then I will have to worry about the latency between when I no longer use them,
nulling there reference say, and when the address space is freed (which I have no control over).

John

briand
Offline
Joined: 2005-07-11
Points: 0

> So am I reading right when I say that if I'm gonna use a mapping of the full file (500MB say),
> then slice it up into smaller buffers which come and go, I only have to worry about how many
> full file mappings exists, each of which will eat up the address space, as these base mappings
> will exist for the duration of the application.

Yes, for the most part. You still have to worry about your java heap size and
any gc pressure that might result from the creation of the view buffers, but this
is far less worrying than the type of issues around using a mapped byte buffer.

Note that the view buffers contain a reference to their 'parent' mapped byte
buffer object and an offset into that buffer and a capacity (along with their
own position, limit, and, mark). Since they contain a reference to their 'parent'
buffer, the parent buffer will never become unreferenceable before it does,
so there's no cleanup needed when a view buffer becomes unreferenced.
So, only the original mapped byte buffer needs processing when it becomes
unreferenced. This greatly reduces the costs associated with view buffers
relative to mapped (or even direct) byte buffers.

>
> Or if I'm gonna use a number of smaller mappings for blocks of data, that I will create when
> necessary, then I will have to worry about the latency between when I no longer use them,
> nulling there reference say, and when the address space is freed (which I have no control over).

Yes, but only if you create and release them frequently. You won't really need to
null the reference, as long as any referring object becomes unreferenced it will
get collected, but it probably wouldn't hurt to do so.

The term 'frequently' above is generally relative to the frequency of garbage
collection. However, if such an object gets promoted to the old generation, then
the infrequency of full gc events could compound this issue. If you run out of
address space due to many file mappings before you run out of space in the
old generation of the heap, this virtual address space problem is much more
likely to occur; i.e. the latency from unreferenced to cleaned up is tied to the
frequency of full garbage collections. If the objects die before escaping the
young generation, then the cleanup operations are likely to occur with less
latency, as minor gc events are generally far more frequent than full gc events.
The problem here is that tuning the heap to assure that the objects die in the
young generation is difficult, and quite possibly impossible. So, you have to
assume worst case conditions.

Using the CMS collector can help with reducing the latency for such objects
that get promoted to the old generation, but it will still be tied to the frequency
of a CMS cycle, which is likely to be longer than the minor gc frequency, but
shorter than you'd get with full gc's from the serial or parallel old gen collectors..

As you can see, using these direct and mapped byte buffer interfaces require
a bit more understanding of the whole picture in order to avoid misuse.

Brian

byhisdeeds
Offline
Joined: 2006-01-06
Points: 0

Thanks for the above discussion. I'm gonna think a little and try each approach. I have a working system with the channel read and write so it should be fairly easy to test a new approach.

John

3dfan
Offline
Joined: 2009-05-21
Points: 0

Yeah thanks a lot for describing such method! Very useful knowledge for graphics designer!