Skip to main content

How to Manage a Large Marshalling?

7 replies [Last post]
eguy66
Offline
Joined: 2004-05-19

I am concerned that my present design using JAXB to marshall java objects into an xml document may strain system resources, given the right conditions. In this application, the xml schema has a large, repeating complex element, representing data used to generate a pdf document. Our initial estimates assume a specific run may contain information for as many as 2500 documents, with a resulting marshalled xml file of about 62Mb. I'm not sure what the jvm memory requirement is to store all those java objects before marhalling, but I have the feeling that if several such concurrent requests are running, it could exhaust memory.
What I'm wondering is, will I need to re-design in such a way that I marshall just the shell of the document to a file first, then marshall each instance of the repeating element and use file i/o to insert the fragment into the file? Or, are there features of JAXB that I'm not aware of that would that would allow me to do something like "start marshall", "marshall component", "marshall component" ... "end marshall"?
Any thoughts?
tia,
Ed

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
kohsuke
Offline
Joined: 2003-06-09

The memory required for marshalling a Java object tree out is a constant overhead regardless of the size of the tree. So if you can keep the tree in the memory in the first place, you don't need to worry about the marshalling aspect of it.

What you should be concerned about is that when you have multiple threads each working with a large Java object tree, it might exhaust the memory. If that is the case, you might have to process XML chunk by chunk.

Using StAX would make this "processing little-by-little" very easy.

ericf2009
Offline
Joined: 2006-06-21

Hello Kohsuke,
I know this an old thread, but it is exactly on my subject, marshalling a really large XML file. I investigated using StAX, but ended up trying to combine StAX with JAXB. I want to use Queue's to pipe my data from my parser to my mapper and ultimately to XML using JAXB marshalling. I prefer JAXB marshalling to StAX because it does whole objects at once w/o my having to deal with each element and attribute. I populate objects in my mapper, with their structure generated by JAXB from my Schema. But JAXB generates List<>'s, not Queue<>'s. Is there a way to direct JAXB to use the class I want, ArrayBlockingQueue<> instead of List<> when processing complexType:sequences?

Thanks for any help or redirection.

- Eric

Message was edited by: ericf2009

Message was edited by: ericf2009

kohsuke
Offline
Joined: 2003-06-09

You can use JAXB's customization for that.

I'm sure one of the samples bundled in the RI shows you how to do it.

ericf2009
Offline
Joined: 2006-06-21

Thanks for the pointer to the examples. However, I didn't find the jxb:globalBindings collectionType setting to be of any help. It only set the instantiation type, not the type of the sequence itself, which remained List. It wouldn't compile when I set collectionType to ArrayBlockingQueue because that class requires a queue size int in its constructor, and jaxb wrote only a no-arg constructor. Even if I got past that somehow, ArrayBlockingQueue doesn't implement List, so it wouldn't work anyway.

I've worked around this by just creating a wrapper class around my jaxb-generated class, holding the Queues for what I need. So I'm ok for now. But it seems with so much customization there must be a way to make jaxb generate something other than List for a sequence.

Anyway, thanks for answering! Nice to know you're there.

- Eric

kohsuke
Offline
Joined: 2003-06-09

I didn't realize that JAXB required a List implementation. Some of the details have already escaped me, but it seems like we should be just fine requiring collection.

Would you like to file an RFE for that? If you'd be even interested in looking at the code, that would be even more great!

ericf2009
Offline
Joined: 2006-06-21

Thank you! I'm new to this community so don't know yet how to submit an RFE, but will find out. Also, you might have had a good reason for choosing List, such as XML element order usually being significant. I'll study more before making any suggestions. And I'd love to see the code, but not right now as I'm behind a delivery schedule.

Regards,
- Eric

Message was edited by: ericf2009

kohsuke
Offline
Joined: 2003-06-09

You can file an RFE from http://jaxb.dev.java.net/issues/

Yes, by default we want to use a List, but when users tell it to use something else, like Set, I don't see any reason why JAXB should reject that.