Skip to main content

MTOM and temporary files

3 replies [Last post]
David Muñoz Guest
Offline
Joined: 2010-12-14

Hello,

When using MTOM in a web service, large attachments are saved in a temporary
file called MIMExxxx.tmp in the OS temp folder.

After the WS consumption, I have noticed that these files are not closed
immediately, but remain open for a while (GC execution?).

Under heavy load circumstances, this is a terrible problem as it can lead
the server to have too many open file descriptors.

Setting @StreamingAttachment.memoryThreshold to -1L leads to not using
temporary files, but this seems only a workaround to me, I would like to
force the temporary files to close.

I wonder if it is possible.

Thanks in advance,

David

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
jitu
Offline
Joined: 2003-06-14

On 12/14/2010 06:02 AM, David Muñoz wrote:
> Hello,
>
> When using MTOM in a web service, large attachments are saved in a
> temporary file called MIMExxxx.tmp in the OS temp folder.
>
> After the WS consumption, I have noticed that these files are not
> closed immediately, but remain open for a while (GC execution?).
>
> Under heavy load circumstances, this is a terrible problem as it can
> lead the server to have too many open file descriptors.
>
> Setting @StreamingAttachment.memoryThreshold to -1L leads to not using
> temporary files, but this seems only a workaround to me, I would like
> to force the temporary files to close.

Are you saying that you need a way to delete those files ? I would
think there are no open file descriptor after the corresponding stream
(DataHandler.getInputStream()) is closed.

Earlier versions of JAX-WS RI used to parse MIME messages in the memory. This used to cause problems for big messages(say big file upload). So the parser keeps the MIME parts> 1MB on the file system. When the MIME part is bound to DataHandler, there is no way to tell the JAX-WS runtime to clean the corresponding file after the use of DataHandler. The DataHandler may be used multiple times by the application, so cannot remove the file.

This usually not a problem since multiple web service invocation calls clear the earlier files used for MIME parts using ReferenceQueue. See org.jvnet.mimepull.WeakDataFile. But it would be a problem(on all platforms) if the application makes only one web service invocation as the runtime will not have a chance to drain reference queue.

For JAX-WS RI, one could configure parsing behaviour(can specify memoryThreshold, temp dir) using StreamingAttachmentFeature or its corresponding annotation StreamingAttachment.

In JAX-WS 2.2.2 onwards, the application can also call close() on the DataHandler provided by the JAX-WS runtime
for e.g:
if (handler instanceof Closeable) {
((Closeable)handler).close(); // removes the MIME part file
}

thanks,
Jitu

>
> I wonder if it is possible.
>
> Thanks in advance,
>
> David

cristiantm
Offline
Joined: 2013-01-17

Hello!

I have a similar problem.

I do have MOTM and streaming working fine. But in my case the request can have many files attached.

What happens is that there are two types of MIME temp files created.

First are those that I understand are mentioned in this topic, and refeer to the attachments, one for each attachment in the request. I close them when I finish using them using the DataHandler, and those files are deleted. No problem here.

But there is also another file, that contains the XML of the request itself (and references to the attachments if using streaming, or the attachments inline if not), that is created and takes a reeeally long time to get closed/deleted. In fact, on my simulations, I find that some files are closed after some time but others are not. The problems is that this is filling up my temp dir, because some clients do not have attachment support and I can´t currently expect them to do so so this file sometimes is really big.

As far as I have investigated, I do not have access to a DataHandler to be closed that refeers to this "index" file. And I could not find any other way to free the file pointer that keeps that file open.

Anyone had the same problem or knows how to get rid of that file?

snajper
Offline
Joined: 2004-10-01

Hi,
there have been some improvements in how the files are handled
recently so I'd recommend to give a latest build try and verify if it
helps your case or not,
MartiNG

David Muñoz Guest 1
Offline
Joined: 2010-12-20

On Fri, Dec 17, 2010 at 2:54 AM, Jitendra Kotamraju <
> wrote:

> On 12/14/2010 06:02 AM, David Muñoz wrote:
>
> Hello,
>
> When using MTOM in a web service, large attachments are saved in a
> temporary file called MIMExxxx.tmp in the OS temp folder.
>
> After the WS consumption, I have noticed that these files are not closed
> immediately, but remain open for a while (GC execution?).
>
>
> Under heavy load circumstances, this is a terrible problem as it can lead
> the server to have too many open file descriptors.
>
> Setting @StreamingAttachment.memoryThreshold to -1L leads to not using
> temporary files, but this seems only a workaround to me, I would like to
> force the temporary files to close.
>
>
> Are you saying that you need a way to delete those files ?
>

No, I'm saying that these files remain open, so a "Too many open files"
exception raises. I'll check the doc you paste, thanks.

> I would think there are no open file descriptor after the corresponding
> stream (DataHandler.getInputStream()) is closed.
>
> Earlier versions of JAX-WS RI used to parse MIME messages in the memory. This used to cause problems for big messages(say big file upload). So the parser keeps the MIME parts > 1MB on the file system. When the MIME part is bound to DataHandler, there is no way to tell the JAX-WS runtime to clean the corresponding file after the use of DataHandler. The DataHandler may be used multiple times by the application, so cannot remove the file.
>
> This usually not a problem since multiple web service invocation calls clear the earlier files used for MIME parts using ReferenceQueue. See org.jvnet.mimepull.WeakDataFile. But it would be a problem(on all platforms) if the application makes only one web service invocation as the runtime will not have a chance to drain reference queue.
>
> For JAX-WS RI, one could configure parsing behaviour(can specify memoryThreshold, temp dir) using StreamingAttachmentFeature or its corresponding annotation StreamingAttachment.
>
> In JAX-WS 2.2.2 onwards, the application can also call close() on the DataHandler provided by the JAX-WS runtime
> for e.g:
> if (handler instanceof Closeable) {
> ((Closeable)handler).close(); // removes the MIME part file
> }
>
> thanks,
> Jitu
>
>
> I wonder if it is possible.
>
> Thanks in advance,
>
> David
>
>
>