Skip to main content

Memory Bloat ! & Fix !!

11 replies [Last post]
nvaidya
Offline
Joined: 2004-08-03

Hope I'm tracking the code correctly...

From what I see, it appears that even when an IndexedGeometryArray has the BY_REFERENCE and USE_COORD_INDEX_ONLY flags set, Java3D is creating needlessly individual copies of the indices array for each of the vertex fields - coords, colors, normals, and for each of the texCoords in the case of multi-texturing.

I've identified the problem to be coming from
IndexedGeometryArrayRetained#createIndexedGeometryArrayData(..).

From the example I posted a few days back for an IndexedTriangleArray, the float memory (coords+color_3+normals) is 13MB and the indices memory is 8MB. Because Java3D creates multiple copies of the indices array, there is a memory bloat of nearly 24MB.
And if you consider multi-texturing, the bloat is a function of the number of TextureCoordSets !

Incredible !!! Am I really seeing what I think I'm seeing ??? Does OpenGL really need separate copies of the indices array for VertexArray specification ? Is this needed for JNI data transfer ? What am I missing here ?

If you find that there really is a problem and should be fixed, I can go ahead and file an issue. The fix will be a tremendous boost for data visualization applications where even a single copy is often just too large.

Thanks

--Vaidya

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
goldw
Offline
Joined: 2003-11-16

We suffer from both the multiple universe style problem and the memory bloat problem. We now have big datasets to visualize and we frequently run out of memory when we change the indexed geometry array. You quote the USE_COORD_INDEX_ONLY but I can't find this in the documentation. Is this a new thing in build 8 and subsequent builds? We are very anxious to get these problems fixed and would be very grateful for your help.

Alessandro Borges

If you update your geometry frenquently, then you must use a geometry with
USE_COORD_INDEX_ONLY (since Java3D 1.3) . This avoid a complete vertex
recreation each time the geometry is updated. See Geometry & GeometryInfo
javadocs.
Alessandro

--- java3d-interest@javadesktop.org escreveu:
> We suffer from both the multiple universe style problem and the memory bloat
> problem. We now have big datasets to visualize and we frequently run out of
> memory when we change the indexed geometry array. You quote the
> USE_COORD_INDEX_ONLY but I can't find this in the documentation. Is this a
> new thing in build 8 and subsequent builds? We are very anxious to get these
> problems fixed and would be very grateful for your help.
> ---
> [Message sent by forum member 'goldw' (Mike Goldwater)]
>
> http://www.javadesktop.org/forums/thread.jspa?messageID=38318閮
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@java3d.dev.java.net
> For additional commands, e-mail: interest-help@java3d.dev.java.net
>
>

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@java3d.dev.java.net
For additional commands, e-mail: interest-help@java3d.dev.java.net

goldw
Offline
Joined: 2003-11-16

I downloaded and examined Alessandro’s version of the Bones.zip example and implemented the USE_COORD_INDEX_ONLY into my code. It reduced the memory requirements by a factor of 2 or 3 but it partially scrambled the picture.
Before applying USE_COORD_INDEX_ONLY, the picture is correct. But after applying the flag the picture indicates the correct shape from the wire grid diagram, but the colours are wrong giving diagonal stripes across the shape.
My guess is that it has automatically compressed the colour data which is NOT what I want. My application has a solid model in which each interior and exterior point is given by an indexed point. My application constructs varying cross-sections and similar views of this solid creating a new surface which I view using J3D. Hence I get it to view many different surfaces in one session, so need to keep my indexing and not someone else’s.
Here is an extract of the code:

boolean ucio=this.gcd.cbUSE_COORD_INDEX_ONLY.isSelected();
int USE_COORD_INDEX_ONLY=(ucio? GeometryArray.USE_COORD_INDEX_ONLY: 0);
ga=new IndexedQuadArray(grid.gridPoints.length,
GeometryArray.COORDINATES
|GeometryArray.COLOR_3
|GeometryArray.NORMALS
|USE_COORD_INDEX_ONLY
//|QuadArray.BY_REFERENCE
,nvertex);

(B.t.w. The BY_REFERENCE flag caused it to throw an exception:
java.lang.IllegalStateException: GeometryArray: cannot directly access data in BY_REFERENCE mode
at javax.media.j3d.GeometryArray.setCoordinates(Unknown Source)
...)
I construct the geometry further down the code. The only difference is the correct and incorrect view is the value of the ucio variable which is changed by selecting a checkbox.

vona@MIT.EDU

If this is indeed the case then our app (Maestro, http://mars.telascience.org) will probably see a major performance boost for large datasets with a fix.

Marty

java3d-interest@javadesktop.org writes:
> Hope I'm tracking the code correctly...
>
> From what I see, it appears that even when an IndexedGeometryArray has the BY_REFERENCE and USE_COORD_INDEX_ONLY flags set, Java3D is creating needlessly individual copies of the indices array for each of the vertex fields - coords, colors, normals, and for each of the texCoords in the case of multi-texturing.
>
> I've identified the problem to be coming from
> IndexedGeometryArrayRetained#createIndexedGeometryArrayData(..).
>
> From the example I posted a few days back for an IndexedTriangleArray, the float memory (coords+color_3+normals) is 13MB and the indices memory is 8MB. Because Java3D creates multiple copies of the indices array, there is a memory bloat of nearly 24MB.
> And if you consider multi-texturing, the bloat is a function of the number of TextureCoordSets !
>
> Incredible !!! Am I really seeing what I think I'm seeing ??? Does OpenGL really need separate copies of the indices array for VertexArray specification ? Is this needed for JNI data transfer ? What am I missing here ?
>
> If you find that there really is a problem and should be fixed, I can go ahead and file an issue. The fix will be a tremendous boost for data visualization applications where even a single copy is often just too large.
>
>
> Thanks
>
> --Vaidya
> ---
> [Message sent by forum member 'NVaidya' (N. Vaidya)]
>
> http://www.javadesktop.org/forums/thread.jspa?messageID=24289&#24289

jada
Offline
Joined: 2004-03-17

Sounds like it. Please file an issue and include your test program in it.

thanks,
Chien.

nvaidya
Offline
Joined: 2004-08-03

Any initial observations on this ?

Does the testcase/Issue23 show that there is an issue ?
Can it be fixed anytime soon ?

Will really appreciate any and every reduction in memory usage.

Thanks

--Vaidya

kcr
Offline
Joined: 2004-03-17

>Any initial observations on this ?

Yes. The bug report has just been updated with an initial evaluation:

https://java3d.dev.java.net/issues/show_bug.cgi?id=23

> Can it be fixed anytime soon ?

Certainly. That's the beauty of a public source project! :)

The file that you need to modify is IndexGeometryArrayRetained.java. See the bug report for more information.

We also need a signed Joint Copyright Assignment from you in order to integrate your fix. Please see the "Contributing to Java 3D Projects" page on the java3d web site at:

https://java3d.dev.java.net/contribute.html

-- Kevin

nvaidya
Offline
Joined: 2004-08-03

> >Any initial observations on this ?
>
> Yes. The bug report has just been updated with an
> initial evaluation:
>
> https://java3d.dev.java.net/issues/show_bug.cgi?id=23

Wonderful. Sorry if I had been a pest. But this fix is crucial to have the leanest possible data transfer to OpenGL.

>
> > Can it be fixed anytime soon ?
>
> Certainly. That's the beauty of a public source
> project! :)
>
> The file that you need to modify is
> IndexGeometryArrayRetained.java. See the bug report
> for more information.

Now, you have dropped the ball on my court - Very Smart !
OK ! I think I can make the changes and compile and verify from my end. I've not signed the JCA yet and need to look at the ramifications and it appears that probably aren't any. Just to ensure that this fix would get into 1.3.2 final version without any bureaucratic delay, I may probably request one of the people already signed in to verify and submit my patch.

Question simply is if the patch gets to you within the next week, would you be able to get it into 1.3.2 release ?

Also, I do hope that the Team would verify the patch - just in case.

And do we, as observers, have the ability to submit follow up comments to the Issues - at the least to our own ?

Many Thanks

--Vaidya

kcr
Offline
Joined: 2004-03-17

> Just to ensure that this fix would get into 1.3.2
> final version without any bureaucratic delay, I
> may probably request one of the people already
> signed in to verify and submit my patch.

There's no real hurry. The schedule we are shooting for still allows plenty of time before our Beta release. We haven't finalized this, so don't quote me on it, but 1.3.2_beta will likely be some time in October. As long as you are comfortable with the terms of the JCA, you can scan & e-mail it to us and we'll record it the same day. Or you can FAX it to us and it may take an extra day depending on how busy our administrator is.

> Question simply is if the patch gets to you within
> the next week, would you be able to get it into 1.3.2
> release ?

Yes. As long as it gets to us before the end of September, it should even make the Beta release.

> Also, I do hope that the Team would verify the patch
> - just in case.

Of course. The main thing we want to ensure is that there are no regressions: method calls that returned without exception in 1.3.1 should still work (and return the correct data in the case of get methods); methods that threw an exception in 1.3.1 should still throw an exception in 1.3.2.

For 1.4, I would like to change the specification such that the index arrays are not ever created when USE_COORD_INDEX_ONLY is set. That will simplify the code by removing the need for lazy allocation of the index arrays.

> And do we, as observers, have the ability to submit
> follow up comments to the Issues - at the least to
> our own ?

As far as I know, all project members (including Observers) can add information to any Issue as long as they are logged in (I know that Observers can submit attachments).

-- Kevin

nvaidya
Offline
Joined: 2004-08-03

> For 1.4, I would like to change the specification
> such that the index arrays are not ever created when
> USE_COORD_INDEX_ONLY is set. That will simplify the
> code by removing the need for lazy allocation of the
> index arrays.
>
> -- Kevin

That will be [i]the perfect solution[/i].

Many thanks

--Vaidya

nvaidya
Offline
Joined: 2004-08-03

OK ! I figured why I wasn't able to see the facilities for submitting additional comments to the issues. The reason was that I was registered as an observer only to the j3d-core and j3d-core-utils and *not* to the main java3d project, d'oh ! And because of that I was presented with a much more restricted "Issue View" page. Have it all sorted now.