Skip to main content

Performance regression in 6u12 b02

40 replies [Last post]
kirillcool
Offline
Joined: 2004-11-17
Points: 0

Running the LightBeam dynamic tests on the final 6u10 and 6u12 b02 shows that there is a consistent performance regression under both core and third-party look-and-feels. Here are the numbers:

Under 6u10

Metal - 6494
Nimbus - 8950
Substance - 9426
Synthetica - 10080

Under 6u12 b02

Metal - 6725 (3.5% slower)
Nimbus - 9090 (1.5% slower)
Substance - 9580 (1.6% slower)
Synthetica - 10257 (1.7% slower)

Is this the result of the new HW-LW related enhancements / bug fixes?

Thanks
Kirill

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
cowwoc
Offline
Joined: 2003-08-24
Points: 0

How does u12 b03 compare to u10?

kirillcool
Offline
Joined: 2004-11-17
Points: 0

You can find the different numbers scattered throughout this thread. Apologies for not having enough time to keep track of all the available final and development versions of 6u*.

ewin
Offline
Joined: 2005-07-15
Points: 0

> The engineers don't even see a bug until it enters bugtraq.

So they can use plausible deniable. As if they couldn't look it up by themselves in the bugparade if they would be interested.

The bug parade is Sun's second most disgusting collection of broken promises. The most disgusting on is the voting on the bug parade (the bug voted number one is six years old, number two is nine years old, and number three 10 years, quickly approaching 11 years).

On a good day I call it a pure charade. A farce. Soviet-style propaganda. Big bullshitting. Piss take. A spoof. A long-running scam. Window-dressing. The Dave Null of problem handling. The poster child of incompetence and ignorance. Pure rubbish. A fake. A demonstration of arrogance with a full frontal kick in the face of every honest Java programmer. Potemkin's second coming.

In short:

[b]The parade of shame[/b]

mbien
Offline
Joined: 2007-04-29
Points: 0

> On a good day I call it a pure charade. A farce.
> Soviet-style propaganda. Big bullshitting. Piss take.
> A spoof. A long-running scam. Window-dressing. The
> Dave Null of problem handling. The poster child of
> incompetence and ignorance. Pure rubbish. A fake. A
> demonstration of arrogance with a full frontal kick
> in the face of every honest Java programmer.
> Potemkin's second coming.
>
> In short:
>
> [b]The parade of shame[/b]

I call it bug tracker... much shorter

trembovetski
Offline
Joined: 2003-12-31
Points: 0

> I have absolutely no way to tell them they're wrong.

When you comment on the bug, the responsible engineer gets a notification email.

Dmitri

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Ahh, nothing like a change of the subject, and a good generalization to boot =)

I don't know about Swing, but 2D (thanks to the efforts of Igor Nekrestyanov) didn't have a (large) bugs backlog for a few years - meaning, submitted but not entered into bugtraq.

They're still working on a public bug management system for 7. It's not a trivial task.

Dmitri

cowwoc
Offline
Joined: 2003-08-24
Points: 0

I have had some luck going through BugParade but on the whole I agree with you. The sooner Sun moves to a two-way communication system like JIRA, the better. There has been countless times where bug evaluators have closed bugs when they misunderstood the report or filed it as a duplicate of another unrelated bug (this happened again this morning!) and I have absolutely no way to tell them they're wrong. I can post comments to my heart's content but they'll never receive it or take me seriously.

If Java was run like Netbeans on a community/daily-level level then we'd have a lot more contributions, bug fixes and user satisfaction. Furthermore, I argue Sun can open up the issue tracking mechanism without giving away control over the overall direction of the Java platform (which they seem to fear).

osbald
Offline
Joined: 2003-06-13
Points: 0

Looking forward to 7.0 which is I believe GPL+Classpath maybe it well be time to revise how bug reporting and submittal of patches is handled. While Sun were a closed shop bugparade was probably adequate. But now may have to consider contributors on par with internal dev teams who need to be informed of pending fixes, priorities, assignments and planning to avoid conflicts. Does being 'open' mean more than just the licence in the readme?

What would Sun do with either too many contributions, or contributions than conflicted with either each other or Suns plans? Who does all the regression testing? will there be a Continuous integration server for the runtime / runtime libs (hmm modularization another issue).

tdanecito
Offline
Joined: 2005-10-10
Points: 0

Actually, there is a older bug that I believe goes back to 1999. I filed the bug you mentioned since I needed it fixed for MyUniPortal many years ago where I was mixing video and swing.

Regards,
Tony Anecito
Founder,
MyUniPortal
http://www.myuniportal.com

kirillcool
Offline
Joined: 2004-11-17
Points: 0

> Basically we need to convince someone from the Swing
> team to take ownership of this task, evaluate and
> adopt/integrate the benchmark into our performance
> testing framework.

Well, they know where to find it :)

trembovetski
Offline
Joined: 2003-12-31
Points: 0

You got to do something to market it better, googling for "lightbeam benchmark" didn't get me there quick =) (yeah, I forgot where to find it)

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Googling "lightbeam" has it on #3 on my machine.
Googling "lightbeam benchmark" at #2 point to my blog that points to that project.
Googling "lightbeam swing" at #1 points to the blog, and #2 points to the project.
Googling "lightbeam testbed" (the official description of the project) at #1 points to the project.

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Running b03 against b02:

* Metal and Windows have less than 0.8% difference
* Nimbus is 4.6% faster
* Substance is 2.0% faster

Thanks
Kirill

trembovetski
Offline
Joined: 2003-12-31
Points: 0

That's probably the results of AA rect performance fix in b03. You're still running the test on your Intel-based video notebook, right? (which means no d3d)

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Yes, that's the configuration.

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Another way to determine if the new loops are being used is to run with -Dsun.java2d.trace=count and see how many Pgrm* primitives are there.

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Dmitri - any news on this? Have you seen this difference in your internal benchmarks (Java2D / Swing)?

Kirill

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Sorry, didn't have time to check.

I doubt we'd be able to do anything for 6u12 though, b03 is pretty much done and it's supposed to be the last build. So unless it is magically fixed in b03 by whatever changes AWT made it'd have to wait until 6u14 (since u13 is supposed to be a security release).

Dmitri

trembovetski
Offline
Joined: 2003-12-31
Points: 0

The good news is that in b03 there will be some changes to the sw rendering pipeline which will hugely improve filling and drawing of un/transformed AA rectangles. May be it'll offset this regression enough =)

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Following your earlier suggestions, i made sure that all fillRect operations are done on non-AA rectangles. If Metal is doing the same, not sure how much effect that will have. A related question - is Java2D benchmark a routine part of the internal builds / regression tests?

Kirill

trembovetski
Offline
Joined: 2003-12-31
Points: 0

> Following your earlier suggestions, i made sure that all fillRect operations are done on non-AA rectangles. If Metal is doing the same, not sure how much effect that will have.

If you render any aa or non-aa transformed rectangles, it would show.

> A related question - is Java2D benchmark a routine part of the internal builds / regression tests?

Not the 2D one unfortunately (it's in the next version of the performance suite that's being run currently for each promoted build), but we do run SwingMark test for Ocean and Native L&Fs.

These benchmarks show statistically insignificant regression of ~4% between b6u12 b01 and b02 (typically statistically insignificant errors are ignored), and about 1% statistically significant regression since 6u11.

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

> but we do run SwingMark test for Ocean and Native L&Fs.

Have you considered either open-sourcing it, or adopting / enhancing the LightBeam testbed?

Kirill

trembovetski
Offline
Joined: 2003-12-31
Points: 0

> Have you considered either open-sourcing it, or adopting / enhancing the LightBeam testbed?

There was some reason for why we can't open source Swingmark, I forget what it is.

We did consider adopting new benchmarks for Swing since SM is very limited, but never had the time to either develop a new one or evaluate the existing ones. May be we should now.

Basically we need to convince someone from the Swing team to take ownership of this task, evaluate and adopt/integrate the benchmark into our performance testing framework.

Dmitri

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Ok, according to the AWT team the introduced regression was planned. Meaning, it was the price to pay for the hw/lw mixing feature.

The performance fix I saw was related, but it's already fixed in b02.
6768230: HW/LW mixing code slows down resize performance

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

> Ok, according to the AWT team the introduced
> regression was planned. Meaning, it was the price to
> pay for the hw/lw mixing feature.

Three things that are missing here, once again due to continuous lack of communicating the plans and new features:

  1. New feature has been added without any input from the community. We're still unclear what does it serve? Is it for the general advancement of AWT and Swing, or are there any JavaFX-related bits that explicitly required this to be put in place (accidentally benefiting those Swing apps that use heavyweight components).
  2. Performance regression that is known to the AWT team, but not communicated in the release notes. This is bad.
  3. Performance regression for all Swing applications. Even those that have nothing to do with heavyweight components. It might be small, but this is still a regression. This is bad as well.

Once again, communication is the key. If you can't be open about your future plans, at least be open about things that are being put into the platform. Me running the LightBeam tests on this specific build was a coincidence that has uncovered this regression, but it shouldn't be this way.

Thanks

Kirill

Message was edited by: kirillcool

trembovetski
Offline
Joined: 2003-12-31
Points: 0

> New feature has been added without any input from the community.

Would 24 votes and multiple duplicates count as community input? Customer visits? What about hearing requests for this feature at every Javaone? If so, the community have spoken and it apparently does want this feature.

> Is it for the general advancement of AWT and Swing, or are there any JavaFX-related bits that explicitly required this to be put in place

The bug is 5 years old. http://bugs.sun.com/view_bug.do?bug_id=4811096
It has been on AWT plate for a while. Please be reasonable with conspiracy theories.

> Performance regression for all Swing applications. Even those that have nothing to do with heavyweight components. It might be small, but this is still a regression. This is bad as well.

I will hazard to guess that one would be hard-pressed to detect this regression in any swing application since even in benchmarks it is only 2-3%.

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Dmitri,

Allow me to clarify my comments.

> The bug is 5 years old.
> http://bugs.sun.com/view_bug.do?bug_id=4811096
> It has been on AWT plate for a while. Please be
> reasonable with conspiracy theories.

My conspiracy theory is this - all the improvements that i have seen in 6u10-6u12 family were driven by the consumer-oriented requirements of JavaFX. You need translucent and shaped windows to compete in this market, you need faster rendering of different graphical primitives, you need faster startup time, and more. Now, this does not mean that i am not glad to see these improvements available for "pure" Java2D and Swing applications. But seeing all these longstanding areas being addressed, i can't help but wonder - should i thank Flex / Silverlight / ... for having such a commanding lead in the consumer market that Sun had no choice but finally to address all the issues that have been long pointed out by the users. This is my conspiracy theory. As a corollary, my assumption is that the enormous amount of attention being put into HW / LW mixing in 6u12 (and beyond) can only mean one thing - that JavaFX absolutely requires it for one or more items on its roadmap. No communication = conjectures = conspiracy theories.

> I will hazard to guess that one would be hard-pressed
> to detect this regression in any swing application
> since even in benchmarks it is only 2-3%.

That is true, but my point is still valid - i'm paying for something that i'm not using.

Thanks
Kirill

trembovetski
Offline
Joined: 2003-12-31
Points: 0

> My conspiracy theory is this - all the improvements that i have seen in 6u10-6u12 family were driven by the consumer-oriented requirements of JavaFX.

That's a fine conspiracy theory as far as they go. Like any good theory, it even sounds plausible, mostly because there are bits of truth in it.

I can tell you that most of the client improvements in 6u10 were planned WAY before anyone inside or outside Sun heard of JavaFX. Whether you believe it or not, up to you - I don't really care.

But sure, some of the features/fixes were pushed by JavaFX. Not necessarily because JavaFX had to have them, but because now we have something in house which uses our stuff, and it exposes issues, some of them serious enough to be fixed right away. It's different when someone from the other end of the hall comes and tells you that your stuff is broken.

> That is true, but my point is still valid - i'm paying for something that i'm not using.

The point is, you're not paying =) See if your users notice the difference of 3% in _benchmarked_ performance.

It doesn't mean that 1%-3% performance regression can't accumulate into something noticeable, of course.

Anyway, we'll ask AWT team to see if the penalty for apps not using can be eliminated. The earliest anything can be done would probably be 6u14.

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

> But sure, some of the features/fixes were pushed by
> JavaFX. Not necessarily because JavaFX had to have
> them, but because now we have something in house
> which uses our stuff, and it exposes issues, some of
> them serious enough to be fixed right away. It's
> different when someone from the other end of the hall
> comes and tells you that your stuff is broken.

Which reminds me of another conspiracy theory that i frequently revisit - that the only way to have an issue fixed is to have it submitted to bug parade by the member of the relevant internal development team just before he / she intends to fix it.

I've stopped submitted issues about two years ago when three consecutive reports were not even acknowledged and never made it to the public site. I believe that this specific theory is shared by many people.

Once again, this brings me back to the issues of communication and opacity, which i talked about at [1]

Kirill

[1] http://www.pushing-pixels.org/?p=822

kleopatra
Offline
Joined: 2003-06-11
Points: 0

>
> Which reminds me of another conspiracy theory that i
> frequently revisit - that the only way to have an
> issue fixed is to have it submitted to bug parade by
> the member of the relevant internal development team
> just before he / she intends to fix it.
>
> I've stopped submitted issues about two years ago
> when three consecutive reports were not even
> acknowledged and never made it to the public site. I
> believe that this specific theory is shared by many
> people.
>

yeah, think so too. My corollary of that is that the only reasonable way to get a bug pushed is to personally know somebody inside who personally takes care of any report and/or issue, at least to the extend of getting it through the first hurdle. Otherwise, even real show-stoppers tend to be ignored ... (shameless high-jack of this thread ) as were those ominous NPEs on standalone JTableHeader (high priority for Vista and only partially fixed and closed):

Issue #6668281 - http://bugs.sun.com/view_bug.do?bug_id=6668281
Issue #6776669 - http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6776669

Cheers
Jeanette

trembovetski
Offline
Joined: 2003-12-31
Points: 0

It doesn't help the situation of course, but the cause of that the lack of people who are supposed to be going through new requests and determining if they're bugs. Swing gets an order of magnitude more reports than 2D, so it's practically a full-time job.

The engineers don't even see a bug until it enters bugtraq.

Dmitri

cowwoc
Offline
Joined: 2003-08-24
Points: 0

> The engineers don't even see a bug until it enters
> bugtraq.

Why? Why do you need a middle-man?

Bare with me on this one :) I argue that if users keep on reporting the same non-bug over and over again then clearly there *is* a bug and the fact that the developers never get to hear about it is a problem.

If Swing has a very high user error rate (you get bug reports for problems caused by Swing configuration problems instead of the code not working as expected) then you clearly have two problems:

- Swing isn't as intuitive, ease-to-use as it should be.
- Swing isn't as well documented as it should be

The fact that developers suffer with numerous "annoying" reports is a good thing because it encourages them to fix the underlying problem. Outsourcing it to someone else doesn't make the problem go away. It just means that you end up wasting money at a lower hourly rate. I'd rather stop wasting money *altogether*. This is one of the reasons I keep on saying: *direct* communication between developers and end-users is vital!

If you're going to put out a so-called "transparent" issue tracker for Java7 but it is not going to be *direct* then you've totally missed the point in my view. Direct *is* transparent. Transparent *is* direct.

Gili

trembovetski
Offline
Joined: 2003-12-31
Points: 0

> Why? Why do you need a middle-man?

Because that's the way the current bug tracking system works.

I didn't say it'll stay the same once we migrate to open bug tracking system.

Dmitri

tdanecito
Offline
Joined: 2005-10-10
Points: 0

Okay I was wrong it was back in 1998 the bug was listed. Hey, after working with Java for over 12 years you forget some things.

http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4154448

Best Regards,
Tony Anecito,
Founder,
MyUniPortal
http://www.myuniportal.com

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Update: the regression caused by the hw/lw mixing is confirmed, and filed as
6789096: HW/LW Mixing: 2% regression in 6u12 b01, alacrity SwingMark benchmark

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Hm. Whatever it is, I don't think it's expected.

Did you try 6u11? Could it be regression in 6u11?

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Under 6u11:

Metal - 6494
Nimbus - 8890
Synthetica - 10082

Thanks
Kirill

trembovetski
Offline
Joined: 2003-12-31
Points: 0

Thanks. I'll file a bug for tracking. I'll also check b03 (should be out next week), I think I saw some performance-related fix from AWT regarding hw/lw mixing.

Do you see the same drop with D3D pipeline disabled? (at least for metal)

Dmitri

kirillcool
Offline
Joined: 2004-11-17
Points: 0

All the numbers are for software pipeline.

Thanks
Kirill

kirillcool
Offline
Joined: 2004-11-17
Points: 0

Additional information - for another machine that is running Windows Vista with dual core CPU and no hardware acceleration.

Up until last week (latest batch of Vista updates), the numbers for 6u10 were:

Metal - 3039
Windows - 3721

Now it's:

Metal - 2925
Windows - 3625

These numbers are pretty identical for 6u10 and 6u11 (before and after the Vista update), and the Metal number for 6u12 is 3037 - around 3.8% slower, consistent with the slow factor on the other machine.

Thanks
Kirill