# Is this a bug?

6 replies [Last post]
alexlamsl
Offline
Joined: 2004-09-02

I took linuxhippy's code to play around - and found out this pecularity:

Original Code
-------------------

```<br />
public static final bx = 0x4e35, cx = 0x015a;<br />
/* ... */<br />
/* within some method */<br />
/* ... */<br />
int ax, dx;<br />
ax *= bx;<br />
dx += cx;<br />
ax++;<br />
/* ... */<br />
/* ... */<br />
```

mean speed: (2.9578

alexlamsl
Offline
Joined: 2004-09-02

Ok t-test....

total N = 20
pooled s.d. / sqrt(N) = 0.016

t-stats = 0.1398 / 0.016 = 8.9

so it's unlikely enough that we should reject null hypothesis, don't you think? :)

alexlamsl
Offline
Joined: 2004-09-02

if I've done this correctly, using the formula provided:

t-stats = 0.1398 / 0.00106 = 131.8
(nu = 18)

which is even stronger.... :-/

(for tu-test, t-stats = 0.1398 / 0.001006 = 138.9, nu = 13.2)

tmarble
Offline
Joined: 2003-08-22

Hmmm.. one set of algorithms for getting
to the T-Test (actually the tutest) is here:

(tutest = unequal variance t-test)
http://www.library.cornell.edu/nr/bookcpdf.html
Chapter 14.2, p. 617

HTH,

--Tom

alexlamsl
Offline
Joined: 2004-09-02

Well the simplest T-Test is the one that assumes both samples are from the same population...

But ok a slightly more complicated calculation always gives a better sense of confidence :D

tmarble
Offline
Joined: 2003-08-22

alexlamsl & linuxhippy:

I really appreciate the energy you have put
into this... We are kind of reaching the limitation
of the forum for facilitating this kind of collaboration.

I was going to suggest adding Wiki pages to:
http://wiki.java.net/bin/view/Projects/PerformanceCommunity
But it appears that wiki does not allow attaching files :-/

So maybe we need to have a subproject which would
allow CVS access for files and docs (although
docs *may* be easier on the wiki).

To take this to the next level we need to publish
the complete code samples in a way that they can
run in a harness. I'm thinking we can leverage
the Japex microbenchmark harness:

https://japex.dev.java.net/

I haven't determined to what degree Japex (or another
harness) can help calculate basic stats
(std dev. is a must, T-Test would be great).

Regards,

--Tom

P.S. I'm swamped with about four fires at the moment
but this is *exactly* the kind of direction
we need to take the performance community.

alexlamsl
Offline
Joined: 2004-09-02

I for one really appreciate your attention in turn; I can understand the potential frustration of being fed with (unintentional yet) misleading information plus perhaps unnecessary flames burning ;)

Japex looks good, although I'm not quite sure why it needs so many average measures but does not include any measures of spread (e.g. variance). May be it's just me overlooking things ^^"

I wouldn't mind writing a few lines of code post-processing the output information to generate these stats + perform t-tests though; being a physics undergraduate + done 3 maths GCE I should know my stuff....