Skip to main content

More flexible build

23 replies [Last post]
opinali
Offline
Joined: 2003-06-17
Points: 0

One major problem for J2SE as an open-source project is that it's too difficult to build. This is a huge barrier to entry for most developers interested in really experimenting with the sources.

The Linux build requires a very specific version of Linux (Redhat Enterprise Advanced Server 2.1 update 2). I haven't yet tried to build this release, but my experience with 1.4.x releases is that the build will fail with thousands of cryptic compiler/linker errors if I use anything but the exact distribution recommended by Sun.

The Windows build is much worse. It requires the MKS Toolkit, which is a commercial product (some time ago I tried to use Cygwin, without success). It requires an old version of Microsoft VisualC++ (6.0sp2) which is also commercial; I have this one, but I'd hate having to install that old junk just for compiling the J2SE. Ideally, the build should work with current VC++ versions, including the now VisualC++ Toolkit (which is freely available and contains the latest and greatest MSVC/C++ command-line compilers) and the latest Windows SDKs.

In addition to disencouraging many developers to go beyond browsing the sources, the restrictions of the Windows build should affect the performance of the JVMs too. Microsoft's VC++ compilers received very important optimization improvements in the last few releases. The JVM and its native libs contain a good amount of C code, so you could give a "free lunch" boost to all Java apps in some areas (loading time, overhead of JIT&GC, APIs depending on native libs) by simply exploiting the best compilers available. BTW, the Intel C/C++ compiler is highly compatible with MS, supports Linux too, and it was even better than MS last time I checked, so it would be great to support that compiler too.

We know that the J2SE is a huge project, full of low-level code, and C/C++ compilers are famous for screwing up code under strong optimization. Validating the project for new compilers must have some cost in additional testing and maybe fixing some compiler-dependant code. But it should be worth the effort.

As for the other issues, in my dreams the J2SE project would have an autoconf script like any regular open-source C/C++ project... for my personal taste, this is more urgent than moving to a "real open source" license ;-) but if that's too difficult, even a big, I'd be happy if I could manually configure the build for my environment, even with an ugly and complex (but well-documented) set of scripts or configuration files.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
jwenting
Offline
Joined: 2003-12-02
Points: 0

Ideally the build should be possible on any ANSI compliant compiler.
In practice any non-compliant (therefore compiler-specific) code should be moved into a limited number of sources, thus ensuring that the smallest possible subset of the code is platform specific.

I've not checked out the sourcecode (yet), no time at the moment, but I'd guess this would be a major undertaking.

Personally I'm not at all enamoured of the value this would have towards an OS version (as I'm more or less diametrically opposed to such a version being released) but it would over time make it cheaper and potentially faster to release JVMs on different OSs under some sort of license (Sun could release part of the code under a license requiring certification of the resulting JVM for example) which can only be a good thing.

kellyohair
Offline
Joined: 2004-09-03
Points: 0

The source itself is mostly organized around OS/ARCH, e.g. src/share vs. src/solaris vs. src/windows. In general I agree that we need to stick with standard C/C++ language constructs, we pretty much do, but it's often a hard thing to police all the time.

It's the Makefiles that have to know specifics about how a compiler spells certain command line options etc. We will try and make this easy to use different compilers, but since we won't be actually using them ourselves we'll need others to let us know how well it works. Stay tuned. And thanks for the comments.

cowwoc
Offline
Joined: 2003-08-24
Points: 0

One of the most important reasons I would suggest the build system support VC++ .NET 2003 as a building platform is that it is *far* more ANSI C/C++ compliant than VC++ 6.0. It will also catch buffer overflows and other nasty bugs whereas VC++ 6.0 will not. And lastly, optimization is obviously better.

True, not everyone will have VC++ .NET 2003 handy, but let's face it: if you're a commercial windows developer, it is highly likely you have this version handy. You will also find that once the code builds without errors or warnings under VC++ .NET 2003 that it'll be much closer to building under the free Microsoft tools. In fact, I am fairly certain the build process would be almost identical (except debug mode, which others have mentioned).

From personal experience, VC++ 6.0 is crap (lots of bogus warnings, bad code generation). I'd even advocate dropping VC++ 6.0 support (if absolutely necessary) in favor of VC++ .NET 2003.

Gili

kellyohair
Offline
Joined: 2004-09-03
Points: 0

Valid points. We have already made some changes with regards to the newer compiler detecting some C++ standard issues in hotspot, these changes will be in one of the builds coming up.

Our plan is to make the first step converting to the new compiler, allowing use of the older compiler for a time, then try and move to the free compiler. But this will take several builds to get done. Early indications are all positive, performance included.

Thanks for the info.

dog
Offline
Joined: 2003-08-22
Points: 0

> I tried to use Cygwin, without success). It requires
> an old version of Microsoft VisualC++ (6.0sp2) which
> is also commercial; I have this one, but I'd hate
> having to install that old junk just for compiling
> the J2SE. Ideally, the build should work with
> current VC++ versions, including the now VisualC++
> Toolkit (which is freely available and contains the
> latest and greatest MSVC/C++ command-line compilers)
> and the latest Windows SDKs.

Sure it should work with new ones. But I don't think they shuold necessarily SUPPORT new compilers. Why? They are building a software platform. Of all the features, stability is the most important. A change of compilers is a serious business decision that should have a benefit that outweighs the cost. Is the code going to be significantly faster?? Is it going to bloat everything up?

In the end I don't care what compiler they use.. as long as the JDK works well for me. Some projects upgrade compilers frecuently for no reason.. I think Sun is correct in avoiding that.

opinali
Offline
Joined: 2003-06-17
Points: 0

> Is the code going to be significantly faster??

Some code may be significantly faster indeed. Not JIT-generated code of course, but still, code that contributes significantly to your performance, like the JIT, GC and fdlibm (native math libraries).

> Is it going to bloat everything up?

In C++, bloat is mostly an issue with higher-level features like templates, RTTI etc., which are not used by J2SE's sources. New compiler may even reduce the bloat, because the latest compilers (both MS and Intel) can use profile-driven compilation so they can do a smarter choice of optimizations, e.g. avoiding inlining or loop unrolling where these won't help, although it takes extra work to generate and use profiling data.

> In the end I don't care what compiler they use.. as
> long as the JDK works well for me. Some projects
> upgrade compilers frecuently for no reason.. I think
> Sun is correct in avoiding that.

I agree that it's not a great idea to rush to new compilers every year. But never updating is just as bad -- VC++6.0 is almost 5 years old, an eternity in compiler technology. And one of the resons that *I* care, is that competitors of Java (like Microsoft's .NET VM) are compiled with the latest and greatest C/C++ compilers.

kellyohair
Offline
Joined: 2004-09-03
Points: 0

We are just getting started on moving to newer compilers. On Windows we want to try and use the free Miscrosoft compilers, but it turns out that not everything we depend on is in the free compilers (we don't know if that was an oversight or done on purpose, and we might be able to find what we need in other free downloads, we don't know yet). For example, we have a need for a '#include "atlbase.h"' but this include file is not part of any free downloads we can find. And I'm still investigating why we need it.

Debug builds in particular are an issue (the debug DLL's from Microsoft are not on the system, or in the free compilers, and cannot be re-distributed). One thought is to by default only build the optimized J2SE (currently by default both an optimized and debug version is built), and anyone wanting a 'debug' build would need to 'make debug' and perhaps have purchased a set C/C++ of compilers/tools that can build a debug version. I'm curious what people think of this.

It sounds like we need to configure the makefiles in such a way to allow for the use of an arbitrary C++/C compiler set to be used. I'll file an RFE on this, it's a good idea.

robilad
Offline
Joined: 2004-05-05
Points: 0

This may sound like a stupid question, but ... why would a Java virtual machine need COM objects, Automation services or ActiveX controls?

As that's what ATL does, according to
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/...

cheers,
dalibor topic

kgh
Offline
Joined: 2003-06-10
Points: 0

This is probably for the Java Plug-in. It runs as
an ActiveX control inside of Internet Explorer (and
other COM containers) and it does its best to integrate
cleanly as a well-behaved COM component.

- Graham

robilad
Offline
Joined: 2004-05-05
Points: 0

Ah, that explains it. Thanks, Graham!

cheers,
dalibor topic

zander
Offline
Joined: 2003-06-13
Points: 0

> The Linux build requires a very specific version of Linux (Redhat Enterprise Advanced Server 2.1 update 2).

XAWT replaced the motif dependency, right?

bino_george
Offline
Joined: 2003-06-16
Points: 0

> > The Linux build requires a very specific version of
> Linux (Redhat Enterprise Advanced Server 2.1 update
> 2).
>
> XAWT replaced the motif dependency, right?

Correct. As long as you only build XAWT Toolkit, But by default the Motif Toolkit is also built, in order to provide
a choice. This might change in Mustang.

zander
Offline
Joined: 2003-06-13
Points: 0

> BTW,
> the Intel C/C++ compiler is highly compatible with
> MS, supports Linux too, and it was even better than
> MS last time I checked

As long as you run Intel processors, I assume?

opinali
Offline
Joined: 2003-06-17
Points: 0

Of course, but what's your point? Notice that I didn't suggest to move from MSVC++ to Intel C/C++, but only to support Intel's compiler as an option. For Windows builds like AMD64, just keep using Microsoft's compiler if Intel's doesn't support that. Besides, Intel has embraced AMD's 64-bit ISA now so this shouldn't even be an issue (and J2SE's sources use ASM, either hand-coded or generated, for the most extremely CPU-specific stuff, so it shouldn't depend on the C/C++ compiler for the dirtiest tricks).

kellyohair
Offline
Joined: 2004-09-03
Points: 0

We are in the process of prioritizing the build improvements we need to do in Mustang, and you have pointed out most of the higher priority ones.

So expect these kind of changes to made over the coming months. It's hard to predict when we will have these done.

As you know, changing native compilers is something we need to do very carefully, but we will be changing, we just have to be very careful. The newer compilers can create some compatibility problems, and we need to make sure we don't break any older products that need to mix with the J2SE libraries.

Thanks for the post, and the suggestions.

wingetr
Offline
Joined: 2004-01-19
Points: 0

I whole-heartedly agree. Free / Open Source projects should have minimal commercial requirements.

I also would be interested in hearing why the Windows build requires an older release of VC++. I can understand saying "VC+ 6.0 or newer", but to force people to downdrage is a bit unusual.

kellyohair
Offline
Joined: 2004-09-03
Points: 0

I think that the J2SE project has historically been stung quite a bit by new C/C++ compilers performing new optimizations, and triggering bad interactions with the JIT compiler or the HotSpot compiler optimizations. So changing C/C++ compilers is always considered a somewhat risky thing to do for us. So even when the C/C++ compiler is relatively bug-free, there can be other issues for the J2SE.

With each J2SE release we try and move forward to the newest publicly available C/C++ compiler that remains compatible for as many of our customers as possible.

It's very possible that the Windows compiler upgrading has been slower due to the major expense it triggers. The free version should change at least the expense reason.

coxcu
Offline
Joined: 2003-06-11
Points: 0

I know this is not likely to happen, but just imagine the benefits of moving everything to GCC. You would have a single compiler across all platforms and an easier migration path for moving functionality from C to Java (the language).

kellyohair
Offline
Joined: 2004-09-03
Points: 0

Certainly using a single compiler with the same command line has it's advantages. And I also have wondered if it might be better to use one compiler. But in reality we have not been able to use the same gcc compiler on all Linux machines, we have to use different versions.

Speaking as someone that builds the J2SE on Solaris, Linux, and Windows all the time, when you crank up the optimization level to the max, you want the best optimizing compiler available, where best includes quality and performance. And the ABI stability of the Linux g++ compiler and runtime libraries is a major pain right now, in my opinion.

On Windows, we have gone down the road of most likely used, and most compatible. We are currently investigating if it is possible to use the free compilers from Microsoft. I honestly hadn't thought about gcc on Windows, that could be a major effort considering how much windows source we have that might be pretty tied to the Microsoft compiler.

But I understand where you are coming from, I just don't know that it will ever happen.

robilad
Offline
Joined: 2004-05-05
Points: 0

Is there some reason why you don't just use autotools on all platforms?

cheers,
dalibor topic

robilad
Offline
Joined: 2004-05-05
Points: 0

What many people don't know is that one can use autotools to build programs with MSVC++ and respective DLLs instead of gcc. It requires the use of a clever m4 macro on sourceforge[1], but it works like a charm for a theorem prover I work on that needs to use the MSVC++ compiled Qt libs on Win32.

So for the theorem prover I have a single build system using autotools covering Win32, Linux, Solaris, MacOSX, and other platforms with the additional ability to cross-compile between them. Depending on what you need to do on the various platforms, using autotools may be a cheap way to unify your build systems, and avoid relying on exotic toolsets like MKS.

I guess the JDK's build system simply grew out of necessity and the lack of useable cross-platform build toolkits back in the 90s :)

cheers,
dalibor topic

[1] cccl.sf.net

kellyohair
Offline
Joined: 2004-09-03
Points: 0

I'm not sure I'd label MKS 'exotic' :^). Anyway, thanks for the information. Obviously the J2SE build system has grown up over many years and unfortunately has lots of baked in workarounds for compiler bugs (some which may or may not still exist) and system problems. We will do the best we can to make this better, but it will probably be done in steps or stages.

When you say 'autotools' you are refering to automake, autogen, and autoconf?

robilad
Offline
Joined: 2004-05-05
Points: 0

Thanks for the fast reply, Kelly!

Yeah, I meant automake, autoconf & libtool. Best of luck revising the build system, I can imagine it's quite a bit of work.

cheers,
dalibor topic