Skip to main content

Administration of Libaries

14 replies [Last post]
mabaumga
Offline
Joined: 2004-10-11

I would like to see implemented an administration of libaries in the next java version:

If you build an application on top of serveral applications you get big problems with the different needed libaries in classpath.

For example, application 1 needs xerces 2.5.0, application 2 needs xerces 2.6.0. How you can solve this?

There should be also a sharing of the libaries. currently every application brings all libaries with it.

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
pdoubleya
Offline
Joined: 2004-02-09

> I dont think it's important to swap versions in and
> out. It's 'side by side' thats wanted. If one library
> requires 2.6.0 of Xerces but some other library
> requires 2.5.0 (both may be 3rd party) I am going to
> be in trouble.

I understand what you need, but I think both have been suggested as "requirements" in this thread: The ability to run multiple versions simultaneously, and the ability to "swap in" a different version that should be API-compatible. I agree with you that the ability to run multiple versions simultaneously is clearly important.

My guess would be that Sun has leaned against all this because it breaks one of the golden rules of the Java language, namely, that what you see is what you get (transparency). Right now, the only confusion is about what is on your classpath. If you can actually run multiple versions, then the question is, which of my multiple classpaths (or jars) is being used at runtime? The VM could report on this (using a trace output or broadcast, as with the debugger), but it seems at first glance to hide something from the developer, and would, also at first glance, seem to make deployed applications harder to debug. This difficulty is really just a side-effect of not including versioning from the beginning, so tools, processes and our way of thinking of Java would have to change from what we've been doing so far. And, IDEs would have more work to do.

My guess is that the basic implementation is not that difficult. At the end of the day, as Anders H. from Microsoft put it, all our classnames are just a complicated string identifiying some code to run. Including the version number inside that string (which I think .Net does internally) is not that much of an issue, it's really just a compiler change and a classloader change. There might be issues for custom classloaders, depending on how they are written. My guess is there would need to be a completely fail-safe default that allows for backwards compatibility of classloaders before the versioning was introduced.

The good thing is that Sun has a golden opportunity in that there are now so many class libraries (outside of the JDK) in common use. There are actual developers using multiple versions of these, so there is a test group to work with.

The existence of all of these also pressures Sun, in that, at some point, the amount of work it takes to keep track of all these libraries and version compatibilities using current processes will just grow too large.

My gut feeling is they won't make this change willingly, because it removes some transparency from the code, but that they will be forced to if Java is to remain usable for another decade or more.

Patrick

paulbrickell
Offline
Joined: 2003-09-17

Nice entry in the thread. Can't fault your reasoning or your conclusions.

In a world where software re-use is achieved at a level greater than that of the class, this is V. Important.

I also think your right about the chances of this happening being slim, but imho that's a strategic mistake.

Here's hoping ;-)

regexguy
Offline
Joined: 2003-06-20

+1, I was thinking of posting something similar.

pdoubleya
Offline
Joined: 2004-02-09

I agree that we need something like this, and actually was just thinking of it today.

IMO, the issue is that the original packaging mechanism, composed of the classloading/java package scheme, then with jars and zips supported, was a great improvement over what existed before. In fact, it has been so successful that now we have an overload of packaged Java libraries available from all sorts of sources. Originally the classloading scheme helped resolve issues you found in C-style includes (specifically, where to find the referenced classes). It was great for that, but with more and more packages in use (I have several dozen on my machine that I use in different combinations) we need to move up one level to multi-package support, with some sort of versioning.

I think we should treat two issues separately, namely location of referenced classes, and versioning. For location, I want one canonical version of Apache Xerces 2.0 on my machine, period. I want it signed and MD5-tested so that I know it is actually from Apache. Right now every Java application I download includes dependent packages they are using with their software, so I have endless copies of common libraries. I want to be able to see all packages on my machine easily, and install, replace, or remove from a CLI or GUI interface.

That much we can do with a repository. The Apache Maven project has one proposal for a repository. It is not bad, but the wide consensus is that it is incredibly hard to set up. I think I have found at least one more group working on this as well. What would be good would be to have a Sun-sponsored spec for both the API layout, as well as interfaces for installing, querying, replacing and removing repository components. The Webstart cache should then be able to use this repository as well.

The second issue is versioning. I think that in the real world this has got to be trickier than it looks. The "contract" for Java packages--the interface and class definitions--are far too loose to be sure that I can swap out "Any version of Xerces including or above 2.0". It would be nice to do that, but we should probably realize this might not work in practice.

In any case, there are some strategies for doing this, one of which requires classloader/bytecode change and the other requires a JVM change. The basic problem is that the JVM does not support having two classes with the same full name which have different implementations in fact.

Using the current JVM, I did find a project (name?) that changes the references to loaded classes so that the classname includes a sort of version reference. All references to the classes are changed in the bytecode itself. So org.helpers.StringUtil becomes version.org.helpers.StringUtil and all references to it in bytecode are also changed. It has problems with loading classes via strings, of course, but is one solution to the problem.

You could also change the JVM to be version-aware, as the CLR is, but that would be a bigger and harder to convince Sun of.

For my part, I think the repository proposal is not that hard. There are issues of authenticity which need to be solved. But there are some advantages--your classpath could make references to repository names instead of paths, and the classloader(s) could use a repository interface to look up to associated Jars.

Patrick

dflorey
Offline
Joined: 2003-10-17

Agreed.
But the only way to solve the version and locating trouble is IMO to convince Sun that this is really a must as the number of small components (e.g. jakarta commons) is steadily increasing.
If we implement a custom component dealing with these issues, every app. vendor must be convinced to use it. This is much more hopeless than convincing Sun that they need to implement it in the JVM.

paulbrickell
Offline
Joined: 2003-09-17

A must have? IMHO yes.

To support re-use of components then some mechanism is required that allows linking to multiple versions of libraries 'side by side'. If I have dependencies on multiple libraries that require different versions of Xerces (for example) what are my options?

The way I see this these are my options:

1. I have to stop using one of the libraries.
2. Wait for updated (compatible) versions of them.
3. Get the source for one or both of the libraries and make a compatible pair. Not always possible.
4. 'Break' the delegation model of classloading in my application with a complex solution that will probably take longer to build than the application I am writing.

For developers working on large systems this really is a 'Must Have'.

pdoubleya:
[i]The second issue is versioning. I think that in the real world this has got to be trickier than it looks. The "contract" for Java packages--the interface and class definitions--are far too loose to be sure that I can swap out "Any version of Xerces including or above 2.0". It would be nice to do that, but we should probably realize this might not work in practice.[/i]

I dont think it's important to swap versions in and out. It's 'side by side' thats wanted. If one library requires 2.6.0 of Xerces but some other library requires 2.5.0 (both may be 3rd party) I am going to be in trouble.

This is an interesting thread. I hope theres more to come.
Paulb.

dflorey
Offline
Joined: 2003-10-17

For people too lazy to follow the thread on jakarta commons concerning the same issue - this is a very interesting post describing how it is handled in .NET:

The .NET equivalent of a jar file is called an assembly. For libraries,
this is basically a DLL. Every time the code is compiled, the DLL is
automatically allocated a unique version number. When you compile your
code that refers to code in a library assembly, your assembly has an
explicit dependency on that library assembly. At runtime, when your code
tries to invoke the library code, an exception will be raised if the
exact version of the library assembly is not available.

It would appear that if there are bug fixes or other improvements to the
library, and a recompiled assembly DLL is substituted for the one you
originally compiled against, then your code will break. At runtime, your
code will fail to link to the library code, since the version number no
longer matches. Obviously, a maintenance release of a library component
shouldn't require a recompilation and redeployment of all software that
uses the library, so .NET provides a mechanism for you to explicitly
define a version number. This allows you to provide updated library
components to users without requiring them to recompile. However, this
only works if you don't break backwards compatibility.

If you break backwards compatibility in a library, then you have to
change the version number. However, .NET still allows you to deploy
different, incompatible versions of the same DLL. When you deploy the
application, your installer has to register both versions of the DLL
with the GAC - the Global Assembly Cache. In this way, if you have a
complex application that contains two components that rely on
incompatible versions of the same library DLL, the VM instantiates two
separate versions of the library DLL, and links the two components to
the appropriate instance.

One possible Java analogy to this would be to bundle all code inside jar
archives. Each jar contains dependency information, perhaps stored in
the manifest, or some other meta-file, that describes the jar's own name
and version number, and a list of the names and version numbers of its
dependencies. A suitable class loader can then use this meta information
to stitch the classes together appropriately. Actually, my knowledge of
java class loaders isn't sufficient for me to assert that this solution
would definitely work, but it's a start, and I hope all of this serves
to illustrate how .NET allows multiple versions of the same library to
coexist.

Chris

paulbrickell
Offline
Joined: 2003-09-17

amen

carmello
Offline
Joined: 2003-09-10

What do you mean with: " build an application on top of serveral applications" ?
You could load every sub appication with his own classloader. This classloader breaks the delegation model of classloaders, so instead to look first in the classpath of the parent classloader, first look in the classpath of the sub application then in the parent.
So it the application classpath contains xerces 2.5.0 and the classpath of your sub app 2.6.0 it will find 2.6.0 first and use it.

I am working on an open source project (www.platonos.org), one of the things we are working on is a pluginengine that has a special classloader that loads the classes of a plugin, this classloader breaks the delegation model but different than I descripted above. Our pluginengine is used in a few commercial products but it could also be used in a non commercial product, its availible under BSD licence. We are also working on a swing ui framework on top of the pluginengine and a swing component library. Our project is hosted at: www.sourceforge.net/projects/platonos

dflorey
Offline
Joined: 2003-10-17

+1 from me to solve this issue in the JVM (classloader).
I've discussed this recently in the jakarta commons mailing list and my proposal to resolve this by using different package names for each version has been rejected (more or less):

http://nagoya.apache.org/eyebrowse/BrowseList?listName=commons-dev@jakar...

There is already a way to add information about the jar dependencies in the manifest, so it should be possible to load the appropriate classes if this is handled in the classloader.
This is handled very well in .net, so this should be possible in java as well!

dflorey
Offline
Joined: 2003-10-17

BTW:
This feature is really a must have! The more java components get spreaded around and will be used by different products (think of jakarta commons, jdom...) every bigger java based project will run into this trouble. If you ever had to assemble different commercial products using different version of the same jar, you'll know what I mean.
This is not a problem in the "perfect world" where you have the sources of every product and every product vendor instantly uses the latest and greatest version of a jar - this is a *real problem* in the *real world*.

hlavac
Offline
Joined: 2003-09-16

Yes, a must have feature. Add to it a generic Java based installation/integration API to allow integration as system services/daemon, WAR in web container, command-line command, desktop application icon/menu, etc (extensible), and a online API metadata repository for published Java APIs with download poitners for implementations and documentation... and you have a perfect world ;)

vhi
Offline
Joined: 2004-10-11

I would suggest looking into OSGi recommendations and providing a OSGi-enabled JRE.

Having a proper specification for versioning of components, running multiple versions of components (side-by-side execution), etc are a must.

BTW, even though there are recommendations for versioning (Java Product Versioning Specification), even Sun is not using it!

There must be some way to load a desired version of component from a package instead of depending upon the user specifying X.jar in front of Y.jar in the classpath to ensure that my code runs properly. It is a very fragile process.

vhi
Offline
Joined: 2004-10-11

In addition to the posting mentioned above..

For example:

A user has Xerces_1.0.jar and Xerces_1.2.jar in his classpath.

Since I have written code to work with Xerces_1.2.jar, I have to ensure that the user does not have to fiddle with Classpath entries/Bat files so that Xerces_1.2.jar comes first in the classpath for my application to run.

As a developer, I can specify the version I want as following:

@Version (">1.2")
import org.apache.xerces.*;

And the JAR file that contains a Manifest entry for the package, which in turn has a package version for "org.apache.xerces" as 1.2 or greater will be used.

In this way, using a combination of JAR deployement, Manifest entries and a Version annotation, we may solve the versioning problem.