Skip to main content

"proven" libraries and what's in there?

7 replies [Last post]
sebastiankirsch
Offline
Joined: 2005-06-24
Points: 0

I just made some experiments with the hprof-feature of the JVM as we had some memory problems on our tomcat.
There, I discovered a single object using 262152 bytes... after some analysis, I found this snippet of source code:

private static final String[] PADDING = new String[Character.MAX_VALUE];

I guess you find this as intriguing as I do. I don't want do discuss a solution or best practice here - my point is:
this piece comes from StringUtils, from the well-known commons lang library (v2.1). So, we have ~ 30 apps running on one tomcat, which sums up to something like 8MB memory usage just for this crap (alright, there's the common/lib/ directory - but is it save to put that lib in there? probably it is, but especially now, I wouldn't count on that).

I wonder which other "leaks" are hidden in there and other commonly used libraries. Do you check the libraries you are using? Did anybody notice similar "pitfalls"? I wonder how good an idea it is using such "all-purpose" libraries if you could also set up your own Util classes - providing just the 3,4 methods you need.
Yeah i know. Who checks my code and now that this is uncovered and all is open source, this flaw will probably be fixed in the next release. But nevertheless...

What do you think about this?

Message was edited by: sebastiankirsch

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
sebastiankirsch
Offline
Joined: 2005-06-24
Points: 0

Hi alexlamsl!

Yes, we are all humans and yes, we all make mistakes. I certainly don't want to state something else.

It's just that I would have thought that a library that is around for ~5 years would no longer contain such flaws. Apart from the fact that, as denka & jwenting mentioned, you scarcely have to chance to update your lib or use the newest lib at all (I guess every company has some common "software" where the company's tools & used libs reside), I just wonder
1. how many hidden flaws/problems exist in such libraries
2. if it wouldn't be better to NOT use such libs.

Okay, I certainly wouldn't want to write such libs as Hibernate, EHcache or log4j for myself. But the 3-4 liners such as +isWhiteSpace(s:String):boolean and so on - seems like its a good idea to write your own stuff.

denka
Offline
Joined: 2003-07-06
Points: 0

The fact that an issue may be fixed in the next version of a library does not help much if dependencies are considered. The result is, numerous applications that use a dozen of outdated libraries. Neither of these libraries can be easily updated as a result of a need of a full regression of a program. I guess such programs would rot for years before there is a chance to update all dependencies at once.

Interesting point, many of those utility methods and classes become obsolete, too, with new versions of JDK. Take StringBuilder. How many a workaround is stuff in Tomcat that mimics this class' behavior and function? But I'm afraid things like this are unavoidable when people can not restrain themselves from squeezing that last bit of performance (frequently without much profiling, too).

sebastiankirsch
Offline
Joined: 2005-06-24
Points: 0

Okay, it just came to mind that I alreday know of other problems. I just wanted to share them here, for what it's worth...

[b]Log4J Filewatchdog[/b]
This one's rather useful, but: for each observed file, a Thread is started. No, we used tomcat's manager to restart specific contexts instead of shutting down the complete server. Guess wat happens? Right, you now have 2 Threads running, observing the same file. Restart 10 context manually and you have 20 threads - and counting.
Unfortunately, you don't have a chance to access that Thread. So we wrote a FileWatchdog on our own which is destroyed with the context

[b]HttpClient[/b]
Another well-known candidate. I didn't yet look for the code, but I just recently discovered that the MultiThreadedHttpConnectionManager also starts an own Thread. I wouldn't be surprised if that one lives forever also...

greeneyed
Offline
Joined: 2003-06-10
Points: 0

Yeah, sometimes OSS is not the cure for everything and developers "scratching their own itch" works until you have a problem that no developer wants to scratch, and you end up having to understand other people's code just to be able to fix a small thing without breaking other people's code, blah blah.

In my case, I prefer using just the libraries that I really need unless, as others pointed out, they are required by third party libraries (Hibernate, Lucene, Quartz, JasperReports come to my mind...) and even if it means repeating libraries, I usually put them in WEB-INF/lib so I can have various web applications with different incompatible versions.

It's not always easy as preference order of libraries gets in the way when there are libraries bundled with the server (XML parser problems anybody).

Then sometimes people complain you re-invent the wheel, you should use what "everybody" uses... etc. Well, I think you have to find some balance between reuse and "jar-hell".

My 2ec

jwenting
Offline
Joined: 2003-12-02
Points: 0

>
> private static final String[] PADDING = new
> String[Character.MAX_VALUE];
>
> I guess you find this as intriguing as I do. I don't
> want do discuss a solution or best practice here - my
> point is:
> this piece comes from StringUtils, from the
> well-known [b]commons lang[/b] library (v2.1). So, we
> have ~ 30 apps running on one tomcat, which sums up
> to something like 8MB memory usage just for this crap
> (alright, there's the common/lib/ directory - but is
> it save to put that lib in there? probably it is, but
> especially now, I wouldn't count on that).
>
Crap indeed. Especially if it's declared as a GLOBAL...

And no, it's not safe to put in the common lib directory. Too many applications use too many different and mutually incompatible versions of OSS libraries and Jakarta commons is more prone to change radically between even minor versions than most.

> similar "pitfalls"? I wonder how good an idea it is
> using such "all-purpose" libraries if you could also
> set up your own Util classes - providing just the 3,4
> methods you need.

So do I. I have a small set of things I often use don't generally use Jakarta commons (or similar) unless they're required by some 3rd party tool (like Hibernate which uses log4j which uses Jakarta commons, talk about hidden dependencies).
And of course Tomcat itself uses Jakarta commons already and probably relies on a very specific version internally.

> Yeah i know. Who checks my code and now that this is
> uncovered and all is open source, this flaw will
> probably be fixed in the next release. But
> nevertheless...
>
I doubt it will be fixed. Far more interesting to implement something new, and who cares anyway?
It doesn't make anything do a stacktrace that mentions their class so it's not their priority...
Note that this is a general attitude I observe in OSS projects, not necessarilly in this particular one.

> What do you think about this?

Confirms what I decided a long time ago: don't use Jakarta commons if you can avoid it.

alexlamsl
Offline
Joined: 2004-09-02
Points: 0

Use of libraries is about code reuse; it saves you time and effort from reinventing the wheel over and over again.

Since AFAIK libraries are developed by mortals like you and me (do point it out if I am wrong!) mistakes and suboptimal designs are bound to exist. That is where the beauty of code reuse comes in again - one fix to the library and all (relevant) programmers will benefit from it.

Best yet, if it is open source you can be the one to investigate the problem (source code is available), just as you did. In addition, you can even contribute a fix to the problem just like everyone else. :)

What is better than living in a programmers' paradise? ;)

cowwoc
Offline
Joined: 2003-08-24
Points: 0

sebastiankirsch,

Just did a bug search at Apache and found no reference to PADDING. Did you file some sort of bug report for this? I would think this is something they should fix ;)

Gili