Posted by bondolo
on April 16, 2004 at 11:43 AM PDT
Getting out of the evolutionary mud
On April 21st the JXTA Community will turn three years old. The "terrible twos" weren't actually too terrible and the future of JXTA looks really, really exciting. The JXTA project has gone through a lot of changes over the last three years and it's been really interesting to see how the technology has evolved. The coolest part is, of course, that the path evolution has taken has been entirely unpredictable. I've decided to devote my first couple of blogs to some of the issues we've encountered in developing JXTA and perhaps there might be some insights for developing evolving technologies.
The Cost of Thoughtless Design
Imagine that you are late in a software release cycle. Users are clamouring for the new release yet bugs remain. A tricky problem surfaces involving communications between two modules. The apparent answer--a new public method is added to one of the modules interfaces. Success and the release ships to triumphant acclaim!
Skip to six months (or five years) later... The tricky problem found long ago was merely an indicator of a weakness in the overall design and it was necessary to refactor several modules to evolve the architecture to correct the design. The refactoring included a number of API changes. Well "changes" is the wrong word. One simply can't change a public API. Users are depending on the API remaining the same from release to release. To ensure adoption of new releases backwards compatibility must be provided. So the responsible developer is left supporting three sets of APIS:
- The original API - Needed for backwards compatibility. You can deprecate the methods/classes and hope that users pay attention and cease using the original API over time.
- The "warts" API - These are the APIs you added at the last minute. Like the original APIs you need to support these for backwards compatibility.
- The new API - Hopefully all of your users will convert to this API the moment it's released. More likely, they will use a mix of the original API and the new API.... hmmm, is that supported? Was the new API intended to be used only with itself? Oh and we also need to add this additional method to fix this really gnarly problem that was just discovered by an early adopter. Hmmm, isn't this new method another "wart" just like the one we had to the old API?
It doesn't take long before the situation get really complicated. The heart of the difficulty becomes supporting APIs, possibly several generations including "warts" rather than supporting functionality. It's a dangerous trap to fall into especially for an evolving technology. If you spend all of your time working on the APIs you aren't working on the functionality. And with all of these APIs users get increasingly confused as to how to use the product. Lastly, just because an API is unchanged does not mean the behaviour is unchanged. This can turn out to be even more of a problem than simply changing an API.
Maybe the answer is to do a rewrite, "2.0". It's tempting and it will of course resolve all of the problems with the current API by completely ignoring it. However, there's always "The Second System Effect" [The Mythical Man Month, Fredrick P. Brooks] to worry about The users also won't be thrilled to have to rewrite their code to use the "2.0" release, there's the risk that the user base will fork between the "1.0" and "2.0" release. The user base of the product is growing though and new users will use only the new APIs. Maybe that's sufficient.
The approach which has been taken by the JXTA team has generally been to try to avoid breaking existing APIs while moving the technology forward in an evolutionary way. Even though the JXTA 2.0 release was protocol incompatible with the JXTA 1.0 release, we did maintain API compatibility and behaviour as much as possible.
In the last 18 months we've been trying to evolve, deprecate and remove old APIs. This is in contrast to JDK which though it has deprecated a number of APIs has, to my knowledge, never removed any of them. Some of the JDK deprecated APIs seem so broken it's hard to imagine anyone depending upon them.
One thing that JXTA did right in the beginning was to separate the public API from the implementation. Early on were equally cautious in making changes to the API of either part. As time has progressed we have been less rigorous about maintaining any form of backwards compatibility in the implementation interfaces. This has had positive effects. A truly gross hack which affects only the implementation and can be refactored away in a future release always beats an elegant hack in the public API. The advantage being that by keeping the hack private you free yourself from maintaining it indefinitely.
I've spoken so far mostly about APIs, but similar thoughts apply to the protocols as well. For protocols the key to reducing future pain seems to be in underspecifying the permitted values of elements and to explicitly specify handling for uninterpreted elements.
JXTA continues to evolve. If JXTA was a biological system I'd estimate we've gone from being a flatworm in 2001 when JXTA was first released to being an a wily newt in 2004. JXTA is still climbing up from the evolutionary mud. While it's still a long way from a being as sophisticated as a primate or even a mammal there's been a lot of progress. It took nature many millions of years to acheive as much progress. Being as evolved as a "newt" isn't too bad either! It's certainly a lot more interesting, mobile, capable and adaptable than it's mud sucking flatworm ancestor.
Here's hoping that the next three years moves JXTA a few more branches up the evolutionary tree!