App-Managed JDBC DataSources with commons-dbcp

In enterprise Java applications, it's considered poor form to instantiate database connections yourself. The enlightened path involves a database connection pool, in the form of a JDBC DataSource, that is provided by your container (Tomcat, Jetty, WebLogic). The container is responsible for maintaining a number of connections to the database service, and all your code has to do is look up the DataSource via JNDI to get a connection. Simple beauty.

Sometimes, though, you want the efficiency and centralization of a pool but the control of managing it yourself. Consider the following scenarios:

• Your app runs in a managed environment and you don't have access to the container's admin console. This is not uncommon in the corporate enterprise, or with some web hosts.
• You want a container-neutral web app that can be deployed by third parties with minimal intervention on your part. Do you really want to provide DataSource how-to guides for all of your customers' containers?
• Your app runs in a special environment, such as a demo CD-ROM, that requires it to be even more self-contained than a standard web app.

Whatever the case, there are ways to have your cake and eat it, too. In this article, I'll share a technique that uses Apache's commons-dbcp [2] (and, by way of dependency, commons-pool [3]) to move the DataSource into your web application. Managing a DataSource yourself requires a some knowledge of object pooling and JNDI, so I'll explain those along the way.

About the Sample App

I have provided a sample web application to help explain the technique - see the resources section for a link to the sample code. Two servlet plus JSP page controller [4]s are the front ends of simple data input and retrieval operations. The servlets use a central object registry (class ObjectRegistry) to look up a Data Access Object [5] (DAO, SimpleDAO). In turn, the DAO is used to move data into and out of the database. The DAO fetches database connections from a DataSource that's bound to JNDI.

Since the DataSource is not provided by the container, it's up to the web app to initialize it before any code tries to use a database connection. A ServletContextListener [6] is a suitable candidate for the task. Implementations of this interface, introduced in the 2.3 servlet spec, are registered in the web app's web.xml deployment descriptor. The container calls each registered listener's contextInitialized() method when its parent web app is deployed, and calls contextDestroyed() when the web app is removed from service.

Much of this article focuses on a single such listener, SetupDataSourceContextListener. This class uses commons-dbcp and commons-pool to create a DataSource and bind it to JNDI.

The sample web app was tested under Tomcat [7] 5.5 and Jetty [8] 5.1. It uses an embedded, in-memory HSQLDB [9] database to make it completely self-contained and ready for testing in your environment.

Pooling Theory

Most DataSources are backed by pools of database connections, so it's tough to describe the former without a brief look at the latter. Pooling is a way to manage resources that are expensive to create and in constant demand, but used only for short periods at a time. Most Java developers are familiar with pools of database connections, or even pools of threads. In languages that don't provide garbage collection, such as C and C++, it's not uncommon to use pools of raw memory.

The premise behind pooling is rather simple: instead of directly instantiating a resource--calling new--code borrows a resource from some central repository. Client code briefly uses the resource and returns it to the repository. Object pooling is not unlike using a rental car: the client code doesn't know the resource's origin, nor what happens when it returns it, nor whether it will get the same object each time it asks to borrow one.

The repository is responsible for the resource objects' lifecycle. It instantiates some number of resource objects and makes them available for borrowing, and later cleans them up. The pool is exhausted when all resources are loaned out. What happens when code requests a resource from an exhausted pool is a pool implementation detail: the pool's borrowing operation can block (not return) until a resource is ready, return null, or throw an exception.

Object Pooling with commons-pool

commons-dbcp uses commons-pool, Apache's generic pooling framework, as its pool back end. ObjectPool is the core of commons-pool. This interface represents a class that manages an underlying object collection that comprises the pool. It has methods to borrow objects (borrowObject()) and return them (returnObject()), and to query the pool state (getNumActive(), getNumIdle()):

public interface ObjectPool {

   Object borrowObject() ;
   void returnObject(Object obj) ;

   int getNumActive() ;
   int getNumIdle() ;

   void invalidateObject(Object obj) ;

   void clear() ;
   void close() ;

   void addObject() ;
   void setFactory(PoolableObjectFactory factory) ;
}

Two other methods, addObject() and setFactory(), merit further discussion. An ObjectPool has only indirect knowledge of the objects it manages. The setFactory() method specifies the factory that the ObjectPool will use to create those objects. In turn, addObject() tells the ObjectPool to call on its factory to create an object and add it to the underlying pool.

The sample code uses the GenericObjectPool implementation of ObjectPool (other implementations, such as StackObjectPool and SoftReferenceObjectPool, offer features above and beyond basic pooling). This class offers several methods beyond its base interface. Notably, setWhenExhaustedAction() specifies what borrowObject() does if the pool is exhausted:

GenericObjectPool pool = new GenericObjectPool(null);
pool.setWhenExhaustedAction(
   GenericObjectPool.WHEN_EXHAUSTED_BLOCK
) ;

The pool is initially empty. It's up to client code to call addObject() a few times to populate the pool with objects from the backing factory. Here, the pool created in SetupDataSourceContextListener will have ten objects:

for (int ix = 0 ; ix < 10 ; ++ix ) {
   pool.addObject() ;
}

It's good form to clean up whatever you create. The container calls SetupDataSourceContextListener's contextDestroyed() when the web app shuts down. In turn, this method calls the pool's close() method to free its resources:

if( null != _pool ){
   _pool.close() ;
}

commons-pool is only half of the story. Mix it with commons-dbcp to create a connection pool.

commons-dbcp at a High Level

commons-dbcp is Apache's database connection pooling framework. As you'll soon see, it works with pretty much any JDBC-compliant database driver. If it's supported by the classic DriverManager, you can plug it into commons-dbcp.

commons-dbcp uses a multilayered concept. Client code only sees the DataSource object--here, the PoolingDataSource--on which it calls getConnection() to fetch a database connection. A lot more happens behind the scenes.

The PoolingDataSource's getConnection() calls its ObjectPool's borrowObject() to fetch a JDBC Connection. This returns a plain Object, so getConnection() casts the result:

...
public Connection getConnection(){
   ...
   Connection result = (Connection) _pool.borrowObject() ;
   return( result ) ;
}

In turn, the ObjectPool returns an object created by is backing factory, here a PoolableConnectionFactory. Remember, the ObjectPool's addObject() called this factory to populate the pool with objects.

To clarify, PoolableConnectionFactory is a factory of PoolableConnection objects. These objects implement the JDBC Connection interface, so they look and act just like any other database connection. They even wrap other Connection objects. The twist is that PoolableConnection intercepts calls to the close() method: instead of terminating the network connection to the database service, it calls the parent pool's returnObject() to return the wrapped Connection to the pool.

public class PoolableConnection {

   ...
   
   // PoolableConnection delegates most methods
   // (such as createStatement()) to this object:
   private Connection _realConnection ;
   
   // the pool from whence this PoolableConnection
   // object came:
   private ObjectPool _parentPool ;
   
   ...
   
   // don't really close it; put it back in the
   // pool so someone else can use it
   void close(){
        _parentPool.returnObject( this ) ;
   }

}

Digging deeper, PoolableConnectionFactory doesn't create those wrapped JDBC Connections itself. It proxies yet another factory, in this case a DriverManagerConnectionFactory. In turn, DriverManagerConnectionFactory wraps calls to the ever-familiar DriverManager.getConnection() to create a database connection.

Your Very Own DataSource: Coding the commons-dbcp API

The commons-dbcp API makes coding this nested matryoshka doll [10] easier than describing it. First, create the underlying factory to furnish Connection objects:

ConnectionFactory factory = new DriverManagerConnectionFactory(
   ... JDBC URL ... ,
   ... login ... ,
   ... password ...
) ;

Next, back the pool with a PoolableConnectionFactory:

PoolableConnectionFactory pcf =
        new PoolableConnectionFactory(

   factory , // ConnectionFactory
   _pool ,   // ObjectPool
   null ,    // KeyedObjectPoolFactory
   null ,    // String (validation query)
   false ,   // boolean (default to read-only?)
   true      // boolean (default to auto-commit?)
);

Don't be fooled by the call to create pcf. It looks like a no-op statement, but the variable pcf assigns itself as the pool's factory. That means the ObjectPool's addObject() calls pcf.makeObject().

Finally, set the pool as the back-end Connection provider for the DataSource.

PoolingDataSource result = new PoolingDataSource();
result.setPool( _pool ) ;

For all of this to work, however, you're missing one last piece: the JAR file(s) with your database drivers. Since these are no longer managed by the container, you must include these JARs in your web app's WEB-INF/lib directory along with your other dependencies.

Binding It to JNDI

Code must be able to find the DataSource in order to use it. One option is to make your shiny new DataSource available via JNDI, where it will look exactly like one that's provided by the container (at least, to client code). Switching from a container-managed to an app-managed DataSource would require changes to just the JNDI lookup string used by client code. And if you've followed best practices, this should be only one place!

JNDI is a directory service, often used in Java as an object store: one piece of code instantiates an object and binds (stores) it under some key; another fetches the object based on that key. Services such as JNDI promote loose coupling because the object creator and user are not directly related; they only have to know of JNDI and the lookup key.

JNDI works on a tree model, not unlike a filesystem. Objects are bound to (located at) contexts (paths). The elements of the context--that is, the branches along the tree--are called subcontexts. One string representation of a context looks like a Unix filename, complete with slashes (/) to separate the subcontext elements.

SetupDataSourceContextListener's bindObject() method takes an object to bind and a string that represents the context. It first breaks down the string and builds the subcontexts as needed:

Context currentContext = new InitialContext() ;
final String name = currentContext.composeName(
   fullPath ,
   currentContext.getNameInNamespace()
) ;

// get the subcontext pieces
final String[] components = name.split( "/" ) ;

for( /* elements of "components" */ ){
   
   final String nextPath = /* current element */

   try{

      // lookup the subcontext
      currentContext = (Context)
         currentContext.lookup( nextPath ) ;

   }catch( final NameNotFoundException ignored ){

      // not found; create subcontext
      currentContext = currentContext
         .createSubcontext( nextPath ) ;

   }

}

subContext.bind( name , objectToBind ) ;

Not all containers provide writable JNDI trees. Others forbid binding to certain context paths, such as the standard java:comp/env. You'll know about any problems right away: the ContextListener will throw an exception if it is unable to bind. If yours fits into this category, you'll have to make the DataSource available some other way, either via dependency injection (a la Spring) or a custom app-wide object registry (similar to the sample code's ObjectRegistry class).

Frankly, if your app already has its own object registry, JNDI is redundant and otherwise of limited use. JNDI is more suitable for loose coupling on a larger scale, such as between applications.

But Is It for Me?

There are few, if any, one-size-fits-all solutions. Consider the following when choosing between app-managed and container-managed DataSources.

First of all, commons-dbcp and commons-pool have very wide usage. For example, Tomcat's container-managed database connectivity is backed by code similar to that used in the sample app. Tomcat just wraps it in a neat XML config file. There are occasional complaints that the format is too complex and error-prone, but that's another story.

If you want to build a container-neutral app, this technique puts you one step closer to that goal. You can combine a commons-dbcp DataSource with an in-memory database such as HSQLDB to deliver demo apps that run self-contained, out of the box, even on read-only media.

Perhaps you run your web apps via a third-party hosting service. App-managed database pooling gives you runtime control of the pool size and configuration--for example, you can drop and recreate pools while the app is still live. In a shared environment, hosting providers (understandably) cannot always grant customers access to the container's admin control panel.

Then again, container-provided DataSources exist for a reason. The loose coupling of web apps and database connectivity centralizes administration. Some containers offer a GUI control panel, from which administrators can keep an eye on managed services. App-managed database pools are invisible to these tools.

Furthermore, you can't use app-managed database pooling to back container-managed services, such as Tomcat's security realms. This is because the DataSource exists in the web app's classloader, making it invisible to the container's classloader. That said, there's nothing to stop you from using both: app-managed connection pools for your business code, and container-managed pools for container-managed security.

Creating the DataSource within your app requires more effort on your part. You have to manage and clean up any associated resources (typically, via a ServletContextListener). You also have to rebuild and deploy your app if you change database drivers, since the JARs are packaged inside of WEB-INF/lib. Finally, it's up to you to create your own management interface from which to control the pool at runtime. Either that, or you must rebuild and redeploy the web app whenever you want to change the number of connections or reset the pool.

Conclusion

Moving database connectivity from the container into your web app addresses some compatibility problems and addresses niche issues, such as creating demo applications. It's certainly not appropriate for all situations; but you now have the theory and practice to apply it when the need arises.

I'd be happy to hear your success stories of trying this technique with other containers. Note that the sample app requires a servlet spec 2.4 container, such as Tomcat 5, Jetty 5, or WebLogic 9. It should work in a 2.3 spec container with a few tweaks to the JSPs.

Resources

• Sample code [11] for this article.
• The websites for commons-dbcp [2] and commons-pool [3] include examples and API documentation.
• The pooling pattern has been around for a while, and in several contexts. Pattern-Oriented Software Architecture, Patterns for Resource Management [12], by Michael Kircher and Prashant Jain, describes it in detail.

Ethan McCallum [1] is a freelance technology consultant.

Related Topics >> Databases [13]   |