Currently I’m working on a major technical upgrade, that involves the upgrade of several infrastructure components including WebSphere Application Server, WebSphere Portal Server and JBoss EAP. This requires all a “fresh” installation of all the applications that reside on these components. All JEE containers currently in the system are being upgraded, so it opens a window of opportunity to get a huge overhaul of the system.

(NB. As I have to rebuild the entire application stack multiple times a day, I’ve been referenced as Lord Builder or Darth Builder by my highly esteemed colleagues)

Current situation

Shared library model

At the time of creation of the first modules (I’ll explain what “module” is meant here later) it seemed like a great idea to have the common components installed as shared libraries and have them referenced directly by all modules that require it. This eliminated the need of deploying several different version of Hibernate, POI etc. Unfortunately it seemed to be a great idea at the time to have the EJB (version 2.1) clients as shared libraries as well, and went as far as offering the deployed EAR-s as a shared library automatically. The applications using these modules remotely place the EJB clients in a shared library and use them from there.

This mode of operation, along with a bunch of less-than-disciplined developers, resulted in a number of applications directly referencing each-other. With respect to the 50+ modules employed currently, I can not cease to wonder why this model never crumbled on top of our heads.

Along with the infrastructure update we have decided to put an end to it’s misery.

Maven dependency resolution

The continuous integration and release system is using Maven. The switch to Maven from Ant occurred well into the operation of the system. The migration was thorough and methodological, and was maintained quite well, resulting in a homogeneous POM structure, and working continuous integration.

As a direct result of the shared library model the structure of these POM-s require most library dependencies set as provided (meaning I need the library for compiling and testing the code, but I trust the environment will have it when I’m deployed).

The problem with provided dependency scope is, that it will break the dependency chain. There is no way to do proper dependency resolution, as the libraries required by the referenced artifact are taken as granted and are not deployed. In order to even compile these modules you have to list all referenced components one-by-one. You will loose the flexibility of the dependency resolution provided by Maven.

Migration targets

As the migration involves a major jump in JBoss version, the JBoss modules have to be changed in order to work with the new JacOrb implementation. The system archives (SAR) can no longer be used as shared libraries, and since the EAR-s are in separated classloaders their modules must be packaged as a self-contained EAR.

To achieve this, all the referenced artifacts (except for the ones used by the containers themselves) must be migrated from provided to compile scope. Since these artifacts are also used in the WebSphere counterparts, and are used remotely in the WebSphere portal components those must also be changed to get rid of the shared libraries.

As the applications will contain the referenced libraries required for remote calls, there will be no need to restart the servers after deployment. This can reduce downtime and release windows can be used more effectively.

Module layout

We use the term of module for a set of packages that provide a set of services related to a specific business task. The ideal layout of a module is on the diagram to the right. The packages are implemented as Maven modules and are separate artifacts.

Most modules contain of a Primitive package managing the back-end communication or the database access. This uses a Definition package as data transfer object.

The Business package contains the business logic to be applied on the data acquired through the Primitive package.

The Facade package is used to provide access to the functions of the business and sometimes directly to the Primitive package.

External applications access the module through the Facade client using the Definition package as the data type.

Some helper functions, such as data converters or formatters can reside in the Utility package to manipulate Definition objects. These can also be used externally.

Problems faced on migration

Dependency creep

The years of using shared libraries resulted in a number of applications violating the model above. The modules on the same server sometimes directly reference other module’s Business package or their Utility package reference their Business or even Primitive package.

When we turn these references from provided to compile scope in Maven, we sometimes end-up with anomalies like Hibernate libraries on the display-only portal applications. There is no way of automatically deciding weather it’s by design or mistake, so each application must be revised and changed accordingly.

Virtual dependencies

Quite a few of these dependencies do not really exist, the stressed developers sometimes just dropped a few dependencies about (see the Wonders of Cut and Paste Programming vol.2), as it had no impact on the resulting artifact (the deployed package) These references can safely be removed the package still builds but the dependency tree cleans up nice.

Classes in the wrong package

Sometimes Primitive or Business packages contain constant definitions, or generic methods that belong to the Definition or Utility package respectively. Earlier these packages were referenced without consequences. Even though most methods could never be called, on account of their referenced libraries not being there to start with, the artifacts were included in the deployments.

Now the compile scope will take all these dependencies and package them in the deployment archive.

Cleaning up this mess invokes a simple refactor to the correct package and straightening out the dependencies.

Utility referencing Business/Primitive directly

The most difficult problem is when the Utility class actually calls the Primitive method. For example I’ve always wondered why is there a data source defined on the Portal for a specific function, that should be in the primitive layer. It turned out, that actually it used that data source to run the method from the primitive it was packaging along!

In this case we can cut the dependency to the implementation modules by using the Facade client instead of the Utility function. Unfortunately as the Facade invocation implies heavy logging, it must not be used internally from the Utility through the Facade client. So if a Utility function is using the functions of Business/Primitive packages, it must first be moved there, the function must be added to the Facade/Facade client  and references to this Utility function must be modified to the Facade client in remote invocations and Business/Primitive calls in local invocations accordingly.

Deploy tool changes

There are two deploy tools written specifically for our environment.

The one for JBoss is quite “simple”, it deploys an application along with it’s configuration to a JBoss instance. I’m not very informed about it, as I mainly work on the WebSphere issues, but since the new application server changed so much, the entire JBoss deployment tool must be rewritten from scratch.

On WebSphere however we have a tool that utilizes the wsadmin interface. This tool is working between releases thanks to the legacy support in the WebSphere for the JACL interpreter that’s been already obsolete two releases ago. The tool however needs to change as well. Currently all deployed applications consist of Client and Application modules, where the Application consists of the module itself, and the Client is the Facade client and it’s corresponding Definition wrapped into a synthetic package created by an in-house developed Maven plug-in. It also contains the RMIC stub and tie classes previously required for remote method invocation.

Deployment takes place on the Application Server using a JACL descriptor, that creates the required resources such as JMS queues, JDBC resouces etc. and deploying the Application. After deployment it shares the application automatically as a shared library, so other applications can reference it. It also sets up shared library references for the application deployed set in the JACL to the other Applications/shared libraries. It can also create “named” shared libraries and reference them. It also has an ability to share itself with other already existing applications, which was required some time ago, but hopefully never used. The PARENT_FIRST class loading required by the shared libraries can be changed back to the properly separated PARENT_LAST class loading.

All these can be removed from the JACL script effectively eliminating most of it’s bulk.

Currently the Client libraries are copied to the defined nodes through SSH. As WebSphere only reads shared library contents upon startup, the respective servers must be restarted after copy, as they won’t be able to otherwise notified of the changes. After migration this will not be an issue, as there will be no client libs to copy.

As there will be no need to copy these anymore, the respective code can be removed as well from the deployment tool.

What’s next?

The actual migration will take place in a two months. Over 100 modules must be revised and fixed by then. The bulk of the changes are already done using mass pom.xml updates, and the modules on WebSphere Applcation Server and Portal are already able to be built, packaged and deployed to the server.

• The module dependencies are to be revised and straightened manually
• Our central “Utility” modules that provide basic services to the module structure defined above are packaged everywhere
  • They must be revised to remove unnecessary code
  • Calls from WebSphere to JBoss are currently an issue, and need to be tested
• WebSphere Portal 8 compatible theme should be installed, as the one currently in use hacks the deployed server.
• Applications need to be tested to see if there are any regressions
• Deployment tool must be created for JBoss
• WebSphere deployment tool must be updated to eliminate the possibility of the shared lib use

There are several risks associated with this move, still it must be taken. There are plans B, C, and D reducing the scope to minimum, but those require compromises we should not take. The single biggest problem is, that I see no way of operating the shared library approach along with the non-shared library approach. The classes from the shared libraries conflict with the self-contained EARs. This might be solved by the class loading policy, but there was not enough time for experimentation.

As a result of this overhaul, the functions in our common Utility designed for EJB2 can be gradually retired, to provide a more reliable EJB3 based approach.

The modified Maven dependencies allow for pom.xml-s with consumable size. Before most pom.xml-s contained several non-direct dependency references, spanning several screens. After the change most pom.xml-s fit on a single screen, and only the real dependencies are included.