Change is for the best when everything changes...

 … Thunderclouds came from the north so we first saw them overhead. Three of us were half way up a high south facing dome which earlier looked like a fun climb, but now looked like a giant lightning rod.  It was time to exit fast by sliding, one at a time, down the rope to ground.
 As the junior climber, I went first.
 Halfway down, as the rain started in earnest my partners tried to yell at me over the rain. Without hearing, I saw their angry expressions and arm waving to mean: "Move it buddy, or else!".   My habitual, safe, slow rappelling was leaving my team exposed above.
The environment had changed but I had not adjusted to the new parameters.  I went faster!

Many Java Applications Merged:

This first post is about an application design change, and an important further optimization that was  discovered sometime later.  The application began as a set of many separate Java processes with a very slick Messaging library for communication among the processes. 

After a couple of years, the hardware and JRE had advanced enough to allow all of the processes to be combined into one.  It made sense: the single merged application started faster, and used less system resources.  The functional speed of the application was better, not the expected speed boost but better; it was time to ship the product and move on.

A couple of releases later during a debug session for an unrelated issue, a javacore showed a callstack where one component was sending a message to another component in the same JVM.  At the top of the callstack, the Messaging library was preparing for a send() by serializing the message.  Code inspection showed that the Messaging library marshalled and demarshalled every message, even when the source and destination were the same JVM.  The same marshall/demarshall also happened on every message response.

The improvement was straightforward and easy:

1. Ensure that message objects were never modified after they were sent.
2. Skip the marshall/demarshall when the source and destination components were in the same JVM.

The benefits were significant and immediately noticeable across every function in the product in throughput and in decreased CPU usage from GC.

During the restructure, the design team had thought about the Message library but only to verify the function: that the library could handle same-JVM messages.  It was easy to miss the idea that the Message library could actually be improved as a result of the overall design change.