RfC: Improving Mavens Performance

I am typically working in projects that are relatively complex, like one parent projects and 20 modules, or so. To handle the complexity, I have learned to use and appreciate Maven. OTOH, after 8 years or so with Maven, I am still missing some aspects of Ant builds, in particular the speed. Maven does a good job when it comes to understand Build scripts (biggest problem of Ant), but it can be painfully slow. Why is that? I could name several reason, but the most obvious seems to be that Maven is always building the whole project, whereas Ant allows to implement logic like

   if (module.isUpToDate()) {
     // Build it
   } else {
     // Ignore it

Of course, Ant's syntax is completely different, but that's not the point, unless you are a fanatic XML hater and really believe that a Groovy or JSON syntax is faster by definition (If so, stop reading, you picked up the wrong posting!)
The absence of such an uptodate check isn't necessarily a problem. Most Maven plugins are nowadays implementing an uptodate check for themselves. OTOH, if every plugin does an uptodate check and the module is possibly made up of other modules itself, then it sums up.
Apart from that, uptodate checks can be unnecessarily slow. Suggest the following situation, which I have quite frequently:
A module contains an XML schema. JAXB is used to create Java classes from the schema If the schema is complex, then the module might easily have severeal thousand Java source files.
This means, that the Compiler plugin needs to check the timestamps of several thousand Java and .class files, before it can detect that it is uptodate. Likewise, the Jar Plugin will check the same thousands of .class files and compare it against the jar file, before building it.
That's sad, because we could have a very easy and quick uptodate check by comparing the time stamps of the XML schema, and the pom file (it does affect the build, does it) with that of the jar file. If we notice that the jar file is uptodate with regard to the other two, then we might ignore the module at all: Ignore it would mean to completely remove it from the reactor and not invoke the Compiler or Jar plugins at all. Okay, that would help, but how do we achieve that without breaking the complete logic of Maven? Well, here's my proposal:

1. Introduce a new lifecycle phase into Maven, which comes before everything else. (Let's call it "init". In other words, a typical Maven lifecycle would be "init, validate, compile, test, package, integration-test, verify, install, deploy" (see this document, if you need to learn about these phases.
2. Create a new project property called "uptodate" with a default value of false (upwards compatibility).
3. Create a new Maven plugin called "maven-init-plugin" with a configuration like

      groupid: org.apache.maven.plugins
           artifactId: artifactid>="maven-init-plugin"
           configuration:
              sourceResources:
                sourceResource:
                  directory: src/main/schema
                  includes:
                    include: **/*.xsd
                sourceResource:
                  directory: .
                  includes:
                    include: pom.xml
              targetResources: ${project.build.directory}
                  includes:
                    include: *.jar
       (Excuse the crude syntax, I have no idea how to dixplay XML on blogspot.com!
        I hope, you do get the idea, though.)
       The plugins purpose would be to perform an uptodate check by comparing source-
       and target resources and set th "uptodate" flag accordingly.
     

Modify the Maven core as follows: After the "init" phase, search for modules with isUptodate() == true and remove those modules from the reactor. Then run the other lifecycle phases.

That's it. Perfectly upwards compatible. Moderate changes. Much faster builds. How about that?

Maven and property files

After so many years (since 2004, indeed when the first version of Maven 2 was still in development), I am still learning new stuff every day. For example, so far I was always specifying properties in my POM file. But you can use external property files! There is a Maven Properties Plugin over at Mojo with a goal "properties:read-project-properties".

Maven is groovy!

Recently, I had another one of those cases where Maven almost does the right thing, but not quite. Let me explain the use case:
I've got a software component that can initialize the database from an SQL script. Such an SQL script (in what follows: The DDL, or data definition language script) is ideally generated by the Hibernate Schema Exporter, aka "hbm2ddl", which in turn is available in Maven by running the Hibernate3 Maven Plugin. But, if just creating the database is not sufficient and you need to run a second SQL script (in what follows: The data script) to populate the DB with some initial entries? Well, I came up with the following solution:

1. At build time, have Maven create the DDL script (below target/classes, so that it is available at run time)
2. At development time, manually create the data script (in src/main/db)
3. At build time, have Maven concatenate these scripts into a third SQL script (in what follows: The concatenated script, also below target/classes, as it must also be available at runtime)

Question: How do we do that last step? The most obvious solution was the Maven Antrun Plugin, Ant even's got a "concat" task, which should do exactly what I want (Including uptodate checks). However, I wasn't really happy with that solution, because Ant, or the "concat" task behaved too unpredictable (For example, no error was produced, if either of the source files didn't exist. An, error checking is, where Ant scripts become really nasty.) In the end, I had to admit: It didn't work.
So I came up with another idea: Why not have a small Groovy Script in the Maven POM. And, as is usually the case, someone else already had that idea and there is a Maven Plugin, which already provides just that:
I can embed a Groovy snippet into my Maven POM and have it executed at a suitable point of my build script. Here's the snippet I came up with:

<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>gmaven-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>execute</goal>
</goals>
<configuration>
<source><![CDATA[
def concat(s1, s2, t) {
def java.io.File f1 = new java.io.File(s1)
def java.io.File f2 = new java.io.File(s2)
def java.io.File ft = new java.io.File(t)
def long l1 = f1.lastModified()
def long l2 = f2.lastModified()
def long lt = ft.lastModified()
if (l1 == 0) {
throw new IllegalStateException("Source file must exist:" + f1);
} else if (l2 == 0) {
throw new IllegalStateException("Source file must exist:" + f2);
} else if (lt == 0 || l1 > lt || l2 > lt) {
java.io.File pd = ft.getParentFile()
if (pd != null && !pd.isDirectory() && !pd.mkdirs()) {
throw new IOException("Unable to create parent directory: " + pd)
}
println("Creating target file: " + ft)
println("Source1 = " + f1)
println("Source2 = " + f2)
java.io.FileInputStream fi1 = new java.io.FileInputStream(f1)
java.io.FileInputStream fi2 = new java.io.FileInputStream(f2)
ft.append(fi1)
ft.append(fi2)
fi1.close()
fi2.close()
} else {
println("Target file is uptodate: " + ft)
println("Source1 = " + f1)
println("Source2 = " + f2)
}
}
concat("target/classes/com/softwareag/de/s/framework/demo/db/derby/initZero.sql",
"src/main/db/init0.sql",
"target/classes/com/softwareag/de/s/framework/demo/db/hsqldb/init0.sql")
concat("target/classes/com/softwareag/de/s/framework/demo/db/derby/initZero.sql",
"src/main/db/init0.sql",
"target/classes/com/softwareag/de/s/framework/demo/db/hsqldb/init0.sql")
]]></source>
</configuration>
</execution>
</executions>
</plugin>

perhaps in combination with a byte array, for performance reasons, but in Groovy a file has got a method append(InputStream), which does exactly that. And, although I am declaring the variable ft above as an instance of java.io.File, it is nevertheless a Groovy file, with all the added sugar of Groovy! Which is, why embedding Groovy into the POM is much nicer than embedding Java!

In the future. I will most likely never ever write Maven plugins and use Groovy scripts instead.

 Second: We are inside a Maven POM, or, to put it different: Inside an XML file. As a consequence, I've got to be careful with characters like '&', or '!'. Which is why I am using the strings ">" and "&" instead. I might as well use a CDATA section, or, even better: An external script (in src/main/groovy) However, I believed to make this postings point better with an internal (albeit somewhat lengthy) snippet. Hope, you agree, so let's be groovy!

When dog food isn't good enough

Recently a proposal came up on the Apache infrastructure list (Sorry, AFAIK, the list isn't archived, thus no pointers from here.) to install Nexus Professional on one of the Apache servers as a repository server. The idea was, in particular, to use Nexus' staging facilities for pushing Apache software releases.

The proposal was posted by Brian Fox, with support by Jason van Zyl. As you can see from my links, both are employed by Sonatype, the company producing Nexus. (Jason is founder and CTO). Obviously, the ASF won't be a bad reference for them. The proposal seems to be rapidly acceptet, almost without objection. (There was some discussion, but mostly about technical details or Maven at all.) As a result, you can view the installed Nexus live today, which is what I did.

In all honesty, I'm going to like it. Having had my share of release management and the related trouble, this is going to help: Pushing some 30 or 40 files to people.apache.org (the number seems big, but you have to consider various signatures, like .md5 and/or .sha1 files as well as GPG signatures, aka .asc files) via SSH to a common place and later distributing it manually to various places is error-prone. But that is like it is: Placement in the common place (typically a users public_html directory) allows the review and vote by fellow developers. Once the release is accepted, the files are being moved to the final target locations. Nexus can help with that and the UI is, of course, very neat.

But that is not the reason for todays entry. What strikes me as rather odd is the fact, how easy a commercial product can make its way into the Apache server park. This is not the first such product: Apache is hosting a Jira server for issue tracking as well as a Confluence Wiki. I observed the discussions when these have been introduced. Jira is basically the successor to the Apache Bugzilla (at least more and more Apache projects leave Bugzilla in favour of Jira) just as well as Confluence is quickly replacing the Apache MoinMoin wiki. In both cases the question was raised, whether an open source product wouldn't be preferrable. Ideally, whether there couldn't be an Apache project to use: "Eat your own dog food" has some tradition within the ASF. The Apache web servers are frequently relatively stable development versions. In both cases there have been no such projects, and the open source alternatives had their share of problems. So I understood the decision.

Which is what I don't do in the case of Nexus as an Apache repository server. There is Archiva, an Apache project, which could do the job. Ok, it doesn't have the bells and whistles, but it does its job. I can tell, because I am using it in my daily work. It is a mature project in active development, obviously also sponsored by commercial companies. Ok, it can't support staging right now, but that wouldn't be overly difficult (Brett Porter has offered to add it, should the ASF require) and within a reasonable time frame. Should be enough at least to consider it as decent dog food.

Alas, noone seemed to be interested in the Nexus discussion. So its Nexus. I can live with it. Understanding is a different matter.