Sunday, August 28, 2011

The mess that is m2e connectors

  • Warning: The following is most likely stubborn, unreasonable,one-sided, and ignores a lot of facts, of which I am unaware.
I am an Eclipse user since approximately 10 years. I am also a user of Maven 2 since may be 4-5 years. As a consequence I am also using the former M2Eclipse (an Eclipse Plugin for using Maven as the Eclipse Builder) since its early days. While I am quite happy with Eclipse and Maven, the experience with M2Eclipse has never been free from annoyance and fights. To be honest, it has always been hard to convince my colleagues to use it. And I fear that battle is lost now. I wasn't the one loosing it: The decisive strike was administrated by M2E itself...

M2Eclipse has recently been moved to the Eclipse Foundation. It is now called M2E, lives at eclipse.org and can be installed from within Eclipse Indigo as a standard plugin, just like CDT, or WTP, which is, of course, a good thing.

So, when Indigo was published (together with M2E 1.0 as a simultaneous release), I rushed to load it down in the hope of a better user experience. But the first thing I noted was: M2E was showing errors in practically every POM I have ever written, and there are quite a few of them, including those of several Apache projects and those at work. So,as a first warning:

M2E 1.0 is incompatible with its predecessors. If you want to carry on using it without problems, don't upgrade to Indigo, or try using an older version of M2Eclipse with it (I haven't tried, whether that works. The reason for this intentional incompatibility (!) are the so-calles M2E connectors, which, I am sure, have driven a lot of people to madness since their invention. In what follows I'll try to outline my understanding of what the connectors are and why I do consider them a real, bloody mess.

I am still not completely sure, what problem the connectors ought to solve, but from my past experiences I guess something like this:

M2E allows you to run Maven manually. You can invoke a goal like "mvn install" from within Eclipse just as you would do it from the command line. That works (and always worked) just fine. Unfortunately, Maven is also invoked automagically from M2E whenever Eclipse builds the project, for example after a clean. In such cases M2E acts as an "Eclipse Builder". It is these latter invocations that people have always had problems with and that the connectors should handle better. First of all, what are these problems?

  1. Builders can be invoked quite frequently. If automatic builds are enabled and you are saving after every 10 keys pressed, the builders can be invoked every 20 seconds or so.
  2. The UI is mainly locked while a builder is running. In conjunction with the frequent invocation that means that the UI can be locked 80% of the time, which a human developer considers extremely painful, in particular, if the builder invokes Maven, which can take quite some time.
  3. Some Maven (I am unaware of any in reality, but the M2E developers are mentioning this quite frequently)plugins assume that they are invoked from the command line. That means, in particular, that System.exit is called once Maven is done. Consequently, they consider use of resources as unproblematic: They acquire a lot, including memory and don't release it properly. The resources are released automatically by System.exit. But that doesn't work in M2E which runs as long as Eclipse does (meaning the whole day for Joe Average Developer) and invokes Maven (and the plugin with it) again and again.
  4. M2E doesn't know whether a plugin (or, more precisely, a plugins goal) should run as part of an automatic build. For example, source and resource generators typicallyshould, artifact generetors typically should not. Consequently, a lot of unnnecessary plugins are invoked by the automatic build, slowing bdown the builder even more, while necessary goals are not. This is not what people expect and leads to invalid behaviour on the side of the developer. For exsmple, I keep telling my colleagues again and again that they shouldd invoke Maven manually, if the test suite depends on a generated property file.
But how should connectors fix this: I am partially speculating here, but my impression is this: When Maven is invoked as a builder, then it is modified by M2E to no longer invoke plugins directly. Instead, Maven invokes the plugins connector, which in turn invokes the plugin. The connector ought to know the plugin and decide, whether a goal must be invoked as part of the automatic build process or not. But that means, in particular:

M2E can invoke a plugin as part of the automatic build process if, and only if, there is a connector for the plugin, or you specially configure the plugin. (More on that configuration later on.)

And that is the main problem we are currently facing: Connectors are missing for a lot of important plugins, for example the JAXB plugins, the JavaCC plugins, the antrun plugin, and so on. The philosophy of the M2E developers seems to be that time will cure this problem, which is why they are mainly ignoring it.See, for example,
bug 350414, bug 347521, bug 350810, bug 350811, bug 352494, bug 350299, and so on. Since my first attempts with Indigo, I am unaware of any new connectors, although the lack of them is currently the biggest issue that most people have with M2E. Try a Google search for "m2e mailing list connector", if you don't believe me.

But even, if the developers were right, they choose to completely ignore another problem: You can no longer use your own plugins in the Eclipse automatic builds, unless you create a connector for the plugin, or create a project-specific configuration. (Again, more on that confuguration in due time.)

At this point, one might argue: If you have written a plugin, it shouldn't be too difficult or too much work to write a connector as well. I'LL handle that aspect below.

First of all, regarding the configuration: Absent a suitable connector, there is currently only one possibility to use a plugin as part of the automatic build: You need to add a plugin-specific configuration snippet ike the following to your POM:


<plugin>
<groupid>org.eclipse.m2e</groupid>
<artifactid>lifecycle-mapping</artifactid>
<version>1.0.0</version>
<configuration>
<ifecyclemappingmetadata>
<pluginexecutions>
<pluginexecution>
<pluginexecutionfilter>
<groupid>org.codehaus.mojo</groupid>
<artifactid>javacc-maven-plugin</artifactid>
<versionrange>[2.6,)</versionrange>
<goals>
<goal>javacc</goal>
</goals>
</pluginexecutionfilter>
<action>
<execute></execute>
</action>
</pluginexecution>
</pluginexecutions>
</lifecyclemappingmetadata>
</configuration>
</plugin>


Neat, isn't it? And so short! This would advice M2E that I want the javacc-maven-plugin to run as a part of the automatic M2E build.

So far, I have tried to be as unbiased as posssible, but now to the points that drive me sick. (As if that were currently required...)

  • The space required for the M2E configuration typically exceeds the actual plugin configuration by far! If there ever was a good example of POM pollution, here's a better one.
  • M2E insists in the presence of such configuration, regardless of whether I want the plugin to run or not.If it is missing, then the automatic builder won't work at all. There is no default handling, as was present in previous versions of M2E. (I won't discuss what the default should be, I'd just like to have any.)
  • The M2E configuration must be stored in the POM, or any parent POM. There is no other possibility, like the Eclipse preferences or some file in .settings. In other words, if you are using IDEA or NetBeans, but there is a single project member using Eclipse, you still have to enjoy the M2E configuration in the POM. As bug 350414 shows, there are a real lot of people who consider this, at best, ugly.
  • I tried to play nice and start creating connectors. But this simply didn't work: I am a Maven developer, not an Eclipse developer. And a connector is an Eclipse plugin. I'm not interested in writing Eclipse plugins. (Which Maven developer is?) But there is nothing like a template peoject or the like, only this well meant Wiki article, which doesn't help too much. For example, it assumes the use of Tycho, which only serves to make Eclipse programming even more complicated.
  • The design of the connectors looks broken to me. Have a look at the AbstractJavaProjectConfigurator, which seems to be the typical superclass of a connector: It contains methods for configuring the Maven classpath, for adding source folders, for creating a list of files that have been created (or must be refreshed): These are all things that are directly duplicating the work of the Maven plugin and should be left to the Maven plugin, or Maven, alone. In other words:
  • Circumventing the Maven plugin is bad. Deciding whether to run or not should be left to the plugin, or Maven. (See, for example, the comment on "short-cutting" code generation on the Wiki page on writing connectors
    .)

To sum it all up:

I fail to see why we can't throw away the whole connector mess and replace it with a configurable Maven goal that should be run by the automatic build ? There is even a reasoable default: "mvn generate-resources". Let's reiterate the reasons for inventing connectors from above and compare it with this solution:

  1. Maven wouldn 't be invoked more frequently
  2. If a single Maven execution takes too long, fix the plugins that don't do a good job at detecting whether they can short-cut. Ant still does a better job here, years after the invention of Maven 2.
  3. If some plugins don't behave well with regard to resources, fix'em. If we can wait months or years for connectors, we might as well wait for bug fixes in plugins.
  4. The question whether to run a plugin or not can be left to the Maven lifecycle, if we choose a lifecycle goal like "generate-resources" Maven knows perfectly well the plugins and goals to include or exclude.

12 comments:

Andreas Schödl said...

And I thought it was just me. My team is half on Eclipse 3.7 (Indigo) and half on 3.6 - and yes, the 3.7 problems tab is filled with "lifecycle configuration" complaints from m2e.
It reminds me that for a long time the Maven community suffered from ill-written POMs and crappy bundles of libraries even in the most important repositories. Then Sonatype came along and cleaned up the mess, with enormous effort, with enormous effect on the usability of Maven as a whole.
So, there was definitely hope Sonatype would do the same for the m2e plugin, when they took responsibility (in what appeared as an unfriendly overtake).
In retrospect, the announcement of merging the plugin into Eclipse should've been a warning sign. Who knows what committee deemed it reasonable to impose a "lifecycle configuration tax" on any Maven plugin ever written?
What now? How's it going with NetBeans? Does it suffer from the same - or more counter-productive other problems? Or would it welcome my (server-) project with arms wide open?

Jochen Wiedmann said...

I can only tell that the NetBeans plugin has no idea of connectors, them beeing Eclipse plugins.

mkleint said...

netbeans in general relies more heavily on maven tiself to do the build. There is no netbeans-only build/project infrastucture, only the compiler sometimes outputs changed class files. For source generating maven plugin, it assumes them all to reside in target/generated-sources/FOO where FOO is typically the plugin name and in most cases it's the default location for the generated sources to appear.
In general NetBeans project philosophy is radically different from Eclipse one's..

Andreas Schödl said...

What the fork?! Installed NetBeans 7, checking out project ... Wait, you're telling me, it already has a Subversion connector, built-in?

Let Netbeans scan for projects? Yes.
What? You've found all generated source packages of the Maven plugin, without any fail? What are you, a miracle?

Why do I even bother with Eclipse?

Can't jump ship now, two days before vacation and leaving three busy guys behind. But ... tempting it is :)

Joe said...

I've tried to raise many of the same concerns on the m2e-users list. Although I believe the m2e developers are listening, fundamentally the decision has been made and sadly the policy line is: it's not going to be reversed. I'm still just baffled by this decision.

For me, points 2 and 3 of your final summary invalidate the whole connector concept. If plugins behave badly, fix them. There's just no need to demand a whole new integration layer between Maven and Eclipse and inconvenience the world and his dog in the process.

LordM said...

I've spent a day trying M2E both without and with the JAXB connector, with Indigo. It is 100% useless. We have dozens of projects with jaxb and web services source generation. Even with the jaxb connector I cannot repeatably get the generated sources to go on the classpath, so that the Java builder sees them. The only way is to manually run a Maven compile, then every time you do a clean you start again.
Our projects build all day, every day from the command line, but the Eclipse situation is worse than hopeless. There's no prospect of pursuading the team to switch from Ant to Maven, entirely because of the Eclipse nightmare.
All of this was after adding an ass-ton of verbosity to the pom to deal with the lifecycle mapping thing.

Ed Staub said...

Thanks for writing this. I'd just spent several months evaluating Maven when Indigo came out. I was still uncertain about it. When I figured out just how dumb this was, Indigo made up my mind: no Maven for me - not if I can help it. Not that it's Maven's fault, per se - but I don't care whose fault it is. A serious dope-slap is in order for the m2e folks.

Anonymous said...

I'm the same "anonymous" as previous post : here is a confirmation of my understanding : http://wiki.eclipse.org/M2E_plugin_execution_not_covered

Mark Pollack said...

Another one word answer to this - IntelliJ....

Anonymous said...

m2e is one reason I jumped over to RubyMine. No more Java hassle, no more Maven hassle, no more broken promises from the maintainers of m2e. They've been promising to fix this for years.

Chinto said...

I agree with you totally.

I'm so tempted to jump to Netbeans, but I can't as the JSF/SEAM supporting JBoss Tools is eclipse based and it is a very nice to have.

But having no proper maven support after this many years is a huge minus for eclipse.

Anonymous said...

And two and half years after this blog was written m2e connectors are still completely borked.