(Transitive) Dependency Hell Pt2: How it could have been avoided

In the last blog post, I went over an issue where multiple logging systems built into the dependencies caused issues in the main application. All of this could have been resolved by delaying the decision on what logging implementation should be used.

The initial fix that would have made my debugging session easier

Trying to resolve issues with conflicting logging frameworks is a huge pain. It would be great if there were an option to show the initial setup of the logging framework used in the first place. If there is already documentation for this, I would love to know where it describes the actions of it.

What are JSRs

Java Specification Requests are community/corporate created specifications that define an interface for a feature in Java. For the most part, they just define the interface and the implementation can be included in the classpath. No actual concrete classes are included in the specification. If all of the sub-dependencies were using an up to date JSR for logging, this issue would have never come up. The only issue that would have shown would be “No concrete class for interface found.” A quick search for logging based JSRs turned up JSR-47.

For example, The REST JSR-311 has the API JAR. If you didn’t understand about JSRs, you’d believe that it includes the functionality to work with REST services. However, it doesn’t it requires a JSR-311 compatible implementation/provider. (Such as Jersey). Another example of this is JPA and the Hibernate JPA implementation.
On a similarly related note, it would be better if library providers (I.e. Amazon AWS SDK) provided a specification/interface collection to their libraries. If this was the case, the libraries would be coded against this and the implementation could be used at the highest level. (Where the code is actually run) This would improve on testing, as that you’re now writing code against an interface rather than an implementation you have to mock out, and it would improve on dependency conflicts. The AWS SDK has had changes in the package structure, and issues with deprecated methods.

If you took nothing out of these two blog posts, take this: Programming against collections of interfaces is much more preferable rather than playing whack a mole with multiple conflicting dependencies. Filling your POM with exclusions is no fun, and it’s incredibly risky.

(Transitive) Dependency Hell (In Java) Part 1

Transitive conflicting dependencies is like letting 20 children fight over one piece of candy. At the end, one kid may have candy, leaving everyone else either crying, fighting, and/or hurt. At the very best situation the right dependency may be used, however frequently that’s not the case and may change if the class loader feels like it. Before I get into the context of what happened, I should mention this happened nearly a year ago, and this retelling is entirely from memory.

How did this happen? I was working on a project that required bringing in Spring CLI for a small utility. Spring CLI made the wrapping of methods into CLI interfaces incredibly easy. Side note: having to build up a CLI interface to do this is absurd and makes the task needlessly complicated. (I’m looking at many of the CLI options out there)

My project had a dependency of Java Universal Logging (JUL), Log4j (which the configuration was conveniently ignored), and SLF4J. Since the project was logging what we needed (and we were lucky that it was), the conflicts between SLF4J and Log4j2 were ignored. JUL used SLF4J as it’s default logger. After all, JUL is a weak logging framework that’ll jump in bed with the next available logging implementation. Each implementation has it’s own respective configuration format, and if the one that is being used doesn’t match the configuration that you have, it won’t take that into consideration.

The downside to bringing in Spring CLI is:

SpringCLI brings in Apache Commons-logging: Oh crap.

The even worse thing about this: When I started looking into this issue It’s not very clear about what logger is being used to log out the information. That means that even changing the, what you believe is the correct, configuration to increase the verbosity of the logging won’t work. To find out which logger was winning out, I had to debug deep into the internals of where the logging was going on. It wasn’t much fun.

From here, it’s key to decide which implementation that you want to work with. I decided to go with Log4j2, as that the configuration was written for it. The next step in the process was to eliminate/exclude dependencies from coming in. If you have a dependency that is using a fat jar,you’re out of luck on this one. Fortunately, I didn’t have this issue. This can be resolved by using the dependencies::tree plugin via Maven. Find all of the alternative implementations and get rid of them.

That still left the issue unresolved. It seemed like other implementations were leaking in and/or missing once you have eliminated the other dependencies you’re going to have to use bridges to resolve the issues with the now missing transitive dependencies. The following new dependencies were added:

Oy, this has now made the POM rather large and not very clear about what is going on or why you need all of that. For the most part, most of the issues should be resolved by this. However, JCL was still being redirected incorrectly within the Spring CLI output. *sigh*

After hours of debugging: I found that Spring CLI used JUL to write out the logging statements, and the logging was still trying to go through the common logger (but not writing it out/respecting the preferred logging).

This was resolved by setting the default logging manager at startup to the Log4j manager via the property:

java.util.logging.manager to org.apache.logging.log4j.jul.LogManager

 

In the next blog post: I will discuss some of the ways that all of this could have been avoided.

 

Installing Maven on Centos 5 or 6/RHEL

At the moment there is no RPM package or yum install available for the latest version of Maven on Centos. The user is left to install Maven manually. To attempt to overcome this, I created a script to install the latest, at the moment: 3.1.1. At the moment, there are many things that should be added to the script, they’re listed in the TODO section of the documentation, but those features may be added later.

Instructions on how to run the script, and the script it’s self may be found at: https://github.com/monksy/centos-maven-install 

Apache Wicket MarkupNotFoundException

If you’ve run in to the error:

org.apache.wicket.markup.MarkupNotFoundException: Can not determine Markup. Component is not yet connected to a parent”

Searches online typically don’t yield very much. They tend to give the hint and reiterate what others have found: The HTML file is either misnamed, the resolving component fails to find the corresponding HTML file to the Wicket java class, or that the HTML files are not included in the WAR.
They don’t offer very many solutions. However in my situation, using Maven, the issue was resolved by including resources reference in the build section:

        <resources>
            <resource>
                <filtering>false</filtering>
                <directory>src/main/resources</directory>
            </resource>
            <resource>
                <filtering>false</filtering>
                <directory>src/main/java</directory>
                <includes>
                    <include>**</include>
                </includes>
                <excludes>
                    <exclude>**/*.java</exclude>
                </excludes>
            </resource>
        </resources>

Debugging/Depedency Issue with Maven

When debugging an application that involves a framework: If you debug into the framework code and the code and functions don’t match up in the debugger. You probably have a dependency issue. This is quite possibly caused by a sub-component that uses the framework you’re using, but it’s using the wrong framework version. When that happens the wrong framework version may be used at runtime.

Also to note: When using the WicketStuff library, the newer libraries have an artifiactId of wicketstuff-*. If you are using a wicketstuff component without the wicketstuff– prefix, then it will limit you to using 1.4.2.

Apache Wicket [In Action]: A Review and How It Relates to the Java World

Java is a great tool for creating software. It is well designed, modular, has a wide array of platforms that it can run on, performs well, it’s very extendable, and it has a large community with lots of support. However, it’s support for websites and related services is severely lacking. It’s bad enough that frameworks that extend the existing infrastructure have massive pitfalls that you later discover.

For the most part Apache Wicket solves many of the web related issues that J2EE (JSP) has. If you are to follow the prescribed way of doing things, it can actually be quite pleasant. However, there are a few thorny patches with Wicket. I will get to those later.

Wicket In Action like a marriage. During the honeymoon, everything is great. Everyone is happy, and then later discontent grows and things go up and down. However, unlike a marriage, you don’t really get an ending. This is a rather good way to end things, but some of the lesser parts of the book were rather disappointing. One of the big selling points of Wicket is that it is a framework that assumes that the developer has already prototyped the pages in HTML prior to starting with Wicket [The pages are adapted into Wicket-ized and previewable pages]. This upside was enough to ignore the placement of the HTML files, in the class path rather than the web resources section.

I jumped into this book with lots of enthusiasm after reading the introduction. I even bore through some of the non-stated setup issues in the book. The book starts off by creating a project, from scratch, however I went the Maven route (which I discovered is the better way to go). The book mentions maven, but it doesn’t mention how to build your application or to generate a project. I believe that I went the correct way because Maven helped to setup all of the application servers and the folder structures. The book started by having the user to jump in, examine a few code segments, and then to start on a sample (full featured) e-commerces application. The store was oddly pleasant; the goal was to sell cheeses online. The application started from a few sample view pages, it went on to creating a reusable shopping cart, and finally on to a membership mechanism. This is a very straightforward and to-the point way of starting a new framework. It’s already addressing the needs of the majority of its audience.

Another nice thing to point out about the introduction is that it did not try to cover all of the material at once. It would frequently describe what you were doing, but would mention the chapter where the concepts were explained in depth later on. Something that pleased me was that the code listings did not include a listing number. They were place in the correct location of the text. After you’re done with the sample application, you should be quite proud of yourself. This is similar to your first website.

However, the book got a little disappointing when describing the more detailed interworkings of Wicket: sessions, bookmarkable pages, and layering/rendering. The book improves when it gets to the Ajax functionality and a brief mention of dependency injection and Wicket. The book gets a little rough in the Spring through Hibernate sections and then better in the testing section. The book ends in a rather low note on SEO, production configuration, and JMX. If I had known more about JMX, I would have probably had a better opinion of the ending.

Overall I am not sure if I can say that the less than stellar sections of the book were entirely the authors’ or the book’s fault. It quite possibly may be the technology’s fault.  I would strongly recommend the book if you are new to Wicket.

Lastly, here are some direct tips that I had to discover on my own that helped out a lot:

 

Two Issues I have With IntelliJ: Inconsistent building, and GUIs

During the massive end-of-days sale, I bought a copy of IntelliJ. I have been primarily a NetBeans developer, but I have used IntelliJ at a previous job. So two of the things that irritated me have been: the inconsistent build support and IntelliJ’s handling of Java GUIs.

IntelliJ has its own internal build system. However, it also has support for Maven. When you have a Maven project, it adds in dependencies to the Maven file, but continues to build with its internal build manager. If you need to add a dependency, IntelliJ will automatically add the dependency in the POM file, but you have to manually add it in its own build manager. Given this inconsistency, it is possible to have two different build configurations within the same project.

Additionally, converting a project from a Netbeans based project with a GUI doesn’t work in IntelliJ. IntelliJ will not interpret the Swing code generated by Netbeans. This means that for anything GUI related, I will have to use Netbeans.

What would be nice?

Support a non-proprietary build system. NetBeans defaults to ANT, and can support Maven natively. If this is not an option, then write an adapter for Maven/ANT/Gradle.

GUI: Reuse the Mattese code. Make Mattise a plugin and write an adaptor for IntelliJ.

Progress on Tech Resolutions of 2013

Earlier this year I made a few resolutions for this year.

So far I have completed the following:

  1. Finished and reviewed “Groovy In Action”
  2. Refresh on networking

Despite this not being much, I have learned many new things about disk storage, and Maven. Also, I’ve created a new GitHub repository for a ogg2mp3 script that I created.

Things left to learn/do:

  1. Finally get around to learning GridGain.   [I would love to see a well published book, or at least a Kindle eBook to get some headway on this] HazelCast would also be interesting, but GridGain has more of what I’m looking for.
  2. Finally understand how network routing works.
  3. Get more experience with DNS, and DNS tools
  4. Master NMap [not just learn the basic uses of it, but to really excel with the tool] This would be similar to the reading up on SSH I did last year.
  5. Get up to conversational level German. [Living outside of German speaking nations makes this incredibly difficult]
  6. Finally develop some strong time management habits.
  7. Learn how to use Python [to the point where you can do some cool stuff with it]
  8. Learn R [rather than haphazardly hack]
  9. Meet/talk with some of the gurus of airfare scheduling/decoding, and the famous Tom Stuker.
  10. Learn how to use GraphViz [This is one of the odd ones here, but it’s interesting]
  11. Get better with Erlang and to find/make real world uses.
  12. Learn/Create a GUI in Apache Pivot
  13. Create a web interface with either Stripes and/or Wicket.

 

In Support Of Maven

In the last few days there has been an article completely bashing Maven on hacker news. I found this a little aggravating. Much of the article spent time complaining about how the files are structured, and how much effort it takes to maintain a Maven file. Also, the article complained about the plug-in structure.

To some extent I can agree with the original author. The plug-in system makes Maven a little difficult to learn and get completely right. On the other hand, the plug-in system makes it easy to deliver a single meta-build system that can easily adapt to what you need it to do. Building a Java web app? It has a plug-in to build and package it for you.

My main beef with the article is that it completely ignores why we got Maven in the first place. It builds, packages, and manages your dependencies better than ANT does. Before, you had to manually acquire, and grab your dependencies. If you had ANT or make, then you had to include the declarations for the dependencies in the build file, and put the dependency in your project. This is a pain for those that are using source control, and for setting up new developer machines. However, with Maven, just declare the dependency and the scope in the build file and you’re done. If you’re using an IDE it’ll bring in the references and may be able to pull the Javadoc if it’s available. Additionally, any new upgrades to the dependencies could be added just by changing the version number. Imagine doing that without a Maven like build system. Maven can even go the extra mile and create a redistributable package [this is a little tricky, but you can do it], deploy your new package to a repository, check in your code for you, or even deploy it to a webserver. For example, if you have a web service and don’t have Tomcat7 downloaded, run it with an embedded instance with mvn tomcat7:run . You’re done, Maven grabs Tomcat7, its dependences and tries to deploy your webapp to the new instance of Tomcat7. No installation of the server required.

In closing, Maven is a build tool. Unless you’re making project structure changes you shouldn’t be messing about with the way it builds things.

Links I’ve Found Interesting in the Last Week/Technical Things I’ve found Interesting (6 January 2013)

  • Getting the “Application Server Libraries not found” error with IntelliJ when you try to add a local Tomcat server? If you’re using Gentoo it’s a matter of file locations. Install the tomcat-api package, and make a symbolic link from the API within the Common and Shared Tomcat directories. Source.
  • Tomcat 7 has changed around how resources can be accessed. You can no longer use “getResourceAsStream” to get files placed in the Web-inf/Classes folder. Instead, you should attempt to get the resource through the context of the local thread. An example: Thread.currentThread().getContextClassLoader().getResourceAsStream(). This is useful for applications that use JasperReports [location of the JRXML file], and that are web applications. Source.
  •  If you are trying to stand up a web server, and don’t want to allow it to accept public connections? Set the accepted connections on tomcat to only listen on the local address. For tomcat see this. For NGinx see and set it only set listen to 127.0.0.1:[port].
  • Converting a GUI Netbeans project into a Maven based project? Most of the needed libraries will be covered by the default Maven repositories. However the swing-layout dependency is a special case. To use include the Netbeans repository reference, and include the swing-layout-1.0,4 dependency.
    Repository:

    <repositories> 
       <repository>
          <id>netbeans</id>
          <name>NetBeans</name>
          <url>http://bits.netbeans.org/maven2/</url>
       </repository>
    </repositories>

    Dependency:

    <dependency>
       <groupId>org.netbeans.external</groupId>
       <artifactId>swing-layout-1.0.4</artifactId>
       <version>RELEASE691</version>
    </dependency>
  • Ignoring failed tests on a maven build: Add this parameter to your mvn command: -Dmaven.test.skip=true
  • Sort directories and files with a human readable format: This is like ls –Sh, but it traverses directories.
  • If you have a set of directories to ignore, SVN propset includes an option to recursively ignore specified names/folders. A word of warning propset overwrites any svn:ignore properties.