Things I’ve learned so far this week (6 Dec)

It’s been a very long time since I’ve done one of these. So I thought I’d share. Not all of this was learned in the last week, but in the recent past.

  • When you do port forwarding with SSH, the default binding address for the forwarded port is 127.0.0.1. This is a security feature that prevents that port from being used by outsiders.
    • If you want to expose this, enable the configuration GatewayPorts in the SSHD configuration
  • Groovy has special DSL support for Cucumber. It makes writing cucumber steps easier and simpler to manage environment hooks
  • The obvious one, but rarely done in practice: Creating a POC in code is a lot better to get your core logic working than building out a full app and trying to make decisions later.
  • Bash: Substring extraction is possible in a very concise and easy manner. (Like most things in Bash, there are magical characters for that) It makes sub-arrays possible.
  • Bash function parameters are errrr different. Unlike most C-like languages, they define them by an array index than a variable name. (I.e. $1, $2, …)
    • You should define your local variables in your function near the top
    • All variables are global unless you declare them as local in bash

Other

  • Magic shows can be pretty cool (Thanks, Michael Carducci! [NFJS])
  • I need to see more firebreathing shows. (Rammstein included lots of fire displays)

Akka/Scala vs Groovy for a Periodic Process

Last fall I created two different processes that were responsible for a reoccurring task of pulling data and broadcasting it elsewhere. The Akka based project pulled the newest meetups from the meetup page and then rebroadcasted them via Reddit and Twitter. The Groovy-based application pulled posts from a forum and then created a summary post of the posts pulled.

Both applications were reoccurring and were only activated on a schedule. However, there were two different outcomes. The Groovy based jar was scheduled as a cronjob and exits when done. The Akka process is set up as a systemd service and remains up.

Last fall I created two different processes that were responsible for a reoccurring task of pulling data and broadcasting it elsewhere. The Akka based project pulled the newest meetups from the meetup page and then rebroadcasted them via Reddit and Twitter. The Groovy-based application pulled posts from a forum and then created a summary post of the posts pulled.

Both applications were reoccurring and were only activated on a schedule. However, there were two different outcomes. The Groovy based jar was scheduled as a cronjob and exits when done. The Akka process is set up as a systemd service and remains up.

How did it turn out?

Both of the solutions work and have required very little maintenance. When running, the Groovy process takes up less than 70mb of memory, and the Akka based process takes more than 200mb  of memory. [It’s showing as 0.3% memory usage on a 65gb machine] (It’s written in Scala, and brings in Akka) Nether of the processes are intense enough to make a noticeable effect on the CPU. The ultimate package size is the following: Akka- 42mb Groovy- 12mb. (That is a little deceptive as that that the Groovy process contains less 3rd party libraries, but the amount of libraries that Scala and Akka bring in are a lot).

Now it comes down to probably the biggest concern: The time it took to develop. It took a month of lunches to develop the Scala and Akka based application. It took so long because I had to find and get up to speed on the Scala-based libraries for working with Rest services (I had spun my wheels with different clients), Scalalikejdbc, and Twitter4j. I learned another lession: Don’t use SBT. It’s a pain to use when compared to Gradle or Maven. On top of all of that I had a very challenging lesson: Don’t mix Scala versions for dependencies that weren’t compiled against the Scala version that your application is using.

The Groovy-based application took three lunches to write. One lunch (~30-40min) to write the main logic, the rest for the packaging. (The worst part of this was researching on how to make a fat Jar for Groovy under Gradle).

What did I learn?

Akka is an amazing piece of technology. I like using it, and I like using Scala. However it turns out for a process that is meant to be periodic, using REST calls: you are far better off writing the process in Groovy, letting Linux handle scheduled execution and getting it done quicker.

When would I use Akka? I would use it for a system that would expect constant requests, and that it had to be highly reactive, may include complexity, and would expect a high throughput. (The REST service is a good example for this)

Project: Meetup Photo Downloader

As it currently stands there is no OS independent way to extract all of the photos from your meetup group. There are other projects that run on Windows, but nothing written in Java. Under meetup they don’t offer the ability to export all of the photos.  This short script is used to gather all of the photos in your group and download them into your local file system via the information provided by the Meetup API.

What technologies are used?

         Groovy and the RESTClient (It’s fairly simple)

For the most part this was a fairly simple project that could be finished in a lunch or two. That is only the case if you already have experience with the Meetup API. It was a pretty simple thing to write. Only thing that I found unique/learned with this project was that you can write to an output stream via the left shift operator.

Meetupphotodownloader source: https://github.com/monksy/meetupphotodownloader/

 

A few things that I would like to see in Docker

I’ve mentioned before that I’ve rather taken a liking to Docker. However, there are a few things that I wish that were better.

Docker Doc

Java doc did something extremely well. (Other than getting people to hate checkstyle) it made sure there there was a standard way to write documentation on your code base, and that it could be formatted in a proper, consistent way.

It would be great to see if there was a similar tool for Docker to produce consistent documentation on what the dockerfile produces. It would also be great to have registry support for this as well.

Some of the things that I would expect to see mentioned on there:

  1. Required environment variables for the application
  2. Documentation on maintenance to be done on the container (i.e. run particular methods via the exec command)
  3. Ports that should be shared
  4. What volumes are assumed to be there
  5. How the configuration should be set  (is it a file, etc).
  6. How to connect the application with other services (databases, message queues, etc)

Docker Templates

It would be great to generate best practice docker files based on if you’re working with a Tomcat application, or a standard Jar. This would get rid of the creation time needed to make a Dockerfile. Along with Dockerfile templates, it would be nice to see better Jenkins publishing support for Docker. However, that’s a subject for another day.

Docker Traits

Multiple inheritance has a lot of issues that Java avoids, and C++ deals with. The issue with this is avoided in Python (with mixins), and Scala/Groovy with the addition of traits. Traits are aways to side load content into the docker file, post the base container, without having to use inheritance. That means that you could create a Docker container with utilities rather than baking them into the base image.

Application Profiles

Applications typically have a limited set of configurations. They are either one time running applications or they’re web applications. There are sometimes deviations from this, but they’re not so common.

For the most part, one time running applications require a configuration set to go into the application and a location to write out the data. For web applications it mostly needs a port to be open, configuration to a data storage system (i.e. MySQL, MongoDB, etc), and configuration information.

The best way that I can forsee this being setup is to have a standard mount location for these folders outside of the container, and to have them set as required inputs for starting the container.

Another item on configuration. I would love to see the configuration copied from the container to its outside storage for the starter configuration. This would mean that the Docker Registry complex configuration would have a new mounted volume outside of the container there with a configuration ready to have a change made to it.

My Second Chicago Java User’s Group: I didn’t know you could do that with Groovy!

Previously I blogged about my first experience in doing a brief tech talk in front of my local Java User’s Group. I recall that I was incredibly nervous and ill prepared. Well, I took the feedback from the talk, prepared better for this talk and came ready to give my talk. The talk was on how Groovy makes quite a few improvements over Java, and it was in the format of a lightning talk. At the moment the video is being processed and it should be up within a week or two.

While you’re waiting for the video, I’ll leave you with the slides and the Github link.

Notes: Using Betamax in Grails (2.4.x)

Betamax is a great tool that records the output of a web service, and serializes the full request to a file. Once the request has been recorded, it can be played back in a test. This comes without any setup on your HTTP requests. This means that your system test no longer has a need to be connected to a working service, or has to be mocked based on a hand created mock.

Since the 2.3.8 release of GRAILs, there have been a few quirks to getting Betamax to work in your tests.

A few things need to be considered when using Betamax with Groovy Grails:

 

  1. Conflicting dependencies with the build process and SnakeYML
    https://github.com/robfletcher/betamax/issues/153
  2. The configuration file isn’t respected.
    Use the rule:

    @Rule
     Recorder recorder = new Recorder(
            tapeRoot: new File(BuildSettingsHolder.settings?.baseDir, 'test/resources/tapes'),
           ignoreLocalhost: true,
          sslSupport: true)
    //The sort property isn’t set: set this up in the test setup
    TapePropertyUtils.metaClass.sort = { Set properties, List names ->
    new LinkedHashSet(properties.sort(true, new OrderedPropertyComparator(names)))
    }
    
  1. Use Groovy to rewrite your RestClient

    def meetupBase = new RESTClient(url)
    BetamaxRoutePlanner.configure(meetupBase.client)
    meetupBase.client.removeRequestInterceptorByClass(ContentEncoding.RequestInterceptor.class)
    meetupBase.client.removeResponseInterceptorByClass(ContentEncoding.ResponseInterceptor.class)
    meetupBase
    
  1. Location of tape files in the project
    ~/ProjectFolder/test/resources/tapes
  2. Note about the hostnames host reference. (All of the tapes are based on the mocked name, so be prepared to modify this if need be.)

 

I gave a technical talk and so should you

Public speaking is a lot trickier than you’d think. It’s not hard, its mentally challenging, however it can be very rewarding. The majority of the population can talk about their interests to their coworkers, friends and family for a long period of time. However, they will struggle to talk to a large amount of people about what they know. It’s a shame, and it’s an odd problem to have and it’s a phobia that some have. It’s a little odd because the perceived threat isn’t a physical attack or something that threatens one’s life.

The common negative perception based on poor performance, making mistakes that are “attributed” to one’s self, and perception from others. The reality of the situation is that even if you do very poorly: It’s incredibly rewarding. In the upside, you may be invited to talk at other places, talk on other subjects, you may be considered a subject matter expert, get paid to give talks, etc. On the worse case of this: You may get stage fright and most people can understand that. Even in the worse case: You gained a huge amount of respect from the audience, and you learned that you have something you can work on and improve.

Recently I gave a quick talk in front of my local technical community. I was encouraged by the local Chicago Java Users group to give a lightning talk. Lightning talks are brief five minute talks on a particular subject. I’ve seen others do lightning talks before and they seem to breeze by fairly quickly. Additionally the format of this isn’t necessarily to show strong talks, but it’s desgined to get people up in front of an audience, and give them a taste at presenting. A friend of mine, Mark Thompson, encouraged me to go through with my talk. My talk was on the Spock Testing Framework and why people should use it. Prior to the talk I was a bit nervous; there were 60-80 people there. During the talk I felt that it didn’t go as smoothly as I thought it would in my head. However, the audience reaction was good, and my feedback was great. I watched the video later: Despite having a few quirks that should get ironed out, I think I did well for my first public technical talk.

Things I learned:

  1. Practice the talk before hand. I believe that I had a few issues during the talk because I hadn’t vocalized what I had planned to speak about
  2. Practice with the equipment that you’re given
  3. Make sure to have it recorded. This will give you valuable feedback later. Luckly for me, this was done by the CJUG crew.
  4. Always have others review your slides (Did that)
  5. Keep your slides as short as possible (I did ok on that)
  6. Dress for the part (Dress up): I could have done much better on this.

Curious about the talk or Spock? If so take a look at the video and feel free to respond with constructive criticism. I’m always willing to listen.

A side note about the talk: I did have something to do with coercing a few colleagues of mine to do a talk. They were: Todd Ginsberg (A short introduction to Akka)  and Mary Grygleski (An introduction into Mule ESB). They did a fantastic job as well!

Spring Security OAuth- The missing Instructions

The problem and lead up to the solution

I have been working on an authentication issue for more than a month. This has been incredibly irritating, and I have not found anything related to this online – not even in Stack Overflow questions. The issue is that I followed the instructions for installing Spring Security, Spring Security OAuth, and the Google [etc] plugins for my existing application. After following the instructions to a T, I clicked on the authentication button for the Google OAuth. What happened? I was sent right back to the login screen.

 

This was rather weird. My first few thoughts were:

  1. Did the authentication fail?
    After an investigation, found this to be false. Chrome’s developer tools showed that the network path redirected to the /oauth/callback and then to /oauth/google/success. So that was a no.
  2. Was the redirection page not working?
    I really had no idea on this. It seemed to work for the locked-down resources. But I still seemed to have received the login page after coming back from Google OAuth. That’s logged within the Spring Security plugin depths.
  3. Was the authentication return point getting rejected?
    Here we would have a loop back to the authentication page.
  4. What controllers were getting hit?
    The debugger won’t go into the controller call itself. Also, the expected controller wasn’t getting hit. So this wasn’t of any help.
  5. Were the Plugins incompatible with my Grails 2.4.3 application?
    I was using the most up to date plugins for Spring Security. (The most recent publication date of May 2014 didn’t help with this concern)

 

After these thoughts, I was stuck. There were other scenarios on Stack Overflow and mailing lists that made claims to the plugin not working. However, most of those issues were resolved by switching plugins or due to syntax errors in the config. Additionally, I checked the plugins’ issue pages and there was apparently nothing about the issue I was having.

 

To answer the third idea of where the problem could be, I looked over the following static set of rules configured in the config:

 

grails.plugin.springsecurity.controllerAnnotations.staticRules = [

‘/’           : [‘permitAll’],

‘/index’      : [‘permitAll’],

‘/index.gsp’ : [‘permitAll’],

‘/assets/**’ : [‘permitAll’],

‘/**/js/**’   : [‘permitAll’],

‘/login/auth’ : [‘permitAll’],

‘/**/css/**’ : [‘permitAll’],

‘/**/images/**’  : [‘permitAll’],

‘/**/favicon.ico’: [‘permitAll’],

‘/oauth/**’   : [‘permitAll’]

]

 

After all the addition of the login/oauth bits were recommended in the documentation.

I even tried testing Google’s side with the console. Everything worked on their end. My assumption was that if one of the plugins was misconfigured it would try to alert me via the logs. (Well, as I learned that assumption made an ass of me.) Finally I did some research within the plugin. I found the package structure in which I could bring in the logs.

 

trace ‘grails.plugin.springsecurity’

That eventually lead me to a lot of logging about only the filters/interception of web traffic, which didn’t help very much, and it still lead me to question what was going on. So I tried logging the service and controller methods. (That lead me to realize that I was never getting the actual OAuth plugin to kick back in on the way back. Additionally I found this bit in the logs (that were enabled earlier):

 

intercept.AnnotationFilterInvocationDefinition no config for ‘/springsecurityoauth/onsuccess’  (which was followed by a redirect to /login/auth.)

 

Google was not much of a help for this method. Nor was it a lot of help for the ‘springsecurityoauth/onsuccess’ path. This meant that the access to the actual controller SpringSecurityOAuth::onsuccess method couldn’t be reached. This was a bit confusing since the Spring Seucrity OAuth defines it’s own URL mappings in: SpringSecurityOauthUrlMappings. Alas, it’s a static call. I’m not able to debug that low.

 

This led to another path: Well how are the other path’s defined, and why isn’t /springSecurityOAuth/onSuccess allowed. SpringSecurityOauthUrlMappings maps the controller to /oauth/$provider/success, and that’s allowed under my rules.

 

The Solution

Well that path isn’t allowed. Why? Spring Security has 2 ways to lock down your application.

  1. InterceptUrlMap Method: Statically define all of your paths/roles via a static map in the config.
  2. Controller Annotations Method: This allows for you to define some static rules requests are allowed and to what accounts in the config file. It also allows for you annotate your controllers/methods to what is allowed by what role(s).

 

Guess which way I went. Yep, I went the progressive route and chose the controller annotations method. That worked really well for what I had, and with authentication outside of this plugin. But it failed when trying to return. Why did this still fail? Well the configuration of springsecurity.controlelerAnnotation.staticRules = [ …. ] was missing the reference to the controller (that looks like a url): ‘/springSecurityOAuth/**’: [‘permitAll’]

After adding that small change: everything worked. A lot of time was lost. I’m going to need a break away from Spring Security. That and probably quite a few drinks.

What could have been improved?

The documentation needs to be updated to mention controller annotations.