TL;DR: Kafka Streams Scala Application:
sbt new monksy/kafka-streams.g8
I’ve been working on a new project, which I’ll give more details on later, that involves a few Kafka Streams applications. As with any new Scala project, you should use a G8 template to start out with it. Starting out with a premade template is just a good practice. Additionally, templates can do a lot of work for you when it comes to good practices, structures, and extra compilation tools.
Unfortunately, with the Confluence/Kafka world, there aren’t many Kafka-Streams Scala based templates. On Github I found two that came up in the search:
sv3ndk template used the Lightbend version of the
kafka-streams-scala. It’s very out of date. So that was out. The
idarlington template used Kafka Streams 2.0. Not ideal, but not unworkable. So I forked it.
What was done:
- Upgrade the Kafka version from 2 to 2.5 (Latest Release)
- Upgrade the testing utilties to use the non-depricated (post 2.4) functionality
- Improved the file layout
- Added Assembly support
- Added a larger gitingore file
- Upgraded the project to Scala 2.13
- Upgraded the other libraries in the project
- Added dependency tree plugin support
- Upgraded the g8 build file
What did I learn about?
- I learned a lot about creating a G8 template and how the variables are substituted. The existing project had a lot of this work already done, however, I did have to do some of my own substitutions.
- MergeStrategies and dealing with “module-info.class” in the assembly plugin. (Hint: Merge strategy rule for: case “module-info.class” => MergeStrategy.discard) The module-info.class is a new addition that came from Java 9’s Jigsaw to define JVM modules. They’ll pop up in the Jackson libraries.
- Some of the built-in Kafka Streams Test utilities. For the most part, I’ve been using the mocked streams library.
- G8 Sbt plugin. Use
sbt g8to build an example copy of the application in
target/g8. From there it’s a lot easier to test and build up your template. Use
sbt g8Testto run an automated test. I’m not sure how to customize the sbt tasks. I’m sure it’s a configuration option.