TL;DR: Kafka Streams Scala Application:
sbt new monksy/kafka-streams.g8
I’ve been working on a new project, which I’ll give more details on later, that involves a few Kafka Streams applications. As with any new Scala project, you should use a G8 template to start out with it. Starting out with a premade template is just a good practice. Additionally, templates can do a lot of work for you when it comes to good practices, structures, and extra compilation tools.
Unfortunately, with the Confluence/Kafka world, there aren’t many Kafka-Streams Scala based templates. On Github I found two that came up in the search:
sv3ndk template used the Lightbend version of the
kafka-streams-scala. It’s very out of date. So that was out. The
idarlington template used Kafka Streams 2.0. Not ideal, but not unworkable. So I forked it.
What was done:
- Upgrade the Kafka version from 2 to 2.5 (Latest Release)
- Upgrade the testing utilties to use the non-depricated (post 2.4) functionality
- Improved the file layout
- Added Assembly support
- Added a larger gitingore file
- Upgraded the project to Scala 2.13
- Upgraded the other libraries in the project
- Added dependency tree plugin support
- Upgraded the g8 build file
What did I learn about?
- I learned a lot about creating a G8 template and how the variables are substituted. The existing project had a lot of this work already done, however, I did have to do some of my own substitutions.
- MergeStrategies and dealing with “module-info.class” in the assembly plugin. (Hint: Merge strategy rule for: case “module-info.class” => MergeStrategy.discard) The module-info.class is a new addition that came from Java 9’s Jigsaw to define JVM modules. They’ll pop up in the Jackson libraries.
- Some of the built-in Kafka Streams Test utilities. For the most part, I’ve been using the mocked streams library.
- G8 Sbt plugin. Use
sbt g8 to build an example copy of the application in
target/g8. From there it’s a lot easier to test and build up your template. Use
sbt g8Test to run an automated test. I’m not sure how to customize the sbt tasks. I’m sure it’s a configuration option.
I ended up consolidating some of the resources from the Awesomelists into a better organized awesome list in my repo:
Please help this repo if you know other projects that weren’t mentioned or with knocking off a few of the items in the TODO list.
So I just made a new thing, and open sourced it.
It’s called the serialization checker. From the readme page it’s here to solve:
The root problem that led to this project’s creation is that REST typically uses JSON, and that JSON is Schemaless. This makes it difficult to create data objects to interact with services. In the case of connecting to a third-party REST service, you typically have lots of examples. This project helps you, the developer, iterate through the creation of the data objects.
Where can you find this?
Github page: https://github.com/monksy/serialization-checker
resolvers += Resolver.bintrayRepo("monksy","maven")
libraryDependencies += "com.mrmonksy" %% "serialization-checker" % "0.1.3"
Or even it’s Bintray: https://bintray.com/monksy/maven/serialization-checker
I’m taking notes on all the talks that I’m attendeding at the Reactive Summit 2018 conference.
Pull requests welcome.
Check it out:
The only thing that it’s missing is interactivity, graphics, rotations, and animations.
It’s been a while since I’ve done a blog post like this.
However, I’ve been busy and I have a few things to show for it:
Exciting Meetups Attended:
- Thomas Nield- Datascience with Kotlin
- This was a fantastic meetup that involved Optaplanner, SMILE, RxJavaFX, and had a few demonstrations.
- This brought back memories of my school days using AMPL and Operations Research.
- Went to the Oracle Code: Chicago Java 10 release
Projects That I’ve been working on:
- Temperature Sensor Data Generator
- This is a utility project that will generate a series of data from sensors that closely resemble a day’s change in temperature.
- This is to generate a large enough dataset to really demonstrate large-scale distributed processing
- CQRS Framework
- An extendable framework used to track events throughout a data object’s lifespan.
- Using Shapeless
- Currently on hold until I can fully wrap my head arround Shapeless
- Sample Akka-HTTP Based Application for Inventory Pricing
- This is a sample Akka HTTP Based application that responds to Time based requests for inventory.
- Akka HTTP was a bit irritating to setup the routing.
Technologies I’m Learning Right Now
- Apache Spark
- The Play Web Framework
- Amazon RedShift
Things I’ve Mastered/Dealt with Cooking
- Sous Viding
- Experimented with Octopus (They were a bit too small to get right, and this was done with Sous Viding)
- The Perfect Steak and Crust on the outside
- Dry Aging Ribeye Steaks
- Lemon Bars
- Tirmasu Fudge
- I’m very close to making a stock
Books I’m Currently Reading
Topics I want to learn/read about
- More with IOT
- I had a chance to work with a Seed WIO wifi based IOT board
- I bought a Nano PI from FriendlyElec.
- Going further indepth with Kafka
- Akka Typed
As it currently stands there is no OS independent way to extract all of the photos from your meetup group. There are other projects that run on Windows, but nothing written in Java. Under meetup they don’t offer the ability to export all of the photos. This short script is used to gather all of the photos in your group and download them into your local file system via the information provided by the Meetup API.
What technologies are used?
– Groovy and the RESTClient (It’s fairly simple)
For the most part this was a fairly simple project that could be finished in a lunch or two. That is only the case if you already have experience with the Meetup API. It was a pretty simple thing to write. Only thing that I found unique/learned with this project was that you can write to an output stream via the left shift operator.
Meetupphotodownloader source: https://github.com/monksy/meetupphotodownloader/
Have you ever had the need to verify an HTTP request? If so the tool “RequestBin” is the way to do it. It is the pastebin of HTTP Requests. When you use the site, it’ll give you a temporary custom URL to post whatever you need to, and it will display the contents when the request comes in.
This is awfully handy for testing Web hooks from Gitlab and Github.
I’ve been meaning to write an article about a few of the communities in the open source world. However, I believe that the article “7 Open Source Projects to cut your teeth on (and the ones to avoid)” by Rikki of ITWorld has said what I wanted to. Some of the open source projects that I’ve had good/bad experiences contributing to have been:
- XBMC [They will not take bug reports or feature suggestions]
- Tiny-Tiny RSS
- OpenStack Folsom Install Guide [The official documentation doesn’t agree with some of the suggestions, and I’ve tried to point this out]
I understand that these tend to be non-work projects, and that it can take a lot of work to maintain a community. But its rather frustrating that people who attempt to chip in to help make the system/application/code better are treated rather roughly.
codeBeamer ManagedRepositories is a free web interface for Subversion, GIT, and Mercurial from Intland. The product contains a standalone web application with their distributed version of Tomcat. However, the Linux version does not include the init.d scripts to start/stop the service on boot or on demand. I’ve written a script that can do this. It can be found on my GitHub page. The instructions can be found in the README.md. At the time of writing, I cannot endorse or recommend against this product as that I haven’t used it yet. However, a review may be coming up in a future post.
To bring down the script, and the read me file [assuming that git is installed], create a new directory and run the following command within it:
git clone https://github.com/monksy/cmr-init