Seven Databases in Seven Weeks: Postgres

After finishing “Coders at Work, “ more on that in a future blog post, and having little experience with non-RDBMS databases, I picked the book “Seven Databases in Seven Weeks” by Eric Redmond. The book appears to be of similar quality to it’s sibling “Seven Languages in Seven Weeks” by Bruce Tate.

The book starts out with the Postgres database. At the time of writing, this database wasn’t as popular as MySQL however it does make a good starting point as a baseline of comparison. It represents the “old guard” of databases. For most of the first week, I found that the first half of the first week was not of much interest to me. However, the fuzzy search extensions and full text search extensions caught my attention. I have always been aware that the capabilities existed, however, I never knew how they worked. Additionally the downloadable source code helped with creating a testing environment right out of the box. This was the same case for the “cube” extension/datatype. I found it very exciting to find out that you could do some rather interesting operations with multidimensional data and queries. I can’t claim that I’m an expert on using these features but its rather nice to have some hands on experience for it.

I don’t believe that having that content was the greatest value of the book. I believe what gives the book the greatest value is that investigating more on the cube package it led me to finding an online directory of the available extensions. I found the Postgres Extension Network. How exciting is it to find a directory of extensions to a fairly standard database that allows you to do some cool things? You can find extensions to interact with JSON data, store bitmaps, keep key/value data, additional aggregation functions, weight averages (This is a VERY interesting addition), and even attempts to do a “connected regions” logic within data items. These are reusable components that others have created, and that I found that I could get the database to perform these actions rather than code them myself.

 

First Thoughts: “Coders at Work” by Peter Seibel

I’ve just started to read the book Coders At Work. The book is a nice, recent collection of interviews from many big name developers. I’ve read other developer interview books before, but this one sticks out in an unusual way: with most “interview” books, the interview is either completely boring or incredibly interesting. In Coders At Work, the interviews have varied between amazing and neutral. I haven’t gotten to a bad interview yet.

A few things jumped out at me and made me think. Jamie Zawinski’s interview made me wonder about the value of non formally-educated developers in “today’s market.” Brad Fitzpatrick’s interview reminded me of the “I’ll build everything,” but you “must know everything” attitudes. Douglas Crockford’s interview didn’t inspire me, but it did make me consider other issues within software development.

Jamie Zawinski’s interview was an amazing conversation about a guy who has many interests in learning and doing work. He is a self taught LISP developer who can occasionally get very opinionated. I found his work experience with Netscape fascinating. As a user of the early versions of Netscape, I never knew all of the politics or construction going behind the scenes. I also found it technically intriguing that the pre-3.0 mail reader within Netscape was not written in C++. I have a lot of respect for Mr. Zawinski for being able to identify a potential bias of his – he appeared very introspective when asked about hiring new developers. He understood that he could distinguish people that he could find reputable, but not those who would make good candidates.

One of the things that struck me as a bit off-putting about Mr. Zawinski was his rejection of automatic unit testing. I feel that if it was made as easy in the 90s as it is today, software would be VERY different today.

Brad Fitzpatrick’s interview left me with mixed feeling about the guy. I’m not sure if he is a guy you would want to work with, however he sounds like the kind of guy that you would want to share war stories with over drinks. He has worked on many interesting projects, mainly LiveJournal, and is one of the early “Growth Hackers {http://en.wikipedia.org/wiki/Growth_hacking}.” I like his recommendation that you should spend some time in reading other’s code. He fights the immediate urge to ignore others’ code and his approach sounds different from what I had expected: I expected his approach to making suggestions on other people’s code would be antagonistic. However, it was described as the following:

  1. Code copies are distributed to the audience – in digital and paper form

  2. The developer presents their code line by line

  3. Q&A time

  4. Suggestions / feedback from the audience

This struck me as different from my experience where code reviews tend to be either technically or personally antagonistic (or both). This approach was more similar to proofreading a paper you just made or audience-testing a book you just wrote.

The two things that really put me off about Mr. Fitzpatrick was one of the questions he asks in interviews, and the other is the insistence of knowing everything. Mr. Fitzpatrick’s “famous” interview/programming question was a recycled question from his “AP CS” exam. The question is to write a class that handles large number arithmetic (rewrite BigDecimal). It appears that he uses his previous experience as a baseline for evaluation. I also got the feeling that it is a way for him to “show superiority over others” (over something he did many years ago a high school student). Second, he is incredibly insistent over knowing everything about how every layer works. He ranted against high-level language developers because they didn’t know that a specific way of polling may not work on a specific deployment machine. He even ranted over those who deployed towards a VM because the VM’s “virtual”->native OS/hardware has been abstracted. I feel that in 98% of the cases he’s picking up pennies in front of a steam roller.

I was not very thrilled with Douglas Crockford’s interview. Primarily because it dealt with Javascript, it was a little too high-level for my taste. During the reading of this interview, my mind went back to Mr. Fitzpatrick’s interview. It made me wonder and realize about how you find the “best” tools. I find it incredibly difficult to keep afloat of all the languages and tools available. Recently, for example, I just learned how – and why – Git, Jenkins (plus the automated unit/lint/reporting/checkstyle plug-ins), and deep Maven knowledge are really good things to know if you’re developing in Java.

When new languages, tools, and frameworks come around, I love to read about them and learn how they work (if they’re useful and interesting enough). However, time is limited: how do you identify the tools that would solve the most pressing need you have?  Prior to Jenkins, I built everything via an IDE. Why would I need an automated build tool? I’m the only developer on the project. Prior to Git, I used Subversion – it did everything I needed. Why would I want to make “sub-commits”? Prior to Maven, why would I want to have the build tool automatically deploy the WAR file or require that all unit tests pass before generating an executable? (I’m running unit tests all the time anyway.)

Later it made me think about the code reading suggestion and I realized: I’m not very happy with the code review tools I know about. ReviewBoard looks nice, but that is for only for Ruby. Should I write my own? Where are the existing tools for Java (which can also integrate with Maven and Jenkins)? Are the tools good? Are there others out there that have solved this issue? Is it worth setting up a code review tool just for this? This are questions I’m not sure how to answer.

Overall, I really enjoyed that this book goes over many topics – personal projects, interview questions, and famous debugging stories. I do occasionally enjoy a story bragging how the developer’s language or tool was miles ahead. However after reading about their accomplishments in a serial fashion, it just gets old. Perhaps interspersing their accounts in a more conversational form would have made this book more interesting, and easier to recommend.

Similarities of the individuals who were interviewed:

  1. All have a strong focus on one particular project

  2. Each interviewee has worked in many companies

  3. None of them focused on the reputation of the company they have worked for

  4. All have interesting debugging stories

An Easier Way to Deal with Thread.sleep

If you’ve ever had to use the sleep method you know how “painful” (well a minor annoyance) it is to convert the amount of minutes/hours/days into milliseconds. Well to make this easier, you can use google to do the conversion for you. Granted this is a rather minor thin in the grand scheme of development but it is rather nice to see Thread.sleep(604800000) and then be able to quickly get the answer of 7 days. (Rather than dividing that by 1000*60*60*60*24) To convert the milliseconds, search google with the phrase: “604800000 milliseconds to days” (or hours, etc). You can also reverse the statement to get the amount of milliseconds in a new unit of time. (For example: “5 hours to milliseconds”).

Word of warning: I would never advise another developer to use Sleep for anything longer than a minute or so. The numbers used above were just as an example.

Are all technical books dry and boring?

After reading many technical books I believe that a trend has emerged: Most technical books tend to be varying degrees of being dry. This is starting to become incredibly irritating, it also hurts in keeping the readers attention. At the moment I don’t have a lot of justification to support why non-dry techincal book, however some of the most memorial books  have included silly examples, real world examples, and a non-referential tone.

 

So far the only books I’ve read that didn’t have a dry tone:

1. Test Driven Development By Kent Beck

2. Effective Java- Joshua Bloch

3. Head First For Design Patterns

4. Apache Wicket in Action - Martijn Dashorst

Today I learned: Use %n rather than \n in String.format

When formatting Strings with the Java String::format function, use %n rather than the newline escaped characters (\n). This will produced the platform specific new line during run time. With windows this will return a new line with a carriage feed (\r\n) and with everything else a standard new line (\n).

 

Also another thing to note about this: There is a rule for this in FindBugs. It is called:

FS: Format string should use %n rather than \n (VA_FORMAT_STRING_USES_NEWLINE)

Pastebin for HTTP Requests.

Have you ever had the need to verify an HTTP request? If so the tool “RequestBin” is the way to do it. It is the pastebin of HTTP Requests. When you use the site, it’ll give you a temporary custom URL to post whatever you need to, and it will display the contents when the request comes in.

This is awfully handy for testing Web hooks from Gitlab and Github.

Kegerators

A few months ago I bought a kegerator. (I have now reclassified it as a boat, but more on that later.) A kegerator is an appliance that combines a mini-refrigerator with a kit for dispensing beer from a keg. In reality, it is more of a small refrigerator with a beer dispensing kit, but thats a minor detail. I bought it with the full belief that I could go from setting it up to dispensing beer quickly. The whole system appears fairly simple, CO2 pushes beer out of the keg and you get it at the tap. Occasionally you’ll have to clean it, but that’s not abnormal.

However, it is far from the case. To explain why, let me explain the components and how it works.

A kegerator is more of a plumbing problem than it is a group of interconnected items. With a kegerator, you have a cooling box [the refrigerator], a tank full of gas [CO2 or a Nitrogen mix], a faucet, a keg coupler, and all of the piping between. A CO2 regulator is directly connected to a CO2 tank. A keg coupler is responsible for locking on to a keg securely, and mixing the beer and CO2. It also is the central point for connecting the CO2 and beer line.

Any kind of bad connection between any of the components and you have a problem. If you have a bad regulator, you could be leaking CO2, sending in too little/too much gas, or even cutting off the CO2 in general. If you have bad piping or bad fittings at the ends of each of the pipes, you could be leaking CO2, negatively changing the taste of the beer, or even lose beer [due to a leak]. Another issue with this, without washers/O-rings, you will not be able to create a good connection with the pipes. CO2 tanks have to be checked and tested every 4-5 years for safety. Also to note, the coupler, regulator, and the tank itself have safeties built-in to avoid high pressures, however you don’t want to get to the point where you need these.

After you have those basic ideas down, you still have more work to do. You must ensure the beer lines are cleaned periodically. If a keg is chilled properly, a keg can last months, but if you don’t clean the lines, after a while beer can sit in the line and “go sour.” I haven’t found anywhere else that mentions this, but it is an issue. The keg/beer isn’t the problem; it’s the beer that is in the lines. So flush out the lines by pouring a LOT out. Cleaning a beer line can take quite a bit of time. You have to disconnect the coupler, the gas line and the line that goes up to the faucet. Completely disassemble the faucet mechanism, flush the beer line and coupler with water and then cleaning solution. The parts to the faucet must be scrubbed and soaked in the cleaning solution. Then you have to let the beer line and coupler soak with the cleaning solution for about an hour. This is a lot more involved than washing dishes. Many bars have the beer distributer or a service professional handle this. It’s irritating, but it must be done to consistently serve beer from the tap. A cleaning kit can be bought from Kegworks.

Another issue is CO2 leakage. CO2 is an invisible compressed gas. Its presence is visible via condensation at specific temperatures. For large leaks, you can hear a hissing sound and easily identify where it is. But – it’s extremely irritating when you have a small leak. It could drain a CO2 tank within a day or two. I’ve had this happen twice so far. To check for leaks you can do two different things: a bubble test or a water test. A bubble test is a soapy solution that is sprayed on the full CO2 setup. If the solution bubbles up, the bubbles indicate where the leak is. The other option is to put the CO2 system in a bucket of water, with the tank upside down, and look for bubbles. Please check more official guides on doing this as that turning a CO2 tank upside down can be dangerous. If the lines, regulator, and tank are setup properly, there should be no leaks and CO2 exiting out of the end of the CO2 line. To make things worse, you could have an issue with the CO2 tank. Research the tanks before you get one. If you buy a kit, it’s rarely ever disclosed, recently a popular manufacturer produced a defective tank. Refillers tend, and should, check the tank brand, date, and if it was recalled before refilling. Also to note with kits, cheap regulators are not worth the money that they cost. The keg/soda supply store that I recommend used to have a bucket full of the same cheap regulator. If you buy a kit, you’re going to get a cheap regulator.

Beer leaks: these are a bit easier to identify than their CO2 cousin. If you have excess beer pooling on the keg or the in the refrigerator, you have a leak.

One of the best things that this kegerator kit has given me is the refrigerator that is formed to strap in the CO2 and the hole for the beer tower. However, I feel that I would have done better financially if I had bought the mini-fridge individually and stripped it down. If you do buy a mini-fridge ensure that it can hold a full sized US keg [half barrel]. Also to note, find a good keg distributor before building/buying a kegerator. Most stores won’t hold kegs and require a 7+ day notice to acquire a keg.

All and all, I wish I had built my own kegerator and bought all of the parts. The downside to doing this is that it appears to be a lot more expensive than a kit. However, I believe that I would have had much less trouble than I did.

Why is my kegerator a type of a boat? It is well known that a boat is something someone enjoys the day they buy it and the day that they sell it. It requires more maintenance than expected and requires quite a bit of monitoring.

Some of the products/companies that I strongly recommend:

·         NFC Chicago – These are the guys who deal with keg systems, and do CO2 refills. They’re good people and have been very patient and helpful. They’re local and they will sell you all of the parts, minus the refrigerator to build a kegerator.

·         Kegworks – These guys are an online keg store that sell all of the parts you need. Buy a cleaning kit from them before you tap your first keg.

·         Binny’s Chicago – These guys have a direct line to the beer distributors and can help you get the keg that you want. Total Wine and More may also help you with this, but I’m not sure about how convenient it is.

Installing Maven on Centos 5 or 6/RHEL

At the moment there is no RPM package or yum install available for the latest version of Maven on Centos. The user is left to install Maven manually. To attempt to overcome this, I created a script to install the latest, at the moment: 3.1.1. At the moment, there are many things that should be added to the script, they’re listed in the TODO section of the documentation, but those features may be added later.

Instructions on how to run the script, and the script it’s self may be found at: https://github.com/monksy/centos-maven-install 

Protip of the Day: Always Delay your Email From Being Sent

This is a rather simple and should be obvious pro-tip: Delay all email from being sent. This helps to catch emails that have the wrong tone, message, audience or content before it’s sent. This tip has saved me from sending an badly formed email, or to the wrong audience many times. Just today I caught a work email that was missing information. It makes a difference from sending a brief and context email into sending a fully detailed email.

 

Why does this work really well?

I believe that this works really well because, like unit testing, it causes the author to separate the content creation with the later review. It forces one to reflect on what was created rather than to falsely believe what was written was perfect. It also gives us a chance to give more meaning and promotes a stronger signal to noise ratio in your messages. From my person experience little is gained from responding and sending anything immediately. Delays are a well accepted and expected situation when dealing with human to human communication.

 

That’s nice, but how would I do that?

Apache Wicket MarkupNotFoundException

If you’ve run in to the error:

org.apache.wicket.markup.MarkupNotFoundException: Can not determine Markup. Component is not yet connected to a parent”

Searches online typically don’t yield very much. They tend to give the hint and reiterate what others have found: The HTML file is either misnamed, the resolving component fails to find the corresponding HTML file to the Wicket java class, or that the HTML files are not included in the WAR.
They don’t offer very many solutions. However in my situation, using Maven, the issue was resolved by including resources reference in the build section:

        <resources>
            <resource>
                <filtering>false</filtering>
                <directory>src/main/resources</directory>
            </resource>
            <resource>
                <filtering>false</filtering>
                <directory>src/main/java</directory>
                <includes>
                    <include>**</include>
                </includes>
                <excludes>
                    <exclude>**/*.java</exclude>
                </excludes>
            </resource>
        </resources>