steven k hicks: the exception catcher
An update of your recent plugin. Now has a popup notification when you go to log in.
If you are using a non admin account, you’ll be sent to a page that claims that “You do not have sufficient permission to access this page.” This is despite any of using any of the options that it supplies. To get this box to go away, you must log into the Admin account and select one of the options. The box cannot be ignored, and it prevents you from using the WordPress functionality.
This is written with an audience of producers of software products/services in mind. Please do not claim that your documentation, used to introduce a new user/developer, is good enough to make the user/consumer (developer) proficient in your technology. Producers of software products [the developers, and organizations] are great at writing references to their products, however they generally are horrible at introducing new users to their product. When quick start guides, or introduction tutorials are written for products, the documentation tends to remain in the domain that the creators made. Solution: Hire a reputable technology writer. This will help you in a few ways:
Given these points, it would be easy to justify even a luxurious price on producing high quality introductions / non-referential documentation for your products. Based on these reasons, I conclude it would benefit the business professionals, developers, testers of the product, users, and the marketing department by separating the author him/herself from the general stress of the entire producing company.
Ever had to change a complex literary or rebook a flight via the phone? If you have, then you can relate to how painful that is. If you don’t already have status with the airline, then the process is even more painful with a longer wait prior to interacting with the reservation agent.
When you tell a reservations agent where and when you’d like to visit a location, and how you’d like to end the trip, they have a very complex task to perform. They must put together an itinerary that fits your requests – and is available. After finding that information, they then must read it back to you over the phone. Each literary includes a lot of details – times and information about the flights [connection time, class, connection location, etc.]. Most of the time, you are only told the first flight that fits the parameters, which is rarely ever the best flight available. You’re rarely ever told your other options [unless you request the specific flight numbers and know the availability], because the agent is not motivated to give you all of the available options at once.
I believe the way that one could solve this problem is to convert this conversation from just auditory into a visual and audio experience. I believe that the creation of a small website that represents the session between the customer and the reservations agent, similar to a WebEx meeting, would solve the issue. The page would display all of the options available. It would make users a lot happier about their experience with airlines, and give the flyer more information on how tight or open the availability may be [alleviating much of the pain of irregular operations].
How would this work? The user could open up a web page, and use a short code that is shared between the agent and themselves to access the information being described, the page would be updated based on the agent’s action. This may even work at customer service desks where their reservation desks have to read itineraries to passengers.
How does this differ from the current procedure of online flight changes? The functionality of the flight changes aren’t always allowed or functional on the site. Try changing a round trip ticket to a different date and different location. When the, allowed, functionality of the reservation change site is exceeded, the user is prompted to make the changes over the phone.
Recently I got a sweet new laptop. After being stuck with laptops with cheap builds and parts, this is a welcomed new addition. When converting my previous workstation over to the new one, a few things came about some bad, and mostly good. The awesome bit about the new machine is going from a core duo to a Core i7 processor. My primary operating system has been Gentoo, so installing a new operating system requires a manual install, configuring a new kernel, and building all of the needed secondary applications.
Let’s start off with the annoyances:
$ cd ~/cxgames/bin
$ sed -i -e "s|#![ ]*/usr/bin/python$|#!/usr/bin/python2|" *
$ sed -i -e "s|#![ ]*/usr/bin/env python$|#!/usr/bin/env python2|" *
$ cd ~/cxgames/lib/python
$ sed -i -e "s|#![ ]*/usr/bin/python$|#!/usr/bin/python2|" *.py
$ sed -i -e "s|#![ ]*/usr/bin/env python$|#!/usr/bin/env python2|" *.py
Some nice changes in the Linux environment/Gentoo:
While driving on a long trip this weekend, I had a bit of time to think. One topic that came to my mind was screen scraping, with a focus on APIs. It hit me: screen scraping is more of a problem with the content producer than it is with the “unauthorized scraping” application.
Screen scraping is the process of taking information that is rendered on the client, and then transforming the information in another process. Typically, the information that is obtained is later processed for filtering, saving, or making a calculation on the information. Everyone has performed some [legitimate form] of screen scraping. When you print a web page, the content is reformatted to be printed. Many of the unauthorized formats of screen scraping have been collecting information on current gambling games [poker, etc], redirecting capchas, and collecting airline fare/availability information.
The scrapee’s [the organization that the scraper is targeting] argument against the process is typically a claim that the tool puts an unusual demand on their service. Typically this demand does not provide them with their usual predictable probability of profit that they are used to. Another argument is that the scraper provides an unfair advantage to other users on the service. In most cases, the scrapee fights against this in legal or technical manners. A third argument is that the content is being misappropriated, or some value is being gained by the scraper and defrauded from the scrapee.
The problem I have with the fighting back against scrapers, is that it never solves the problem that the scrapers try to fix. Let’s take a few examples to go over my point: the KVS tool, TV schedules, and poker bots. The KVS tool uses [frequently updated] plugins to scrape airline sites to get accurate pricing and seat availability details. The tool is really good for people that want to get a fair bit of information on what fares are available and when. It does not provide any information that was not provided by anyone else. It just made many more queries than most people can do manually. Airlines fight against this because they make a lot of money on uninformed users. Their business model is to guarantee that their passengers are not buying up cheap seats. When an airline claims that they have a “lowest price guarantee” that typically means that they show the discount tickets for as long as possible, until they’re gone.
Another case where web scraping has caused another issue is with TV schedules. With the MythTV craze a few years ago, many open source users were using MythTV to record programs via their TV card. It’s a great technology, however the schedule is not provided in the cable TV feed, at least in an unencrypted manner. Users had to resort to scrapping television sites for publicly available “copyrighted” schedules.
The Poker-bots are a little bit of an ethical issue. This is something that differs from the real world rules of the game. When playing poker outside of the internet, players do not have access to real-time statistic tools. Online poker providers aggressively fight against the bots. It makes sense; bots can perform the calculations a lot faster than humans can.
Service providers try to block scrapers in a few different ways. The end of the Wikipedia article lists more; this is a shortened version. Web sites try to deny/misinform scrapers in a few manners: profile the web request traffic (clients that have difficulty with cookies, and do not load JavaScript/images are big warning signs), block the requesting provider, provide “invisible false data” (honeypot-like paths on the content), etc. Application-based services [Pokerbots] are more focused on trying to look for processes that may influence the running executable, securing the internal message handling, and sometimes record the session (also typically done on MMORPGs)
In the three cases, my point is not to argue why the service is justified in attempting to block them, my point is that the service providers are ignoring an untapped secondary market. Those service providers have refused to address the needs of this market – or maybe just haven’t seen the market as viable, and are merely ignoring it.
If people wish to make poker bots, create a service that allows just the bots to compete against each other. The developers of these bots are [generally] interested in the technology, not so much the part about ripping-off non-bot users.
For airlines, do not try to hide your data. Open up API keys for individual users. If an individual user is trying to abuse the data to resell it, to create a Hipmunk/Kayak clone, revoke the key. Even if the individual user’s service request don’t fit the profile; there are ways of catching this behavior. Mapmakers have solved this problem a long time ago by creating trap streets. Scrapers are typically used as a last resort, they’re used to do something that the current process is made very difficult to do.
Warning more ranting: with airline sites, it’s difficult to get a very good impression on the cost differences of flying to different markets [like flying from Greensboro rather than Charlotte] or even changing tickets, so purchasing from an airline is difficult without the aid of this kind of tool. Most customers want to book a single round trip ticket, but some may have a complex itinerary that will have them leaving Charlotte stopping over in Texas, then to San Francisco, and then returning to Texas and flying back to my original destination. That could be accomplished by purchasing separate round trip tickets, but the rules of the tickets allow such combinations to exist on a single literary. Why not allow your users to take advantage of these rules [without the aid of a costly customer service representative]?
People who use scrapers do not represent the majority of the service’s customers. In the case of the television schedules example, they do not profit off the information, and the content that they wished to retrieve wasn’t even motivated by profit. Luckily, an organization stepped in and provided this information at a reasonable [$25/yr] cost. The organization is SchedulesDirect.
The silver lining to the battle on scrapers can get interesting. The PokerClients have prompted scraper developers to come up with clever solutions. The “Coding the Wheel” blog has an interesting article about this and how they inject DLLs into running applications, use OCR, and abuse Windows Message Handles [again of another process]. Web scraping introduces interesting topics that deal with machine learning [to create profiles], and identifying usage patterns.
In conclusion, solve the issue that the screen scrapers attempt to solve, and if you have a situation like poker, prevent the behavior you wish to deny.
I saw the article “Four Insights to Selling and Marketing Open Source Software” by Paul Salazar, and I’m a bit surprised. I am not sure that Mr. Salazar can relate well to the customer side of the open source business. In his article, he presents an argument that selling a piece of open source software requires service and a large customer commitment. That is opposed to selling the software as a common off the shelf piece of software.
He is partially right. It does take a lot of commitment, but the issue is not due to the product being open source, it’s more of an issue due to a lack of reputation, and support. Enterprise-ready software is expensive. This is a given. Microsoft, Oracle, HP, RedHat, and other vendors charge A LOT of money for their software. They charge this, because the requirements are more complex for the customer size, fixes-post sale are required, support is required, and a reputation is required. Open source also has the disadvantage to be flexible enough to have frequent changes in the product compared to proprietary software. Many customers look at this issue and are concerned that this may hurt their ability to perform their operations and makes them a little reluctant to use the product. This requires yet another individual to oversee development for an external tool.
Often times open source solutions are better [personal opinion] than proprietary solutions. However, the solution is merely a tool. If an organization is looking to use a tool, they must know who is responsible for maintaining the tool, and where they can find other people that know how to use the tool. Companies are frequently focused on solving the problem within their constraints, not about becoming a fan boy for solution X over Y.
I would not have any qualms with his article, if he had mentioned that the problems that he saw with the business side of open source technologies were very similar to those of new startups. Startups have the same issues of open source software. They do not have the reputation, the ability to scale to the needed level to support the product and are often unprepared for the level of customizations needed for corporate clients.
It is frequently recommended that, readable/”good code,” should closely resemble speech. For example name a variable representing a collection “library” and a method “remove.” From those two items one can have the library remove a book. For example:
library.remove(book);
Services, in an odd situation, are meant to hide the client from the actual networking background, but yet be treated as the neighboring code within the application. Without the proxies and similar method calls, one should just write their own protocol and avoid using webservice. Therefore, I thought it would be a little interesting to do a thought experiment on how one can reference web services.
REST Services ask two basic questions “Can you _____ for me?” and “What do you know about ______?” The subject “you” in this case is a reference to the webservice. “Me” is the service client. Asking if the service can do something for you is referring to REST calls with the POST, DELETE, or PUT action code. Asking for something from the REST service is hinting at the GET [and sometimes post] HTTP action code will be used. With REST running over HTTP all information has to be in Base64 encoding, meaning that there is only one encoding send to the receiving webservice.
SOAP is a little more complex. Scratch that it is a lot more complex. When SOAP was created, the deciding power to be went a little overboard for 90%* of the use cases. SOAP includes information about how the webservice is composed, structure of complex data types, error handling, and the calls themselves. Therefore, there should be a new way to refer to them. Web services can be asked the following questions:
Given any extra extensions to the WS-* specification, more questions may be asked.
* Statistics here are not an accurate measure, more of a wild guess. Nevertheless, it sounds right, that what counts, right?
When considering the importance of APIs, I took a look at the book APIs: A Strategy Guide by Daniel Jacobson and Dan Woods. It didn’t meet my hopes, but it did go on an interesting journey. I was going to write some of my thoughts on APIs, but this book covers them better than I could. When I saw the subject of the book, and the publisher [Oreilly Meida], I had the impression that the book was going to be developer centric and would cover many of the popular APIs [Twitter, Facebook, Weather.com, Reddit, Last.fm/Audioscrobler, Amazon, etc.]. I was sorely wrong. This book is more for selling the idea of creating an API, marketing it to future developers, and discussing the technical problems introduced by an in place API.
What I had hoped: I had hoped that the book would be more of a Rosetta stone of APIs. How to critique an API, how to find weaknesses [it does hint at this, but it wasn’t a serious look into it], optimize usage, and how to potentially adapt software into an API. I can’t qualify myself as an API designer; however I do have some authority on the subject. My experience includes adapting a massively large legacy system into a license controlled API. I can’t go into too many of the details of this as that it was an assignment for work.
While I’m still off subject I should also add to my rant. APIs are notoriously annoying to learn. Every provider has a new way of accessing data, and passing around data IDs [an identifier specifying a user, their credentials to make API calls, etc.]. Providers may include good documentation, some don’t. Some providers make their API incredibly difficult to understand, some don’t. Some APIs have synchronous and asynchronous calls. These technical details were not mentioned, compared, or criticized in the book. I was left wishing that they were. As a developer those details interest me more than how to sell the idea to my boss.
Getting back on the subject, I wish that the book discussed more about the communication channels in which the APIs may be transferred over. The book highly promoted REST, and mentioned SOAP. That’s great, but what about XMPP, in-proc messages, binary web services [Hessian], etc. I finished the book wondering, what about separating the API layer from the actual implementation. It also did not address the issue of multi-language environments. Should you support non-Java based clients, etc. I realize that the answer to that should be “it must be language independent.”But it brings up an important question on how you, the producer, want the API to be used. That is an important consideration if your service/product is mean to be scalable, or if the tasks are intended to take a long time. [Hint: Use a message queuing service].
According to the write-ups on the authors, they have the credentials to talk about APIs from a business sense. But when it comes to actually implementing, I wish they had a subject expert[s] go into more detail. The technical detail and ridiculous amount of depth is what I come to expect from an O’Reilly published book.
Overall, this book is interesting in an architectural sense. However in a developer perspective, it’ll give you things to argue about in the planning stage, but won’t help out later during implementation. Want to give your developer users an API for your product? Give your manager this book, and soon you’ll be asked to write an API.
I have a bone to pick with the industry I associate with. The tech industry struggles to clearly define the expectations of whatever is desired. This is pretty much a universal issue with the industry. There are always hidden requirements. Job listings are no different.
The bone I have to pick is with the titles/job requirements. One of the worst offenders of this is for a Java Software Engineer. Based on the availability of frameworks, and meta-frameworks, asking for a Java engineer is quite ambiguous. This could ask for a graphics developer, API designer, web developer, Computer vision expert [CV in java is possible with native libraries, trust me!], core libraries developer, or even a micro-JVM developer. The amount of variability with the language makes a generic “Java developer” title frustrating.
As a potential candidate, it is incredibly frustrating to see the “Java Developer” title. Some titles ask for experience in Spring, Hibernate, Solr, Lucene, JAI, J2EE, etc. It’s incredibly frustrating to go into an interview, where the listing asked for all of these and then only be grilled on the minute [rarely used] inner workings of Spring RMI. Just the Spring framework alone asks for a lot. Of all of the potential avenues of spring that one could master are: Message queuing, Web services, Roo, Security, Integration, Web flow, MVC, BlazeDS, Batch, Social, and mobile. That’s not even accounting for the frameworks that you can substitute between the pathways for Spring. I’ve worked with a late Spring 2.x and early 3.0, I was not even aware of the new BlazeDS, Batch, social, or even mobile options for the framework. Things change quite quickly.
Besides ranting, what is the purpose of this article? I believe that the Java title should be a bit more specific. If you want a Java developer that knows a lot of frameworks, still label it as a Java developer position. However, don’t expect him or her to know all of the inner workings: that is just silly. If your business involves multimedia display, request a developer that knows the Java 2D and 3D graphics, and maybe the JAI libraries. There is a lot there for practically any task you want to throw at Java, given you know which library to use.
Scientific or mathematical tasks, ask for a Scientific Java Developer. What libraries should they know? Colt, EJML, JAMA, etc.
Writing a Java API? Ask for a Java API Designer. Expect for them to list their favorite APIs, what works, what doesn’t, and why.
Web applications? Ask for a Java Web Developer. Maybe they should have experience with a SOAP framework, MVC framework, Play, GWT, basic Web skills, and maybe even a non-SOAP based webservice library [REST, Hessian/Binary]. Please don’t use the enterprise title unless you need someone who knows EJB, and/or ESB frameworks.
Moreover as an employer, be more specific the first interview/inquiry on what the job is asking for. If an job application asks for the common ORM Hibernate, how much should one know about it before applying for the position? Should they be able to know how to wire up beans to their DB analog? Should they know how write their own dialect for a new data source? With JUnit, should the applicant be expected to transition your current development methodology to TDD? Should they know how to extend the JUnit framework?
At this point, you should be getting the picture. Different tasks demand different skills. Specify what you’re looking for. Find people that can learn new skills and that are interested in the same problems you are. People who have an active interest can learn what is needed to get up to speed. If you go to a butcher and ask for meat, you’re always going to get what you asked for, but not what you were craving.
Addendum: This can apply to an Erlang, C++, Python, and most other developers. The role of Java currently has such a great demand, and such an ambiguous title, making this article a little easier to communicate.
After watching the video “The Web Will Die When OOPs dies” and reading a few hip articles, on “how to improve your code,” gave me an idea. If someone is writing to give advice on improvements they should provide real world examples to why their suggestions/fixes to a language/framework are necessary.
Instead of articles of “Top 10 ways to improve your code;” I’d rather see an article about how something was improved. Take small segments from your own application or an open source project and write up a small report on how your suggestion[s] improved it. If you’re writing an article for developers, we’re interested in the proof and technical content. If your suggestions didn’t work out, we’re still interested in the story about the journey, or the technical observations that you found. Making vague suggestions and recommendations (without evidence) bore a technical audience. When you make vague suggestions to someone, you’re making an statement that you know more about their work or problem than they do. Your audience should be intelligent, if not then still write for an intelligent audience. Vague statements don’t address interesting problems, or new approaches.
So going back to the video I mentioned earlier. I like his tactless approach. He’s taken a good deal of experience from web development and he is presenting his argument. He doesn’t attempt to ask for others for opinion, nor does he attempt to tell his audience how to do their job. He just merely presents issues that he found, and his way of trying to solve those. The last thing that I loved about his talk is that he never once tried to push the latest hip language or framework. He mentioned a few frameworks, and then made a critism.