Author Archives: Scott Selikoff

Great Developers Are Not Afraid to Experiment

Posted on by
7

As a moderator on the JavaRanch, I often come across people asking “What would happen if I executed the following code”. Many times the author of such posts can answer the question by copying and pasting his/her code into a Java main() method and running it. Some might chalk these posts up to being lazy, but, clearly, taking the time to write a post on a message board – often signing up as a member for the first time – takes some amount of effort as well. With that, I’m going to be go with the assumption developers avoid experimenting with code because they are scared or unsure of their own knowledge. Besides which, if it is a matter of laziness, there’s not much advice I can give except to say “Don’t be lazy”.

Experiment

Why is experimenting with code can be scary

In my experience as a teacher, development lead, and moderator I often come across developers who are unsure of their own knowledge. Often times they don’t fully understand what it is the code is doing and are afraid to experiment for fear it will either demonstrate their own personal weakness or harm the existing code. To them, I believe that experimenting is most crucial, if only because some of the doubts and questions that linger in their head can often be answered in an surprisingly short amount of time. If you are staring at a piece of code, puzzled by what you do not understand, take my advice: Step back and play 20 questions, ask yourself “What are questions I have about this code that can be answered with a simple yes/no?” then set up test code to answer each question. Once you have your answer, your doubt about the application should vanish, replaced by first-hand knowledge of what is going on. Keep in mind that sometimes these experiments lead to even more questions, but that’s good; it’s part of the learning process. In those cases, perform even more experiments on the code.

What is an experiment?

What constitutes an experiment? Oftentimes, it involves just writing a short line or two of code, then writing a logging statement, or, more commonly, if you don’t have a logger, a System.out.println() statement that displays the value of some variable or object. For example, if you don’t know why a method is behaving a certain way, add a dozen output statements throughout the code so you can follow two things: 1) the path the code is taking and 2) the value of the data throughout the code. Many times, you may be staring at a section of code, wondering why it’s not working, only to find out that section of code is never reached at run-time. Experimenting can be about changing values and recording the inputs, but sometimes its just about outputting where/what you think a process is doing.

Some people will recommend a debugger for experimentation but I’m not one of them. Aside from often being unwieldy and confusing to use, especially for beginners, sometimes running code through a debugger can affect what it outputs. For example, in a server environment, remote debuggers can be especially difficult to use. Services may have transaction timeouts of 30 seconds, and pausing the code in the debugger can cause an exception to be thrown before the method is complete. If you like using debuggers, more power to you, but as someone who has used both output statements via logging tools and debuggers, I greatly prefer the logging tools. Primarily, this is because the output statements give you more of a trial and error structure to work with: either a test succeeded, or it failed, and the output is all there in front of you.

Stand-alone Safe Experiments

The easiest and safest experiments are those you create that are completely separate from any other code. In terms of Java and Eclipse, this is akin to creating a new Workspace and a new Java Project to run the test code in. Every developer should have a temporary, throw-away workspace like this to perform low-level Java tests with. Simply create a file which in some way asks the question you want answered, and execute the program to evaluate the results.

Safe Experiments within Existing Code

Let’s say the code is highly framework dependent, such as often is the cases with J2EE and database-driven applications. For example, creating a temporary workspace to house your test case may be too cumbersome to implement. In such a case, you can run the experiment inside your existing code provided you are careful and follow some general guidelines:

  • If you are working with code that is backed by a repository, such as Subversion, CVS, ClearCase, etc, make sure your experimentation code does not get checked in or you may end up with applications such as this, this, or even the impossible (the last one). The DailyWTF has literally thousands of such cases. It is perfectly fine to experiment, just be careful to cleanup when you are done!
  • If you are working with code that is *not* backed by a repository, then install a developer repository! I cannot tell you enough the power and value of using a coding repository for all development work. In lieu of that, though, you should just make a copy of the project and/or workspace and experiment with the copy, keeping the original intact.
  • If you are working with code that connects to a database, make sure it’s not one other developers use. Making changes to a shared database as part of a test could affect other developers, so, if possible, you should have your own local copy of the database. This does not mean you should have a production database, but merely a copy of a QA or Development database

Final Thoughts
In my experience, one of the things that separates a great developer from the rest of the pack is that the great developer fully understands the code they are writing. When a bug appears (even great developers can cause bugs), this person often has a good idea where to look for the problem right away, and save valuable support time. Ultimately, if you ever find yourself mystified by your own code base, run some experiments and learn why things work the way they work. You’ll be a better developer for it!

The Joy of Null: Continued

Posted on by
Reply

In Part 1 of The Joy of Null I discussed a variety of ways null-equivalent values make it into the software design. Often times, developer laziness or immutability of the database tier drives many developers to insert values that simulate null values, rather than using a database null itself. In this second half, I’ll talk about other ways null-equivalent values arise as well as problems associated with using them.

Null

Part 1: Revisited
The most consistent comment I saw for why some developers choose to use empty strings instead of null values was for performance reasons. As I’ve mentioned on this blog before, database query optimizers are often a loose association of greedy algorithms and indexes, and its certainly possible a DBMS may perform better when a field is an empty string rather than a null, but I doubt its a general rule. In fact, from a theoretical standpoint, testing whether a field is null should be faster than comparing two strings. Consider that testing whether a field is null is a binary test, it either is or it is not null. Alternatively, testing whether a string is empty, or in the more difficult case is a set of blank spaces equivalent to an empty string, requires a string comparison which, depending on how it’s implemented, could be slow.

I think when people consider performance, they might be referring to the fact that no string = null in a database. In order to query on null, you must use “IS NULL” syntax since (x = null) returns false for all values of x, null or otherwise. Yes, having nulls and empty strings in a query could complicate the logic, but your goal should be to remove the empty strings, not the null values.

Part 2: Data Ambiguity
The first problem with null-equivalent values is you may unintentionally have more than one of them. For example, if you consider an empty string to be a null-equivalent value, do you also consider a 1 or more blank spaces null-equivalent? If so, your software will have to formulate SQL queries with syntax such as “trim(widget) LIKE ””, or alternatively “length(trim(widget)) = 0”. Either way, you now have to perform a database function on every string in a table in order to determine if a value is present, whereas “IS NULL” should be a lot faster.

The second problem with null-equivalent values is you may intentionally have more than one of them. I’m going to refer to enumerated null values as a set of null values that all represent null, but might mean slightly different things. As one reader pointed out, you may have a DBMS that supports enumerated null values such as NULL_MISSING, NULL_REMOVED, NULL_UNKNOWN, etc. I have no objection to using enumerated nulls except that very few, if any, major DBMS systems support it due to the fact it would be difficult to get a group of developers and DBA’s to agree on an enumerated set of nulls that would work across all domains. With that in mind, the vast majority of times enumerated null values are used, they are set as strings in the database. Such as Name = ‘NULL_MISSING’. This has all the performance and string comparison problems of my previous argument, but one even worse – someone’s name/data might actually match a null equivalent value. Your system would be a lot more prone to SQL injection if such a thing were allowed and would require constant conversion between the enumerated value and a useful value in the UI, since you don’t want to expose NULL_MISSING to the user. Keep in mind, this includes alternative null-equivalent values such as using -1 or 0 for a positive numeric field. At some point -1 or 0 may be allowed, or accidentally displayed to the user as a real value. No matter how you set it up, enumerated nulls can often lead to bad data such as the issue with little bobby tables

Finally, there are many times null-equivalent values are used side by side with null values leading to unruly queries such as “WHERE widget IS NULL OR length(trim(widget)) = 0″. As most good DBA’s already know, disjunctive searching (using OR) in SQL queries can significantly hurt a query optimizer. Disjunctions are among the most common paths query optimizers will ignore because searching them is not possible in real-time. In short, if you are using an empty string as a null-equivalent value then you should use it everywhere and make the column not nullable. This will at the least simplify your queries and remove the disjunction. There are other ways to formulate the query above without disjunctions, such as using the negation of length > 0, but it still leads to complicated queries.

Part 3: Choose one
I know there are some developers uncomfortable with using database null or “IS NULL” in their queries, so to them I say, at least be consistent. Either use a null-equivalent value, such as an empty string, everywhere or use null everywhere. The ambiguity is when you allow both, which in some data models may mean different things. Overall, allowing both will likely cause a performance hit. As for which you should choose, null or null-equivalent, if you have strong reason to believe your DBMS will handle empty strings better than null values, then go with empty strings. I, on the other hand, will stick with database null values since they should be faster for the vast majority of queries.

The Joy of Null

Posted on by
8

Often in the database world, you do not have all the information needed to create a record. For example, you may have a person’s full name but not their middle name or initial, or you might be missing their date of birth. In such cases, the recommended solution is to fill that field with a special database value referred to as Null. Over the years and for a variety of reasons, I have noticed some database designers and software developers have invented their own equivalent of Null. This article is about how this came about and why the practice should be discontinued.

I’m inventing a new term today called null-equivalent value. A null-equivalent value is a value in a system that is not actually null, but meant to imply null. The most common null-equivalent value is a blank string.

Null

Part 1: What happened to null?

Oftentimes, database designers will (perhaps incorrectly) mark a column as “Not Null”, which translates to: this field is absolutely required for all records and we forbid anyone from inserting a record without this value populated. Some time later, software developers will then build an application on top of the database and realize they are missing some of the ‘required’ information needed to create a record. At this point the developer has one of three choices:

  • 1. Acquire the missing information before inserting the record
  • 2. Modify the database design to remove the “Not Null” attribute on the field
  • 3. Insert a null-equivalent value such as a blank string

Clearly, #1 is the best solution since it was the intention of the database designer that this field be populated, but, as I alluded to, that decision may have been incorrect. It may be that the field is not required, and creating records without this field make sense within the data model. In that case, the developer should consult a database designer and go with solution #2 – to remove the “not null” attribute from the record. Often, though, developers are prohibited from making changes to the database and the process of updating the database is long, vast, and sometimes risky. So the vast majority of developers will go with the ‘quick fix’ of #3, and insert a null-equivalent value since it’s the least work in the short term.

While this is the most common reason null-equivalent values are inserted into the database, it’s by far not the only reason. The second driving force results from a poor mix between the presentation layer and data layer. Assume a developer has the freedom to insert null values into a column, ie, the column is nullable. They may choose to instead insert a null-equivalent value such as a blank strings for a variety of reasons. For example, if a developer outputs the contents of the database directly to the screen, they often prefer to use empty strings rather than nulls since it’s a much more lazy approach – they do not need to convert the null values to empty strings, and can display them directly.

I’ve mentioned two of the main driving forces for null-equivalent values, but there others. I use the term null-equivalent instead of blank strings because developers will sometimes insert “special values” meant to imply null as well as have a secondary meaning such as “Lost Data”, “Missing”, or, in one case, “Null” (yes, these are real examples). In many of these cases, the data model should have been updated to include a boolean column, such as “Lost Data = true”, to explain the reason why the original field is blank. But many developers are nothing if not lazy, and the time required to make a database change, as I mentioned, is great. So, they will overload a column with a special single value, used to imply multiple values, all of which should have been fleshed out in a better database design.

The next article will continue this discussion and consider reasons why null-equivalent values are harmful to the database. In other words: TO BE CONTINUED.