Michael's profileMike's BLOGPhotosBlogListsMore Tools Help

Blog


    July 23

    Testing Theories and Assumptions


    Reading a fascinating essay titled "The Practice of Theories: Adding 'For-all' Statements to 'There-exists' Tests" written by David Saff and Marat Boshernitstan..... The problem space is one of my favorites...automated testing. The essay begins by discussing the limitation of traditional unit tests
    Traditional unit tests in test-driven development compare a few concrete example executions against the developer’s definition of correct behavior. However, a developer knows more about how a program should behave than can be expressed through concrete examples. These general insights can be captured as theories, which precisely express software properties over potentially infinite sets of values. Combining tests with theories allows developers to say what they mean, and guarantee that their code is intuitively correct, with less effort. The consistent format of theories enables automatic tools to generate or discover values that violate these properties, discovering bugs that developers didn’t think to test for.
    This problem space.....of having a testing domain where there are many many possible data values....but only a few concrete unit tests .....is one that I have seen before. One approach to this problem that I found earlier that looked interesting is called Pair-wise testing ....and is a very clever approach for narrowing down the seemingly near infinite combinations in a reasonable manner; but, this approach looks even better...as it doesn't narrow down test coverage as with pair-wise testing, logical as that may be....it's an approach where you express expected functionality as theories and assumptions ....and then you test your theories. Fascinating! Here is an example of a testing theory from this website called Popper....
    @Theory public void multiplyIsInverseOfDivideWithInlineDataPoints(
    @Between(first=-100, last=100) int amount,
    @Between(first=-100, last=100) int m
    ) {
    assumeThat(m, not(0));
    assertThat(new Dollar(amount).times(m).divideBy(m).getAmount(),
    is(amount));
    }
    How you supply data to your testable theories....I think there are several approaches. The example above uses parameter suppliers, as defined by the annotations. There are existing parameter suppliers, or you can write your own custom parameter suppliers for your own specific testing needs. But also, I think once your test is defined this way, it is exposed or open to custom JUnit test runners for exploratory testing or other approaches....and with a little googling I think this is the approach that Agitar may use in their commercial test runner called Agitator.

    Imagine improving the effectiveness of a simple JUnit test 10, 100, or even 1000-fold with just a few annotations and a few lines of code!

    Or so goes the theory.....
    January 24

    Gapminder

    GapMinder is an interesting presentation of life expectancy data by country and region over time....  What will the people at Google think of next.

    March 08

    Testing on Rails

    While Ruby on Rails is heralded for its development productivity in rapidly creating web sites its features of code-generation, persistence layer, language simplicity, use of MVC architecture, ease of inclusion of AJAX and web services grab all the attention.  What catches my attention is the built-in unit and functional testing frameworks.  Indeed, as you use Rails to generate your application, you unwittingly generate test artifact stubs.  How sweet is that!
     
    When you create an MVC Rails application, you also create unit tests on the models, and functional tests on the controllers.  You get infrastructure for test fixtures, which in the database-centric MVC app is a database loaded to a known state.  Support for mock objects.  A tool called 'Rake' for running tests and tasks such as creating test databases. That's not all....you get code coverage statistics, albeit not at as glamourous as Clover. 
     
    And you get information on performance!  Performance is so often neglected totally, or done only post-development when it is difficult and expensive to improve.  Integration of performance tests early into the development process is something that I have been a huge advocate of with tools such as JUnitPerf.  Creating performance assertions in the same way as creating functional assertions is a great way to catch performance regressions as they are introduced, not 3 months later.   And you get a performance benchmarker for benchmarking any method in your application.
     
    Having worked as a QA Engineer, Performance Engineer, JUnit Test Developer, and Test Framework Architecht in the Java/J2EE space, I know the huge positive effect that robust, integrated, high code-coverage, high regression testing testing can have on a development process.  In this regard, Ruby on Rails really brings the horse to the water.  It is just up to the Ruby developers to drink.
    March 07

    Ruby on Rails

    Holy mackeral....I am reading the book Agile Web Development with Rails by Thomas and Hansson.  Two things really jumping out at me:
     
    • this is very easy
    • this is very powerful
    What a great website development approach! 
     
    This is going to lead me into learning Ajax I see. 
     
    February 28

    Using Little's Law

     
    As I was reading about performance testing of Java and J2EE systems I came across Little's Law.  It seems simple enough:
     

    The average number of customers in a stable system (over some time interval) is equal to their average arrival rate, multiplied by their average time in the system.   [from Wikipedia: http://en.wikipedia.org/wiki/Little's_law]

     

    So in non-computer terms, if 10 people entered a store each hour and each person stayed in the store for .5 hour, then the number of people you could expect to find in the store is 5.  10 * .5 = 5.  OK, simple enough.  It all just seems like the commutative principal of multiplication.  What's the big deal?  Well, the big deal is when you start flipping the equation and making practical applications to understanding your performance tests and your application server configuration.

     

    One application of Little's Law is validation of your performance tests, as described in the article Software Infrastructure Bottlenecks in J2EE, written by Deepak Goel.  Little's Law is used here to validate the test ensuring that there was not an unknown limitation of the testing tool. 

     

    Another potential application is sizing the JDBC Connection Pool.   I have heard the question asked, "How big should my JDBC Connection Pool be?"...and the common answer is something like "Big enough so that you never run out of connection."  While that is true, Little's Law could be easily applied here.  Say under a particular load there are 10 SQL statements issued each minute.  And the average time of these SQL statements is 30 seconds (.5 minutes).  Then according to Little's Law the JDBC Connection pool should be 5.  In reality you may want to make it a little bigger....but it is better than wild guesses like "I set the JDBC Connection Pool to 50, and it hasn't maxed out yet so I will just leave it there."  

    Similarly, one could size JMS queues, the number of HTTP Connections, etc... could they not?  Other uses perhaps in optimizing Oracle? 

     

    Something to think about the next time you do something related to queues, pools, or performance.  And with J2EE technology, that might be sooner than you think.

    February 22

    Software Developers Gone Wild

     This is Steve Ballmer, Microsoft CEO, pitching Windows in a 1986 TV advertisement.  I think he has drank the proverbial Kool-Aid.



    Courtesy of IFILM
    February 21

    The Inventor of CTRL-ALT-DEL speaks out

     

    CTRL-ALT-DEL
    David Bradely, inventor of the Windows CTRL-ALT-DEL function, describes the origins of the shortcut and makes a joke at the expense of Bill Gates. All of you nerds are going to laugh it up at this one.


    Courtesy of IFILM