One of the differences between testing commercial-off-the-shelf software (COTS)and a hosted service is the time dimension.  With COTS, you are answering the question:  Does the software work as expected?  With a hosted service you are answering that same question, but also additionally answering the question:  Does the software continue to work as expected?   Which brings me to one of my favorite topics — monitoring.  Monitoring is the activity that needs to be performed to ensure that quality of hosted services is achieved every day.  I prototyped a web service monitoring approach that I thought I’d share.  It uses Soap-UI, Maven, and TeamCity.

Engineers that develop or test web services are usually familiar with a nifty tool called Soap-UI.   The tool is really useful with ad-hoc exercising of web services.  The tool helps form Soap requests that can be sent to a web service, can send the requests, visually display the responses, develop test cases, assert Soap responses, etc….

Building on this, there is a Maven plug-in for Soap-UI.  Maven, the Ant-like tool for building Java projects, has a plug-in which can run the Soap-UI tests.   To do this, you have to write a pom.xml …which is very simple.  The pom will download all dependencies, including the Soap-UI tool itself, and run the tests that you defined through your interactive use of Soap-UI.  The pom looks like this ….

Once you have a Maven pom, you can then run your tests consistently from the command line …. with a simple Maven command:

mvn eviware:maven-soapui-plugin:test

When you execute this command, your tests run automatically.

But you don’t want to be responsible for running the tests all the time.  This is where TeamCity comes in.  You add your project to TeamCity, and with cron-like precision, your tests will run…every hour, every day, monitoring your web service endpoints over time to make sure they behave as expected.

And not only does TeamCity run your tests, it has all the email notification of test passing and failing built-in.

Build successful x_monitoring_test::x_monitoring_test #15.134888 – Build results: http://toolsbuildserver:8111/viewLog.html?buildId=269528&buildTypeId=bt324

So if the  web service endpoint  misbehaves, you can proactively address the issue.

Monitoring ….quality over the time dimension.

These are just my notes on listening to Martin Fowler present his thoughts on "Evolutionary Software Design" ....

• What is the relationship between Agile, Extreme, etc <<<<----------->>>>Design, Architecture ?
• Design Stamina Hypothesis
  • good design promotes the ongoing addition of new features and change
  • no design may be quicker at the start
  • where is the payoff threshold?  it is in weeks ....not months
• Design and Architecture are to manage complexity
• Two types of complexity
  
  • Essential
  • Accidental
• Design mitigates eliminates accidental complexity
• Design controlls and organizes essential complexity
• Technical "debt" as a metaphor for complexity
  • pay interest on your complexity
  • like being in debt
• Two schools of design
  • Planned Design
  • Evolutionary Design ...Fowler is in this camp ...iterative...design and development together
• Important role of Continuous Integration and Test (Yeah!!!)
• YAGNI (you ain't gonna need it) in opposition to Planned Design
• Priorities ...from high to low
  • tests pass
  • code expresses intent
  • no duplicate code
  • least code
• Designs emerge from eliminating duplicate code
• What is Architecture?
  • The highest level concept of a system
  • Major components and interactions
  • stuff that is hard to change
  • THE IMPORTANT STUFF (whatever that is)
• Last Responsible Moment
  • defer decisions to last moment
    
  • late decisions
• The Role of Architect
  • Technical leader
  • Involved
  • Mentor
  • Programmer ...hands-on
  • 
    

A recent software failure known as #AmazonFail has lit up the Twitterati these past few days. And as with all bugs, I find it fascinating....

To start with the basics of #AmazonFail ....

• the "#" pound-sign is how people began tag their tweets.
• Amazon is the online retailer of books, of course.
• Fail ....means problem. There's a whole "Fail" phenomena of images on the web of spectacular and funny failures.

According to Janet D. Stemwedel, Phd and author of the blog Adventures in Ethics and Science


Those impacted ... de-ranked authors ... began tweeting. And tweeting. And tweeting. And with Twitter, with all the immediacy it brings to communication and issues ...the issue escalated into a Severerity 1 bug yanking Amazon employees away from Sunday dinner. And the issue seems on a trajectory to enter the category of web folklore and uber-bugs that are known by all who test software. There are so many fascinating aspects to this bug and its aftermath. But here's a few things that fascinate me...

1. It was a bug in data, not in code. In #AmazonFail, a flag was set changing some data, which changed the behavior of a program. At previous places of employment I have heard debates on the issue of data. "Is data part of the system under test?" To the effect, "It's only a data issue, the code is fine." And apparently, according to this blog post ...the data in the catalog becomes "broken" all the time. Ok....just data.
2. The bug affected ...not Amazon and it's transactional data...but it effected "The Community" of partners and authors dependent on Amazon's ranking system. Amazon may not have had a direct vested interest in the integrity of the ranking system, but others did. And this ranking system is based on user-generated data ...comments and ranking.
3. The perception of quality. Only a mere 57,000 of the millions of items on Amazon.com were affected. But the negative perception of quality and fallout is much larger.
4. The handling of the issue with the public seems to have left many dissatisfied.

So did people over-react. Probably. Was it just a minor data issue? Yeah, not a particularly fascinating root cause. But still...a fascinating software bug ....

At work they are having a contest called "Mathworks Idol". To enter, you submit a humorous MATLAB or The Mathworks-related video. Here's my submission...   

First there was The Joel Test: 12 Steps to Better Code .... a good article by Joel Spolsky where he articulates 12 steps to better code...

The Joel Test

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

As I read this I found myself nodding "Ok, ok, good, ok..." through this article.

And in response to this essay, apparently many people suggested adding a 13th step ... "100% unit tests of all your code." Yeah! Seems like a good goal. But in follow-up Podcast 38 Joel Spolsky takes the bold stand to argue against against this additional 13th step... but also responds "yeah" to this ...

And I find, sadly, to be completely honest with everybody listening, quality really doesn't matter that much, in the big scheme of things... There was this quote from Frank Zappa: "Nobody gives a crap if we're great musicians." And it really is true. The people that appreciate Frank Zappa's music aren't going, "that guitar was really off." They're hearing the whole song; they're hearing the music,

Oh my. Is he off his rocker and uttering some sort of software blaspheme in a weak moment? Or is he saying something else? So I am rereading this article very carefully to see if I can figure out what he is saying. So here is Joel's first argument against 100% code coverage ...

...if a team really did have 100% code coverage of their unit tests, there'd be a couple of problems. One, they would have spent an awful lot of time writing unit tests, and they wouldn't necessarily be able to pay for that time in improved quality.

To summarize, I think he is saying there is a point of diminishing returns on code coverage as it approaches 100%... Here's Joel's second argument against 100% automated unit test coverage ...

...that the type of changes that you tend to make as code evolves tend to break a constant percentage of your unit tests. Sometimes you will make a change to your code that, somehow, breaks 10% of your unit tests.

Hmmm....having unit tests in place allow you to refactor the class-under-test with confidence knowing that the behavior is not inadvertantly changing. And yes, if you change the behavior of your classes-under-test, then yes...unit tests will break. Unit tests are supposed to enforce a consistent behavior in the class under test. Of course in a test-driven environment, the developer would be changing the test first, and then changing the class-under-test so that the test would pass.

as your project gets bigger and bigger, if you really have a lot of unit tests, the amount of investment you'll have to make in maintaining those unit tests, keeping them up-to-date and keeping them passing, starts to become disproportional to the amount of benefit that you get out of them.

So yes, there is a cost to maintaining unit tests if you are going to change the behavior of the class-under-test. But isn't the cost equation on automated testing one of high initial cost in developing them, and then reap the rewards as they keep running for 'free'. So then he recognizes the "cost" of maintaining unit tests is highest when there is a lot of churn...

To me it's about churn.

True. But when there is the most churn...isn't that also where the benefit is also greatest? So then comes this ...

I might do more black-box tests [instead of unit tests..]

And by this they mean not unit tests ... but integration tests covering use-cases or workflows ...I assume. Okay, I sort of agree here. Unit tests are not the end all of automated tests. You also want higher level integration tests ..or GUI automation tests. The benefit to these sorts of automated tests is that they often cover a lot more code, they often more closely match what the customer will do, they often more closely match business requirements, etc... But black box and higher level integration tests are costly too...when they fail, you often don't know where exactly the failure occurred. Next up in the essay...is I think a gaff on his part where he argues against programming to an interface. And then another gaff...

And I find, sadly, to be completely honest with everybody listening, quality really doesn't matter that much, in the big scheme of things... There was this quote from Frank Zappa: "Nobody gives a crap if we're great musicians." And it really is true.

Okay, I respectfully disagree with that. Is he really trying to say, albeit poorly: "Customer-perceived quality may not be directly proportional to unit test code coverage"? Or ...is this some version of the Time, Quality, and Cost Triangle: .... with Joel favoring cost and time ...at the expense of quality (unit tests)? Or is it that he thinks 100% unit test coverage is not practical (untestable code, diminishing returns, etc...)? This latest essay by Joel Spolsky just left me scratching my head....

Interesting approach to load and performance testing of websites with BrowserMob.... It leverages the growing-in-popularity Firefox Selenium extension....and then you upload your script to BrowserMob. At BrowserMob, you schedule your test, define how many browsers you want to execute your script, what types of browsers, ramp up times, etc... It's inexpensive and pay-as-you-go.   BrowserMob Overview (HD) from BrowserMob on Vimeo.

Interesting use of EasyMock yesterday: a co-worker is developing a web service client to communicate with an endpoint outside of the build-and-test system. Because the endpoint may or may not always be available, from a testing perspective I consider the endpoint a volatile external resource.   So I want to test the web service client without actually communicating with the endpoint.

So the class-under-test here is the web service client....SegvWebservice. Like all Axis2 clients, it uses a "stub". This stub is automatically generated by WSDL2Java, and the stub's job is to bind to the web service endpoint. For unit testing in the build-and-test system, the trick is to replace the stub with a mock that is created by EasyMock. Then this mock needs to be injected into the class-under-test. I learned quite a bit on this unit test....which I will share.

My understanding of "dependency injection" is still very much in its infancy. I became acquainted with the topic when, as I try to become a better unit test writer, I found some classes that are hard to unit test. One reason for a class being hard to unit test is that when you new it up, you either can't...or that when you do new it up it has dependencies that you can't control....ie, the class wasn't designed for testability. So dependency injection is the conceptual answer to that problem.....injecting a class with its dependencies....and my original understanding of dependency injection was constructor injection...passing in dependencies as arguments to the constructor....and at test time...sending in a mock instead of the real thing. Then, through defining different behaviors on the mock...you are able to traverse and verify different code paths in the class-under-test.

But with the Axis2 auto-generated web service class here, I didn't do constructor injection.  I added a 'configure' method that sends in the service details...either the WSDL2Java auto-generated stub ....or for unit testing...it sends in the mock I created with EasyMock. So this is a different type of dependency injection...setter injection. So this new type of dependency injection...setter injection...is something I learned on this test.

I also learned more about EasyMock. First, reinforced how useful this tool is ....to create an entire implementation of ANY interface...including a web service endpoint... in 1 line of code. Unbelievable. Second...this was the first time I've seen a webservice endpoint mocked in this fashion.

    @Test
    public void testServiceWithEasymock() throws Exception {
        ServiceInterface myServiceInterface = createMock(ServiceInterface.class);
        SendReturn res = new SendReturn();
        res.setResult(SUCCESS_RESULT);
        expect(myServiceInterface.send(isA(SendRequest.class))).andReturn(res);
        replay(myServiceInterface);
        ServiceUnderTest srv = ServiceUnderTest.bindToWebServiceNow("",tcp);
        srv.configure(myServiceInterface);
        srv.send(new SendRequest());
        verify(myServiceInterface);
    }

This is a pretty basic example, but I am pretty sure I can expand on this concept and what I've learned on future testing challenges.

Very good video on scrum ....
   

Ever want to get some quick Java performance profiling information, but don't want to go through the hassle of configuring a tool such as JProbe? Well, I found a neat little switch that helps.....

-Xrunhprof:cpu=samples,file=c:\temp\hprof.txt

Just put that on the command that starts your java process. This creates a file that shows you a sorted list of methods by CPU utilization percentage descending...

CPU SAMPLES BEGIN (total = 92947) Thu Dec 11 16:26:57 2008

rank   self  accum   count trace method

   1 61.57% 61.57%   57231 300990 java.net.PlainSocketImpl.socketAccept

   2 17.77% 79.34%   16514 301195 java.net.SocketInputStream.socketRead0

   3 15.85% 95.19%   14735 301217 java.net.SocketInputStream.socketRead0

   4  0.52% 95.71%     482 300322 java.util.zip.ZipFile.read

   5  0.34% 96.05%     314 300619 java.io.FileOutputStream.writeBytes

   6  0.24% 96.29%     219 300450 java.io.WinNTFileSystem.getBooleanAttributes

   7  0.19% 96.48%     178 301339 java.net.PlainSocketImpl.socketConnect

   8  0.19% 96.66%     174 300475 java.io.WinNTFileSystem.getBooleanAttributes

   9  0.17% 96.83%     156 300465 java.io.WinNTFileSystem.getBooleanAttributes

  10  0.16% 97.00%     153 300521 java.io.FileInputStream.readBytes

  11  0.16% 97.15%     146 300106 java.lang.ClassLoader.findBootstrapClass

  12  0.15% 97.31%     142 300478 java.io.WinNTFileSystem.getLastModifiedTime

  13  0.09% 97.39%      81 300476 java.lang.ClassLoader.defineClass1

  14  0.07% 97.46%      65 300791 java.io.FileOutputStream.writeBytes

  15  0.07% 97.53%      63 301024 java.io.FileInputStream.readBytes

  16  0.06% 97.59%      54 300209 java.util.zip.ZipFile.getEntry

  17  0.05% 97.64%      46 300229 java.io.WinNTFileSystem.getBooleanAttributes

  18  0.05% 97.69%      46 300650 java.io.FileOutputStream.writeBytes

  19  0.05% 97.74%      45 300723 java.io.FileOutputStream.writeBytes

  20  0.05% 97.78%      43 300345 java.lang.ClassLoader.findBootstrapClass0

  21  0.04% 97.83%      40 300448 java.io.WinNTFileSystem.canonicalize0

  22  0.04% 97.86%      35 300442 java.io.FileInputStream.readBytes

  23  0.04% 97.90%      33 300108 java.io.WinNTFileSystem.canonicalizeWithPrefix0

  24  0.03% 97.93%      29 300581 java.util.zip.Inflater.inflateBytes

  25  0.03% 97.96%      29 300594 java.net.URLClassLoader$1.run

  26  0.03% 97.99%      27 300461 java.io.WinNTFileSystem.getLastModifiedTime

  27  0.03% 98.02%      25 301268 java.io.WinNTFileSystem.getBooleanAttributes

  28  0.03% 98.05%      25 300613 sun.misc.Launcher$AppClassLoader.loadClass

  29  0.03% 98.07%      25 300599 java.lang.ClassLoader.loadClass

  30  0.02% 98.10%      23 300681 java.sql.DriverService.run

  31  0.02% 98.12%      23 300558 java.net.URLClassLoader.findClass

  32  0.02% 98.15%      22 301200 sun.rmi.transport.DGCClient.registerRefs

  33  0.02% 98.17%      21 300849 java.io.RandomAccessFile.readBytes

  34  0.02% 98.19%      21 301278 java.io.FileInputStream.readBytes

  35  0.02% 98.21%      21 300560 java.net.URLClassLoader$1.run

  36  0.02% 98.23%      19 300431 java.util.zip.ZipFile.open

  37  0.02% 98.25%      18 300447 java.io.WinNTFileSystem.getBooleanAttributes

  38  0.02% 98.27%      18 300463 java.io.WinNTFileSystem.getBooleanAttributes

  39  0.02% 98.29%      18 301140 java.io.WinNTFileSystem.getLastModifiedTime

  40  0.02% 98.31%      18 300015 sun.misc.URLClassPath$JarLoader.parseExtensionsDependencies

  41  0.02% 98.33%      17 300449 java.io.WinNTFileSystem.getBooleanAttributes

  42  0.02% 98.35%      17 301158 java.io.FileInputStream.readBytes

  43  0.02% 98.37%      17 300355 java.text.MessageFormat.makeFormat

  44  0.02% 98.38%      16 300580 java.lang.Throwable.fillInStackTrace

  45  0.02% 98.40%      16 300481 java.lang.ClassLoader.loadClass

  46  0.02% 98.42%      16 300573 java.beans.Introspector.getTargetPropertyInfo

  47  0.02% 98.43%      16 300479 org.apache.catalina.loader.WebappClassLoader.findResourceInternal

  48  0.02% 98.45%      16 300559 java.net.URLClassLoader.findClass

  49  0.02% 98.47%      15 300652 java.io.WinNTFileSystem.getBooleanAttributes

  50  0.02% 98.48%      15 300750 org.apache.naming.resources.ProxyDirContext.lookup

  51  0.02% 98.50%      14 300217 java.io.WinNTFileSystem.canonicalize0

  52  0.02% 98.51%      14 300718 java.io.FileInputStream.readBytes

  53  0.02% 98.53%      14 300451 java.io.WinNTFileSystem.checkAccess

  54  0.02% 98.54%      14 301141 java.io.WinNTFileSystem.getBooleanAttributes

  55  0.02% 98.56%      14 300611 org.apache.catalina.loader.WebappClassLoader.findResourceInternal

  56  0.02% 98.57%      14 300492 sun.nio.cs.ISO_8859_1.newDecoder

  57  0.01% 98.59%      13 300586 java.lang.Throwable.fillInStackTrace

  58  0.01% 98.60%      13 300445 java.io.WinNTFileSystem.getLastModifiedTime

  59  0.01% 98.62%      13 300458 java.util.zip.Inflater.inflateBytes

  60  0.01% 98.63%      13 300649 org.apache.naming.resources.FileDirContext.lookup

  61  0.01% 98.64%      12 301142 java.io.WinNTFileSystem.getBooleanAttributes

  62  0.01% 98.66%      12 300462 java.io.WinNTFileSystem.checkAccess

  63  0.01% 98.67%      12 301201 sun.rmi.transport.DGCClient$EndpointEntry.<init>

  64  0.01% 98.68%      12 300850 java.io.RandomAccessFile.read

  65  0.01% 98.69%      12 300226 java.util.Hashtable.entrySet

  66  0.01% 98.71%      12 300138 java.io.WinNTFileSystem.getBooleanAttributes

  67  0.01% 98.72%      11 300473 java.io.WinNTFileSystem.getBooleanAttributes

  68  0.01% 98.73%      11 300502 sun.security.jca.ProviderConfig.getLock

  69  0.01% 98.74%      11 300869 java.util.Collections.synchronizedList

  70  0.01% 98.76%      11 300011 sun.misc.URLClassPath$JarLoader.getJarFile

  71  0.01% 98.77%      11 300638 java.util.Collections.unmodifiableMap

  72  0.01% 98.78%      11 300497 sun.security.jca.Providers.<clinit>

  73  0.01% 98.79%      10 300616 java.lang.Throwable.fillInStackTrace

  74  0.01% 98.80%      10 300252 java.lang.Class.getDeclaredConstructors0

  75  0.01% 98.81%      10 300510 java.lang.Class.forName0

  76  0.01% 98.82%      10 300579 java.net.URLClassLoader.findClass

Some other permutations on this too that help show memory leaks....

Task of the morning ....conduct some light loadtesting of a Flex application. I've done performance and loadtesting work before, and usually much depends on a sound methodology: establish SLAs, develop realistic usecases, controlled environment, data setup, etc... Usually the effort you put into your methodology pays dividends, in fact...meaningful results require it. Take shortcuts, and the results aren't solid, or the results aren't accepted, or they are challenged. However, this loadtest is nothing like that....just click a button on a Flex app.....10 users....see what happens.

So in one regard, this is easy. But there is the issue of Flex. Flex, Flex, Flex....How to test Flex....Selenium won't do it. What tools will? Well, I typed "opensource load testing +flex" into Google....bingo. Webload by Radview. Opensource as it is, Webload's commercial roots are fairly apparent. Documentation. Ease of use. And it is up-to-date with its support of Flex, which many other opensource tools do not support. Like the screenshot show...really easy as 1-2-3:

1) Create and Edit scripts. As easy as record and play. Apparently it uses Jython as a scripting language. Never used Jython...and didn't have to...this is just record and play...click a button...10 users...see what happens. But it was nice to know the script was recorded with a known language. Maybe someday I will need to tweak those scripts. Played the script back a few times to make sure it was doing what I wanted. No issues.

2) Creating and Running Load tests. You use your recorded scripts. I set it up for 10 virtual users. Noticed there is functionality to distribute these virtual users across other machines....nice if you have a real lot of VUs...but not needed for this little load test. Noticed there is some functionality to configure ramp-up of VUs...must be gentle to the system-under-test. Some nice wizards to get you going quickly. Some terminology with "templates" and "agendas" ...the ability to mix use-cases. Nice, but not needed now. Impressed by the ability to show realtime metrics of the machine-under-test...CPU %, etc... And some nice realtime graphics. There was a "throttle" knob gradually increasing the VUs and seeing the response...that was cool.

3) Analyze and create reports. I maybe didn't spend enough time in this section because it looked a little rudimentary to me. Sometimes it takes some good reporting to glean meaningful results out of the test. But...I didn't have to spend too much time with reports....the results of this load test were clear -- the application stopped functioning. Bingo. Note to self: revisit reporting and analytics of Webload.

I downloaded a whitepaper from Radview titled "why-load-test-flex-applications?" No need to sell me on the need to performance and loadtest server applications, that is like singing at the choir; but, I was curious what it had to say specific to Flex... Okay...some of that is not Flex-specific...but some is. Note to self: If an application has multiple UIs...ie HTML and Flex...not sufficient to just loadtest HTML UI and skip the Flex UI. Should do both.

Not sure why it took me so long to try this, but I am using Selenium IDE for the first time today.   One word summary:  wow.

Says their website:

Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE includes the entire Selenium Core, allowing you to easily and quickly record and play back tests in the actual environment that they will run.

Selenium IDE is not only recording tool: it is a complete IDE. You can choose to use its recording capability, or you may edit your scripts by hand. With autocomplete support and the ability to move commands around quickly, Selenium IDE is the ideal environment for creating Selenium tests no matter what style of tests you prefer.

Features:

• Easy record and playback
• Intelligent field selection will use IDs, names, or XPath as needed
• Autocomplete for all common Selenium commands
• Walk through tests
• Debug and set breakpoints
• Save tests as HTML, Ruby scripts, or any other format
• Support for Selenium user-extensions.js file

Play and record GUI tests!

Geez ...and if that is not enough...Selenium RC lets you run your tests on multiple browsers and on multiple platforms.  So what a beautiful and painless way to do your cross-browser testing.  A Selenium RC test would look like a JUnit-style test in Java

import com.thoughtworks.selenium.*;
import junit.framework.*;
public class GoogleTest extends TestCase {
    private Selenium browser;
    public void setUp() {
        browser = new DefaultSelenium("localhost",
            4444, "*firefox", "http://www.google.com");
        browser.start();
    }

    public void testGoogle() {
        browser.open("http://www.google.com/webhp?hl=en");
        browser.type("q", "hello world");
        browser.click("btnG");
        browser.waitForPageToLoad("5000");
        assertEquals("hello world - Google Search", browser.getTitle());
    }

    public void tearDown() {
        browser.stop();
    }
}

But...you don't have to program in Java, you could write the same test in C#(NUnit), Perl (PHPUnit), Python (unittest), or Ruby (Test::Unit).

And that's not all...Selenium Grid lets you run your tests across multiple servers to run your tests in parallel.

Plus...Selenium integrates with Maven build system so your integration tests can run as part of the build.

Quite a belated discovery for me!  Not totally euphoric...I assume typical GUI testing issues of data, test fragility, etc... still exist.

Is there anyone in the company using this tool?  I am particularly interested in cross-browser testing, and Maven integration.  Also support for Javascript and Adobe Flex.

IntroductionToSelenium  

¡looɔ ,uıddılɟ os sı ʇǝſddɐ

er...

Appjet is so flippin' cool!  

Billed as "Instant Web Programming" ...with features such as:

• Server-side Javascript
• Free hosting in the cloud
• an IDE in the browser
• and a persistent object store (aka database)

In 20 minutes I created my first Facebook application ....the Mike Foley "Hello World Facebook Application"....(try it, it's the best-est Hello World in the whole world!) ....then I went on to try the persistent object store with the "Who will win the Tour de France 2009 Poll?"

Sounds like I am some kind of genius web programmer?  No, not really...you begin every application by stealing, er, let me rephrase that, you begin every application by cloning another.  Talk about open-source!  This turns the 2 word phrase "Intellectual Property" practically upside down!

It's all very interesting...but...

...ʞɹoʍ oʇ ʞɔɐq ʇǝɓ oʇ ǝɯ ɹoɟ ǝɯıʇ

Fasincating article on Agile/Lean documentation ....

 

As a QE, I just like a little documentation so I know what is in the iteration and what I need to test. But the article lists some things even better than documentation .... executable specifications...a.k.a. integration tests. These executable specifications, or integration tests, not only describe the new feature or behavior, if implemented in some sort of automated test suite become the basis of a regression suite.

Something the article doesn't address is the timeliness of the documentation. The article says document stable concepts...well...I am not sure that captures the essence of it for me. Stable sort of means...old; whereas I like documentation on the new stuff, from the new iteration.

The bullet points on documenting with purpose, I think that goes along with basic writing of knowing your audience.

Describing documentation as one of the worst ways to communicate....I have mixed thoughts. Yes, face-to-face, whiteboard discussions is a richer medium. But sometimes I don't get invited to all the meetings, or not have time for every meeting, or miss a meeting. And sometimes things are technical enough that a few written words are better ...instead of "go to the URL for the application or SOAP endpoint" ....it's nice to actually have that full URL written down for reference, nothing really more heavyweight than a README.txt.

The article is very interesting and goes into more detail...but here is the high-level suggestions:

1. Prefer executable specifications over static documents
2. Document stable concepts, not speculative ideas
3. Generate system documentation
4. Keep documentation just simple enough, but not too simple
5. Write the fewest documents with least overlap
6. Put the information in the most appropriate place
7. Display information publicly
8. Document with a purpose
9. Focus on the needs of the actual customers(s) of the document
10. The customer determines sufficiency
11. Iterate, iterate, iterate
12. Find better ways to communicate
13. Start with models you actually keep current
14. Update only when it hurts
15. Treat documentation like a requirement
16. Require people to justify documentation requests
17. Recognize that you need some documentation
18. Get someone with writing experience

Argh, my 10 day ej-Technologies JProfiler license expired and I need to verify a memory leak bug. Time to learn Quest Software's JProbe.

I bumbled around for a bit trying to configure JProbe so that I could see the JVM. Some non-productive time here messing around with java.opts files and .jpl files. But then I realized, I didn't have to do anything other than start my application, and from JProbe just attach to running JVM. That was easy.

So having used both tools, here is my compare/contrast:

JProfiler

I really liked this tool. It was easy to configure and do heap analysis, CPU profiling, and threading synchronization issues. The UI was fairly tame and easy to use. Easy download and trial process.

JProbe

The configuration was not so straightforward and had me confused, until I realized it was really very simple. And then the different UI for the heap analysis had me puzzled for a bit. I didn't find it initially as intuitive as JProfiler. Also, the memory analysis seems less interactive, and depends on taking entire snapshots of the entire heap and then diff'ing them, whereas in JProfiler you could just mark the heap and look at the differences.  Not licensed for CPU Profiling with JProbe...so I didn't do that, but I see the concept of a concurrent license server is a good idea, especially for large companies. I'd imagine that the usage pattern of these tools is sporadic, use the tool for a while, and then not use it for a while. So having shared licenses allows many people to use the product, rather than just licensing the tool for one developer. Also JProbe has some additional functionality to help you solve memory leak issues.

Summary

Now that I think about this JProfiler vs. JProbe comparison, I see the real meat-and-potatoes of Java profiling value-add as not belonging to either of these tools. The enabling technology is in the JVM itself in the Java Virtual Machine Profiling Interface (JVMPI) , and thank you Sun Microsystems for exposing this API, and in an open manner. The JVMPI exposes voluminuous information about the JVM for others (JProfiler, and JProbe, et. al) ....and I say thank you Sun for not making everybody use only your own profiling tools HProf and HAT (ack).

The value-add of both JProfiler and JProbe is in the UI....collecting, presentating, and manipulating this JVMPI data. Both tools are just UIs sitting on top of the same JVMPI interface. The JProfiler and JProbe tools differentiate through features of UI, configuration, and licensing. Also I suppose, IDE integration (didn't do this), and application server integration (didn't do this) either.

So for basic JVM heap analysis.....JProfiler or JProbe...eh, six of one, half-dozen of another. 

Today I met LISA.  No, Lisa is not a person; LISA is a product by a comnpany called itKO.  I attended a  webinar today titled "itko Lisa, Eliminating the Constraints of SOA Testing" with this as the description....

Many companies attempt to bring their quality efforts to bear for SOA, but find that there are several difficulties that limit their ability to deliver the needed business functionality on time. Join us for this session and learn how iTKO LISA can:

• Achieve Complete and Collaborative test coverage across all heterogeneous technologies that make up your SOA strategy.
• Help you realize better productivity from your existing Test Lab teams and assets.
• Eliminate the constraints of developing and testing against unavailable or incomplete services and systems.

Okay...that seemed interesting and relevant -- The projects I have worked on --Activation, File Exchange -- have all been web service based, if not true SOA-based. And in particular, the constraint listed -- testing of these services at various layers, in the context of an Agile process where a service or system may be incomplete or unavailable -- interested me. It turned out to be an interesting presentation. In particular, this image resonated somewhat with me ....

In this old world, there was not much sharing of information and test assets among these teams, beyond the occasional text “bug report” that carried little context into the root causes of problems that occurred. This occurred largely because the available test tools created scripted test assets that were incompatible, and largely irrelevant to the next team’s process in the chain. Unit tests were good for developers to test for structural bugs in code, but those tests didn’t translate into any business context for the QA team. The QA team would instead script their own UI tests, which don’t exercise components or code in a way that developers can use to track down root causes.

It's just images and sales pitch....but this is what the itko Lisa promises, and it looks like a possibly better way to me...

The goal of all application process tools is to enforce some kind of workflow into an aspect of the software lifecycle, be it around testing, development, operations or governance and integration of the overall IT infrastructure. Direct testability and validation of most SOA and enterprise software components that are managed by leading process tools is a must.

There's more, a lot more.
 

LISA is a complete no-code software testing solution that supports the entire team's development lifecycle for dynamic web apps, web services, ESB messaging layers, enterprise Java, .NET, data and legacy objects, and more. LISA will carry developers, QA teams and business analysts from unit testing, to regressions, functional testing, end-to-end integration, load testing, and monitoring after deployment.

If anyone is interested, here's a link to their homepage: itKO

Fasinating new open-source tool from Google called Testability Explorer.  Says their website,

Testability-explorer is a tool which analyzes java byte-codes and computes how difficult it will be to write unit-test. It attempts to help you quantitatively determine how hard your code is to test and, where to focus to make it more testable.

Test metric tool can be used:

1. As a learning tool which flags causes of hard to test code with detailed breakdown of reasons.
2. To identify hard to test hair-balls in legacy code.
3. As part of your code analysis-toolset.
4. As a tool which can be added into continuous integration that can enforce testable code.

Currently the tool computes:

1. Non-Mockable Total Recursive Cyclomatic Complexity. Cyclomatic Complexity is a measure of how many different paths of execution are there in the code. It is computed, by counting the if, while, and case as branching primitives. It is recursive because cost of the method as well as any methods it calls are counted. It is total because cost of object construction as well as any static initializations are counted. And finally, it is non-mockable because any code which can be mocked out in test is not counted as part of the cost.
2. Global Mutable State. Counts the number of fields which are globally reachable by the class under test and which are mutable. (A mutable global state can make testing difficult as tests are not isolatable)

It produces output that looks like this:

 testability.sh -print summary com.google.test.metric -whitelist com.google.test.org.objectweb.asm.
      Analyzed classes:   125
 Excellent classes (.):   123  98.4%
      Good classes (=):     0   0.0%
Needs work classes (@):     2   1.6%
             Breakdown: [..................................................]
       0                                                                     98
     3 |......................................................................:    98
     9 |.........                                                             :    12
    15 |......                                                                :     8
    21 |..                                                                    :     2
    27 |...                                                                   :     3
    33 |                                                                      :     0
    39 |                                                                      :     0
    45 |                                                                      :     0
    51 |                                                                      :     0
    57 |                                                                      :     0
    63 |                                                                      :     0
    69 |                                                                      :     0
    75 |                                                                      :     0
    81 |                                                                      :     0
    87 |                                                                      :     0
    93 |                                                                      :     0
    99 |                                                                      :     0
   105 |                                                                      :     0
   111 |@                                                                     :     1
   117 |                                                                      :     0
   123 |                                                                      :     0
   129 |                                                                      :     0
   135 |                                                                      :     0
   141 |                                                                      :     0
   147 |@                                                                     :     1

Highest Cost
============
com.google.test.metric.Testability 148
com.google.test.metric.asm.MethodVisitorBuilder 113
com.google.test.metric.method.BlockDecomposer 29
com.google.test.metric.asm.MethodVisitorBuilder$GetFieldRunnable 27
com.google.test.metric.asm.MethodVisitorBuilder$PutFieldRunnable 27
com.google.test.metric.asm.MethodVisitorBuilder$2 23
com.google.test.metric.Type 22
com.google.test.metric.asm.ClassInfoBuilderVisitor 18
com.google.test.metric.asm.FieldVisitorBuilder 18
com.google.test.metric.MetricComputer 17
com.google.test.metric.asm.MethodVisitorBuilder$30 17
com.google.test.metric.asm.MethodVisitorBuilder$12 16
com.google.test.metric.collection.KeyedMultiStack 15
com.google.test.metric.method.BlockDecomposer$1 14
com.google.test.metric.method.op.turing.MethodInvokation 14
com.google.test.metric.asm.MethodVisitorBuilder$28 12
com.google.test.metric.asm.SignatureParser 12
com.google.test.metric.asm.SignatureParser$TypeVisitor 12
com.google.test.metric.report.TextReport 12
com.google.test.metric.asm.MethodVisitorBuilder$7 10

If you are in the business of testing...it is definetly worth a look-see....

I am learning about Design Patterns.  So as a project to reinforce my learning, I am writing unit tests on each pattern.  So this post if the first of a series of posts where I will investigate testing of Java Design Patterns such as factory, singleton. memento, proxy, ...everything.  This post is on testing a pattern called "The Clone Pattern".

I think a test of the clone pattern would look like this

@Test
    public void testCorrectClone() {
        ClassUnderTest aClassUnderTest = new ClassUnderTest();
        ClassUnderTest clone =  aClassUnderTest.correctClone();
       
        Assert.assertNotSame("Clone should be a different object than original", aClassUnderTest, clone);
        Assert.assertEquals("Types should be the same", aClassUnderTest.getClass(), clone.getClass());

    }

So this asserts my expectations of the clone...that it is a different object of the same type.