Mike's BLOG

 

My camera wasn't fast enough to freeze the action in the dimming light ....but these are the photos I got.  Fun night ....

 

 

One of the things that keeps me awake at night is the 8 Fallacies of Distributed Computing.   As computers and software were transitioning from standalone entities to networked and distributed entities, all sorts of new unexpected issues and problems arose.  In 1994, Peter Deutsch recognized these issues as …in the general sense…failed assumptions of the new computing landscape.  Then in 1997 James Gosling of Sun added another failed assumption …and together these failed assumptions became what is known as “The 8 Fallacies of Distributed Computing.”    Says Peter Deutsch….

Essentially everyone, when they first build a distributed application, makes the following eight assumptions. All prove to be false in the long run and all cause big trouble and painful learning experiences.

1.	The network is reliable
2.	Latency is zero
3.	Bandwidth is infinite
4.	The network is secure
5.	Topology doesn’t change
6.	There is one administrator
7.	Transport cost is zero
8.	The network is homogenous

In a nutshell, I think it all sort of boils down to you can’t rely on the network or internet that connect the various tiers of a distributed system.  Stuff happens.  And most importantly, stuff you never thought of before…. will happen.    Your connection to a backend database may break.  The browser’s connection to the internet may become very slow, or fail.  Hostnames may not be resolved.     Your RMI connection to a back-end process may be broken.  Things that you assumed would be true as you developed the application …well, they may not be true when the application is deployed.

So as a QE Engineer, I am recognizing that …in addition to all the usual functional, performance, and usability aspects to software quality ….there are also the quality issues related to these 8 fallacies when testing distributed systems.  All sorts of new possibilities for test cases appear.   What happens when I yank the network cable out of my computer (or gently disconnect it) while using a distributed application?  And then what happens when I plug the network cable back in?  What happens when I the internet becomes very slow?  Is the application just slow? Or unusable?   The list of failure scenarios for distributed systems can be much greater than when testing a closed system that is installed on a single standalone computer.

I found in myself there is a mental shift that needs to occur when considering testing distributed systems.  Whereas in testing a closed system, a broken network connection may be thought of as an invalid test scenario or a problem in the test environment…with a distributed system these sorts of issues can be considered valid.   At first, it doesn’t seem right to log a bug that says (metaphorically) … “When I pulled the network cable out of my computer…and then put it back in …my distributed application no longer worked.”   Well initially you think …ummm….don’t do that and you won’t have any more issues like that.  Yes, but the application needs to be robust and handle that situation…which is may be a real situation.

So there are testing tools available to help find these sorts of issues.  One such tool is NetLimiter … which is used to introduce network latency into a test environment.  With such a tool you can find and flush out issues that traditional testing may not find.

Another approach I was thinking of …rather than a “post-development 8 fallacy testing phase” …are testing approaches that “inject”  the fallacies (such as latency)  right into the developer sandbox and build and test system.   For example, right now I am in a Maven build environment ….and the system-under-test is started with a command like this:

mvn jetty:run

This is simplifying a bit, but that is it in a nutshell…it starts a J2EE application server hosting the system-under-test.   This J2EE  application server  communicates with a MATLAB Distributed Compute application.   Now the concept is just this….introduce latency into the test environment by passing a property on the command line:

mvn jetty:run -Dlatency=1000

And ….by instrumenting the code in the right place …. well….I am then simulating latency on the RMI communication between the application server (Jetty) and the MATLAB Distributed Compute Engine.

String latency = System.getProperty(”latency”);
System.out.println(”latency property = ” + latency);
if (latency != null) {

System.out.println(”sleeping …”);
try {
Thread.sleep(Integer.parseInt(latency));
} catch (InterruptedException e) {
e.printStackTrace();
}
}

Now …Thread.sleep() …that’s not rocket science.  But, simple as this is …by simulating latency, I am seeing all sorts of new behaviors in the Flex UI that I didn’t notice before.     Latency issues that may have gone unnoticed ..may now be flushed out….and that is what is important.

I had a fantastic race for myself!  Fun!  And I think I finished in the 2nd quartile ....will see on the results.  And a zillion thanks Gina, Bill, and Pete ....and Lorraine too! ...for coming to the race! 

First time up the big climb ...poorly positioned.  But stayed attaached.

Very strong headwind on route 12 .... but about 1/2 mile before the center of Sterling ... I attacked the peloton!  Yes ...this is the very first time I have ever been detached from the peloton ...on the front!  I felt great ...but probably not the smartest thing.  I was off the front for maybe 2 minutes .... then I let them catch me.  Recovered some ...but stayed in the top 10 positions ...and took the 2nd climb very well-positioned.  Yeah!

I asked this guy Jim who I met ...(who eventually got 3rd) ...what was going on at the front of the group when I attacked.  He told me they were like 'WTF? Who is this guy?' etc... and they really had to work some to catch me.  In hindsight ...probably not the smartest thing to do ...I should have used just enough energy to position myself well at the front.  Actually, it was probably lunacy ...but I don't care.  It was damn fun! 

So by the 3rd climb I was dangling ....and then detached.  Argh.  Working with a group of about 8 all the way around ...and on the headwind on route 12.  But I felt good ...and same spot ...as where I attacked before...I attacked again.  Dropped them all but one ... had about 400 feet on them.  I can see from the video and photos they were catching up .....but no one passed me on the climb.

Ultimately ...2nd quartile.  Still got dropped.  But did execute some overt aggressive cycling tactics ... had big fun.  Rode pretty well for myself ...

Looking at a test tool called JUnittestmaker ….

It’s an open-source project circa 2001 by a company called WebMind …. apparently no longer in business.

The purpose of the test tool is to create JUnit test skeletons for entire packages of untested code.   The thought being …. if you have the skeletons …then you are over the initial hurdle of writing a unit test.   You point the tool at a package …and voila …. 1 test skeleton for every class.   The skeleton is JUnit 3 style …and you get an empty test method for every public method in the class under test.   I just tried it …voila …126 test skeletons on a particular package.

Initially there is a “wow” factor as you think …”wow, instant unit tests!!  unit test generation en masse!!”

But then there is a bit of a letdown after the initial wow.  The tests are empty and don’t do anything.

As I think about the challenge of unit testing …yes, there is some initial inertia sometimes in getting started, stubbing about the test classes, etc…. And this tool certainly does grease the skids on this initial part of the test writing lifecycle.

But the real challenge of unit testing still remains.  Meaningful tests that reflect the developer intention.  Meaningful tests that reflect the RFAIN.  Meaningful tests that prove the class is working as intended.   Meaningful tests that enable developers to refactor a class or package with confidence.   Mocking.

JUnittestmaker …. a bit of a misnomer.  It’s really JUnitTestSkeletonMaker.

Worthwhile?

Mike,

You drove that chase group and got it all organized. I told you you had a lot more in you than you let on! Great day.

Jose.

One of the JUnit add-ons I like is JUnitPerf.  For very little effort, you can decorate your JUnit tests into performance tests — timed regression tests, and multi-user multi-iteration load tests.    Hey, the approach is not the end-all of performance testing, but I do think there is big value (and for little cost) to the approach.    But I had been recently stymied with the approach since switching to JUnit 4.x.  It just didn’t work.

So months later…on the exciting MATLAB Server EDU project ….I am once again wanting to use JUnitPerf to get some performance and load testing value built into the automated continuous integration and test  environment.   But as we are using JUnit 4.x …argh…stymied.  And I am loathe to change the dependency in the pom.xml to JUnit 3.x and refactor the tests.   But ….I found a neat little tip on the web for using JUnitPerf with JUnit 4.x, and here it is:

1) Create a JUnit factory:

import junit.framework.JUnit4TestAdapter;
import junit.framework.TestCase;
import junit.framework.TestSuite;

import com.clarkware.junitperf.TestFactory;

class JUnit4TestFactory extends TestFactory {

    static class DummyTestCase extends TestCase {
        public void test() {
        }
    }

    private Class<?> junit4TestClass;

    public JUnit4TestFactory(Class<?> testClass) {
        super(DummyTestCase.class);
        this.junit4TestClass = testClass;
    }

    @Override
    protected TestSuite makeTestSuite() {
        JUnit4TestAdapter unit4TestAdapter = new JUnit4TestAdapter(this.junit4TestClass);
        TestSuite testSuite = new TestSuite("JUnit4TestFactory");
        testSuite.addTest(unit4TestAdapter);
        return testSuite;
    }

}

2) Then from the test class ....instead of doing this ...

 public static Test suite() {

        int users = 10;

        Test testCase = new TestMatlabHtmlConsolePageStatefullSession();
        Test loadTest = new LoadTest(timedTest, users);

        return loadTest;

use the factory...

     public static Test suite() {

        int users = 10;
        JUnit4TestFactory mytestfactory = new JUnit4TestFactory(TestMatlabHtmlConsolePageStatefullSession.class);
        Test testCase = mytestfactory.makeTestSuite();
        Test loadTest = new LoadTest(testCase, users);
        return loadTest;
     }

So this allows JUnitPerf with JUnit 4.x.  Not complicated, but very useful, 
as it turns an automated single-user integration test into a much 
more useful automated multi-user test for stateful sessions.

“Are you ready?” asked the United States Cycling Federation race official to the 40 or so cyclists each standing with one foot on the ground and the other foot cleated to their bicycles. But not one of the cyclists said a word in response to the race official. Only a red-tail hawk, high on a branch above, had anything to say, letting out a long SCREEEECH, and then swooping low, instinctively knowing that in this place here, something was about to happen.

Although I did not answer the question of the race official, I thought about it: Am I ready? Can I climb? Can I descend with reckless abandon at speeds approaching 40 mph? Can I handle the shoulder-to-shoulder jostling in the pack? Can I ride my bicycle at 23 mph for 1 hour over hilly terrain?

The answer to that last question – can I ride at 23 mph – is no. That is too fast for me, too fast for all the other racers too. If I ride 100% maximum effort for 1 hour I can hold about 18-19 mph, about 5 mph short of the requisite speed today. But I am not dismayed at my apparent shortcoming in this regard. There is a magic of sort that happens at a race that will enable me to ride faster than I can alone. The magic happens when you join the contentious fellowship called a bicycle peloton. The peloton is a 100-legged beast that is able to power through the wind in a way faster that any individual rider can ride alone. This magic of the peloton and group riding is what I am counting on, indeed, why I am here at all. But while I am optimistically counting on the magic of the peloton to ride fast, I am fully cognizant the peloton is not a benevolent beast. Indeed, the peloton is absolutely merciless, and if you so much as slip a gear with a mechanical incident, fatigue one iota on a climb, become cautious on a descent, the peloton will leave you behind in an instant. And once dropped, you will never be able to catch back on ….your 18 mph solo pace will never be enough ….and your race is essentially over. Or, if you think you can outrace the peloton by trying to breakaway off the front…Hubris! Madness! If you breakaway off the front, the peloton will surely hunt you down and take back it’s own.

The magic of the bike race, for me anyways, happens when you lose yourself and become a part of the peloton, a cell in a larger organism, a gazelle in the herd. Perhaps that is why no one answered the race official’s question, the racers are already beginning to lose their individuality in the 100-legged, but voiceless, peloton.

While at the starting line, I took a quick look around to absorb my surroundings. The Blue Hills,so by early explorers who gave these hills beyond Boston Harbor that name because of their blueish hue, are all around me. But this morning, I see no blue, only a kaleidescope of colors from the water droplets on my wraparound sunglasses. Each water droplet showing spectacular colors: yellow from the blooming forsythia, violet from azaleas, glistening chrome from deraileurs, and all sorts of colors from the advertising-plastered lycra clothing of the racers.

Not that I can predict the future or the outcome of this race, what will happen over the next hours is, at some level, already known. The race will begin at a furious pace and only get faster and faster like the unrolling of a carpet, and at the end of the carpet’s unrolling the pace will be such that the peloton, or what is left of it whittled down as it will become through massive attrition, will be stretched out like a snake and the strongest will be at the front and the weaker sorted towards the back. For me, the race is not about victory at all….it is about the ride, the journey. My race is one of survival, to fight to stay in the peloton as long as possible, to belong, to not be dropped.

People have advised me on race strategy. The best strategy advice I have received, and from an experienced racer, sort of now my ‘coach’, is to stay near the front of the peloton. For at the front, you avoid getting caught behind gaps that may open up in the peloton because of fatigue or crashes. But I also find this advice amusing and it reminds me of the secret to winning basketball games that I received once: “You need to stop your opponent from scoring, and you need to score yourself.” Yes indeed, the team that does that will win basketball games. And yes, if I can stay near the front of the peloton then I will have raced well. Both good advice, but both easier said than done.

The race officials rattles off a list of disclaimers and warnings: "Watch out for the broken pavement on ____ Street." "Do not dive into the corners, the race will not be decided on your positioning coming out of any corners. The race will be decided on the climbs." "Do not cross the yellow line." The racer next to me says snidely "The real issue is the steel plates across the road where there was construction yesterday. It's at the bottom of the fastest descent, and it will be a skating rink. Watch out for the crashes and bodies flying there." I ignore all these dangers. While normally risk-averse and methodical, the next hour is all out and I have no intention of touching my brakes under any circumstances. I am a rolling mid-life crisis on a 3k carbon fiber bike.

“Well, have at it.” The race official says, and all the riders do just that stepping on a pedal, cleating in with the other pedal, permanently attaching themselves to their steed, barring crashes, for the next 60 minutes. The pace quickens rather abruptly and my heart rate more than triples from a resting rate of 50 bpm to 160 bpm, nearly 3 beats per second, where it will stay for the next hour desperately trying to supply my legs with oxygen and energy needed to propel myself around on 2 wheels.

Soon the Blue Hills are under our wheels. The main climb is max 10% gradient, in old money that's 1 in 10...for every 10 feet forward you climb 1 foot. Hills are one of the great sorting mechanisms in a race. With cruel efficiency, the weak are culled and dropped from the peloton. And those who are not dropped will begin to be sorted in order from strong to weak. I shift down to 39x25 to begin the intent, turning my legs over at 90 rpm. I begin to focus and repeat my climbing mantra – “up”. I think of this one word only when I am climbing. Up, up, up, saying the word with each pedal stroke. It is mind over matter. This one word crowds out all else, like a light in an otherwise dark room…keeping the fatigue, pain, and self-doubt at bay. This pace is too high, I need to ease; up. My thighs are burning, I need to slow down; up. This hill is too long and steep; up. Up, up, up.

And then we begin to reach the crest of the hill – the end of the up -- but does the pace ease? Is there respite at the top of the hill? No. Why is there no respite at the top of the hill? I have no idea, but if you don’t hold the same climbing exertion level over the top of the hill and down the other side, someone else will, and then you will be dropped just when you thought you made it to the top. If the purpose of the hill was to cull the weak, then the crest of the hill is to cull those foolishly optimistic that the pace will relent. I hold that exertion level right over the top and get right up to speed as quickly as possible.

And then comes the descent. Madness. Hands off the brakes, chins resting on the handlebars, knees and elbows tucked in, trying to present the most aerodynamic shape to the wind. I lift my head up just a smidgen and I can see my mph drop on my cyclecomputer; stay in my tuck. And at speeds up to 40 mph, I fight to stay on the wheel in front of me, ticking a big 53x12 gearing over at 120 rpm, 6 inches behind the whirring wheel of another racer.

I hear the clash of metal and the uttering of profanities behind me – someone has crashed. More likely 2 or more have crashed. While riding at your limit, these accidents are bound to happen. And those caught behind the crash will be separated from the peloton and have to spend precious energy to catch back on to the peloton, if indeed, they can do that at all. Yet another reason to ride near the front.

Over and over this repeats: Climbing, cresting, descending, cornering. And all the while the race is like a millstone grinding away at the peloton. Wearing it down. Separating the wheat from the chaff. Dropping riders along the way like a meteor entering the earth’s atmosphere. A race? Nay, it is mostly about attrition. And the struggle of anti-attrition, to stay attached to this wonderous and monstrous beast called the peloton.

The word “peloton” is a French word meaning “ball”, and maybe this is the best way to explain or understand a bicycle race. The peloton is just a ball or mass of riders. This ball is subject to 2 very strong and opposed forces acting on this ball. One force draws the peloton together and is based on the need of the riders to cooperate in the fight against wind resistance. Riding behind another rider, drafting as it is called, requires 30% less effort. This is why there is a peloton, or ball, in the first place. Every racer recognizes the winning recipe of opportunistic altruism of riding in a group. The second force acting on the peloton is a force that acts in the opposite direction trying to dissolve or explode the peloton. This force is based on competition, and everyone is trying to get to the finish line first, not necessarily tow their rivals in their wake as they make a competitive effort. The other thing to recognize about the peloton is that this ball or mass of riders is not comprised of a single consistency. Some racers specialize in climbing, others specializing in flats or descending, and everyone of different fitness levels. So this ball can be seen as a social entity subject to both the forces of cooperation and competition, as well as external forces of hills, descents, corners, crashes, and what not. Sometimes the ball stretches out in a long line, and sometimes the ball fractures as it did on the hill on the second lap.

On a hill the cohesiveness of the peloton, as well as the individual racers, is under massive duress. Gaps open, and it is work, work, work to close the gaps down. But one gap opens ahead of me and the peloton splits in two. This is a major issue, and when it happens, it's business time. Time to go to work and chase back on. Another rider and I, recognize the decisiveness of this gap. I accelerate hard to get in his draft, with a few riders doing the same to get on my wheel and in my draft. This is how pacelines form, and one did. A paceline is an ad hoc cooperative effort between racers, each taking a short turn or pull in the wind, and then rotating to the back. It is an extremely efficient cooperative formation, and together as a group we chase back on to the peloton. But even though we made it back to the peloton, there was a high price to be paid. Our chase took 2 or 3 miles, and we all made a payment out of our ver-dwindling precious reserves for our time outside the peloton.

By the third ascent I had worked my way back up to the front 5 or 10 positions. Astutely alert to the possibility of gaps, when one opened in front of me, I raced around the rider in front of me to close the gap. I stayed attached on this third ascent, although I find myself sorted towards the back. On the flats, I focus on drafting and pack positioning, working my way back through the peloton to one of the top 5 positions.

Just before the final ascent, the pace mysteriously eases, a lull before the storm. I take a look around and take a few deep breaths. See who smiles. See who nods. See who slumps their shoulders in resignation. I look at my gearing to make sure – 39x25 – and again begin the intent for the final time. Up. Up. Up. Turning the gears over at 90 rpm. Up. Up. Up. This climb is about who can suffer the most, and I begin the climb in suffering debt. I’ve been splunking in the pain cave for the last 5 miles, and as the hill accelerations and hill sprints start, I drop the flashlight.

A rider next to me turns his head to look at me, and I realize my “up, up, up” mantra is no longer my inner voice, but has leaked over to my external voice. Embarrassed for a moment, I realize I have become the cycling equivalent of the grunting Monica Seles, a tennis player whose vocal grunts upon hitting the ball scare not only the opponent, but also those watching tennis on TV.

And its not just my metabolism that is on the brink of meltdown, so is the cohesiveness of the peloton. The cooperative nature of the peloton is dissolving as the pendulum swings to raw self-interest and competition. It is wheel-to-wheel, shoulder-to-shoulder, physical jostling and verbal jousting…”Move up!”, “On your left!”, “What you waiting for?”, “If you ain’t got no legs then get out of my way” etc…

They are really drilling it, little gaps opening, and I am closing the gaps down, but I am on my limit. With the hill accelerations and sprinting, the sorting process of this final climb is ruthless. My thighs are loaded with lactate. Gaps open that I cannot close, and I finish just a few seconds behind the leaders. 17th place out of 38 racers. I’ll take it, that was a strong ride for me.