One aspect of the testing vs. quality issue is that of  scope.  Testing focuses on measurement of application fitness for use, while quality focuses on the process of creating, delivering and running an application.  As an illustrative example, from a quality perspective, requirements are a key element of application quality.  From a pure testing perspective, a requirement is merely an input, a given.  Something to be taken at face value, and perhaps worse, something that may not even exist in the first place.  Simply put, quality requires more than testing…it requires a process from requirements through development to delivery and operation.  (Now – whether a tester has much impact on all those areas is up for debate, but I digress.)

On November 5th at 1pm (EST), Brian Beveridge from Greenridge Business Systems will speak about his own experiences and the benefits of a collaborative process for delivering software quality with Rational requirements and quality management solutions.  His is more of a quality perspective – and the Rational Tester expects this to be an interesting interview.  I’ll be dailing-in, and I encourage you do to the same.  On my wish list of questions, Mr. Interviewer, I’m hoping there’s an opportunity to find out about  some of the most common problems he sees at his client sites with regards to requirements and quality management?

I love to learn from others ‘ mistakes.  Hopefully Brian Beveridge will have some good ones to share.

The conference team at Pulse 2010 called for an interview earlier this week.  The transcript of this interview just went live.

For the uninitiated, Pulse 2010 is IBM’s annual service management conference.  For the first time at the conference this year, the conference team has invited Rational to participate.  As a result, there are three Rational Tracks each of which will have 12 sessions.  The Rational Tracks are:

• Smarter Products Delivery and Management
• Change Management for Applications and Services
• Quality Management for Applications and Service

The Rational Tester was asked to chair the Quality Management track.  The main responsibility here is to weed through all the conference submissions to select the cream of the crop of presentation content for the conference.

During the interview, I was asked a couple of key questions: What content are we looking for?  and What makes for a good presentation?

To read my responses, please see the Pulse 2010 blog page with the Rational Tester Pulse 2010 interview.

The week of February 21-24, IBM will be hosting their annual Pulse 2010 conference in Las Vegas!  Woohoo!

In the past, the conference’s main focus has been on the IBM Tivoli tools which IT operations personnel  use to manage a dynamic IT infrastructure.   This year though, the focus is on the larger objective of Service Management.  The larger Service Management focus means that while it remains critical to keep your applications up and running, it is as important to focus on the development of those services.  Accordingly, at this year’s Pulse there’s a “Software Delivery Lifecycle Management” stream which covers the Quality Management, Change Management and Delivery of Services.

The call for papers is now open, and consider this your invitation to participate.   If you’re a software delivery professional creating, building and testing applications and services we want to hear from you.  Come share your knowledge and expertise with a larger service management community, and in turn hear what other software delivery professionals are doing as part of their service management efforts.

Speakers get a free conference pass, which entitles them to attend all sessions, keynotes, meals and events at the conference.  I haven’t seen the keynote line up yet, but last year Magic Johnson was on board – have a look.  The Rational Tester will be there – this is one not to miss!

Learn how IBM Service Management combines industry-leading software, hardware and services that drive innovation through visibility, control and automation.

The Rational Tester has been traveling a lot lately – you may have noticed a recent posting drought.  One of these trips was to the STANZ 2009 conferences in Wellington, New Zealand and Sydney, Australia.  The presentation I delivered was the Wisdom of Crowds.  This presentation is essentially a report on testing best practices followed by IBMers – from our technical implementation teams, our tool building teams and our consulting groups.

Another of the presentations at this conference was Becoming An Expert Tester from James Bach.  An excellent presentation in its own right, I found that in combination with my own brought out an interesting conundrum faced by testers.  The Expert Tester presentation focused on things an individual can do to hone their skills as a testing professional.  The Wisdom of Crowds presentation  focused primarily on quality processes that teams can implement to improve the quality of their end product.   Put another way, one presentation focused on testing, one on quality.

It struck me afterward that this is an issue that testers struggle with on a daily basis.  It’s crucial to consistently improve one’s skills as a tester, learning new approaches and techniques – but the big payoff, in terms of delivering a better product comes when the whole team focuses on quality driven software delivery.  The catch, of course, is that as an individual, it’s much easier to improve one’s skills as a tester than it is to impart change on quality practices that impact the entire software delivery team.  Put another way, you can be the best auto mechanic in the world, but if your job has you working the production line for the Edsel, you’re not going to have that much of an impact.

While I have no grandiose plans for an elaborate series on this topic, I have labeled this post as “Part 1″ – as there’s lots more to explore on this thread.   Let the discussion continue…

Greetings to all the fine attendees of the STANZ 2009.  I appologize for the delay in posting, I simply did not get to a stable connection long enough to get this up any sooner.  Hope everyone enjoyed the presentation, and look forward to hearing from you soon.

Rational Tester – Brian Bryson STANZ 2009 Keynote Presentation

Completing the trio of IBM Rational v8.1 updates is our look at IBM Rational Service Tester v8.1.  In terms of updates, the primary enhancements in this release are:

• Updated integration with IBM Rational Quality Manager Express Edition and IBM Rational Quality Manager Standard Edition V2.0 and V1.0.1
• New Service Stubbing enables testing of SOA applications prior to the development of all components
  • Automatic stub creation from Web Services Description Language files
  • Visual editor to modify stub behavior
  • Automatic stub deployment prior to test execution
• Expanded environment support for NTLM and Kerberos authentication and support of attachments using the DIME format

The first and third items in the list are architectural features.  The updated integration ensures continued compatibility with the upcoming Rational Quality Manager 2.0 release.  The third item shows Rational Service Tester’s  continued commitment to supporting all Web Service protocols and technologies.

Service stubbing  is a very powerful technique for unit and component testing which enables developers to start testing portions of their applications before the entire application is complete.  A service stub is temporary replacement for a web service.  It returns a preset value for a given input, without really performing any computing. As an example, consider a web service that validates a credit card.  The true web service would take as input the credit card number and expiry date, verify this information, confirm it’s validity and return a “valid/invalid” result.  However, during development, this complex piece of code may not be available on day 1, but it may be required in order to start testing a order processing use case.  In order to enable testing of an order, a developer could create a stub credit card validation service that would simply return a “valid/invalid” result for a given credit card number.  No processing would happen in the stub, just a simple lookup that would return valid for a given credit card number, and invalid for another.

Service stubbing with Rational Service Tester enables the testing of your SOA application prior to the creation of all services. Stubs are automatically generated from WSDL files and will be automatically deployed before test execution. After the generation of service stubs, the Visual Editor can be leveraged to modify stub behavior to meet specific testing scenarios. For example, simulation of error conditions such as an external service being unavailable or answering with a fault message are typical uses.  This enables web service testers to test incomplete systems, or put another way, to test their applications sooner.

Earlier this week, IBM Rational released IBM Rational Performance Tester v8.1.   Your basic highlight reel of “What’s New” is as follows:

Updated integration with IBM Rational Quality Manager Express Edition and IBM Rational Quality Manager Standard Edition V2.0 and V1.0.1

Expanded reporting capability for Service Level Agreements (SLAs) for results assessment

Web 2.0 Encoded data support

Live browser rendering for multiple user runs to ease issue resolution

These are actually all pretty interesting enhancements.   The first is an architectural change to facilitate integration, the next three are all productivity enhancers with direct impact to the performance tester.

The encoded data support will make it easier for performance testers to co-relate or datapool data that was previously hidden in binary data streams.   Most content sent from a web server to a browser is clear ASCII text.  RPT can, and has always been able to, parse this text to extract any data that might be required later in a test (like a session ID) or data that might need to be variablized (like a user id / password) for each user.  However, sometimes web servers don’t send ASCII text, instead they send their data as binary or as serialized java objects.  To Rational Performance Tester, this data looked like scrambled characters.  Users would not be able to use this data for correlation or data pooling.  With the new Rational Performance t v8.1, the tool can now convert binary data to ASCII text, and Java Serialized Objects to XML.  Users can now use this data for correlation and datapooling effortlessly.

Live browser rendering is a nice debugging feature.  This enables the Rational Performance Tester user to examine the data being sent to any individual virtual tester.  The user can get a browser view, rendering the server response for each user, enabling the tester to see exactly what data is going back and forth.  It’s like a mall security camera – you can peer in on the activity of any user.

Finally though, the most interesting, and the one the Rational Tester has been waiting the longest for is the Service Level Agreement reporting.  Quite simply, Rational Performance Tester v8.1 now allows you to set service level agreement levels, such as “page response should be no longer than 5 seconds”, as rules in your test.  At the end of the test, any page that violates this rule is reported on.  You are effectively linking business objectives to your performance results.  Here are some of the new Performance Requirements reports in the new Rational Performance Tester v8.1.

Rational Performance Tester 8.1 Requirements Overview Graph

Rational Performance Tester 8.1 Performance Requirements Table

Yesterday, IBM Rational released the latest and greatest versions of

• IBM Rational Functional Tester v8.1
• IBM Rational Performance Tester v8.1
• IBM Rational Service Tester v8.1

This is a significant release for all tools, with each receiving many significant updated features.  In the next three blog posts, we’ll be picking the top feature from each tool and diving a little deeper.  First up, Rational Functional Tester v8.1.  Next, Rational Performance Tester v8.1.  And finally, Rational Service Tester v8.1.

Rational Functional Tester v8.1 was introduced to the world yesterday, and brought forth a significant new direction in functional testing for IBM Rational.    Here’s the full laundry list of updates:

• Enhanced visual test representation for no code test creation
  • Drag and drop of controls into test script
  • Story board thumbnail test representation
• Updated integration with IBM Rational Quality Manager Express Edition and IBM Rational Quality Manager Standard Edition V2.0 and V1.0.1
  • Multiple project area support in the Execution Adapter
  • Support for keywords in IBM Rational Quality Manager
• New integrations with IBM Rational Team Concert
  • Assets maybe managed within Jazz SCM
  • IBM Rational Team Concert V2.0 as an optional feature in IBM Rational Functional Tester
• Expanded packaged applications and environment support

The first item, Storyboard Testing, deserves a little more inspection.

Storyboard Testing is a new, visual approach to automated test script creation.  It provides the tester with a new visual representation of test script actions.  A three pane view of the test provides:

• A natural language description of test actions
• A screenshot view of the screen being tested, with control highlighting to further indicate the control being used
• A storyboard view of the script flow, showing thumbnails of all screens accessed during the test in sequential order

This new visual representation significantly simplifies test script authoring and editing.  The natural language steps together with the screenshots provide testers with an easy way to understand the test flow.  Test editing can be done from this view as well.  You can insert verifications, data driven commands and even Java code from this perspective.

This feature does not come at the expense of the traditional test script approach.  This new storyboard testing view is merely an overlay of the Java test script as it has always existed.   Users already familiar with Rational Functional Tester are able to work as they always have, directly with the test script.

This new method of testing will accelerate test creation by making it simpler to read and edit tests as well as open up test automation to those who were previously intimidated by having to work with test script code.

Recently, the London Stock Exchange has faced criticism  related to the failed implementation of a new trading system, TradElect.   As reported by Reuters on September 8, 2009:

“The London Stock Exchange (LSE.L) suffered its worst systems failure in eight years on Monday, forcing the world’s third largest share market to suspend trading for about seven hours and infuriating its users.”

This failure is one of the more spectacular in terms of publicity and business impact.   Making matters worse, the failure occurred at the epicenter of the US credit crisis, right as the US government decided to bail out Freedy Mac and Fannie May.

TradElect is written in c#, leverages the .NET platform and runs on Windows servers.   Computerworld’s Steven J. Vaughan-Nichols lays the blame squarely on Microsoft’s shoulders:

“The programmers and serious database administrators in the audience can already see where this is going. Sorry, Microsoft, .NET Framework is simply incapable of performing this kind of work, and SQL Server 2000, or any version of SQL Server really, can’t possibly handle the world’s number three stock exchange’s transaction load on a consistent basis.”

I find this logic frightful and shortsighted.  Having been involved in hundreds of system deployments, I have seen systems fail for a multitude of reasons;  poor design and inefficient coding being the primary suspects.   Rarely can one solely blame the underlying platform or operating system.   To suggest that the only problem here is Microsoft, and that the same programmers, following the same design and methodologies, but implementing on a Linux platform would not have suffered an equivalent fate is myopic.

As testers our job is to identify the presence and cause of system deficiencies.  Blamestorming based on personal bias and hype, such as found in the Computerworld articles, has no place in the world of software and systems delivery.

Computerworld --;