Thoughts about Problem Releases

["Chris Travers" <..hidden..>]

Hi all;

My ideas here are my own.  I do not speak for the core team in this
but I think that it is important to share my ideas and thoughts with
everyone and get some feedback.

1.2.5 has been one of the more problematic releases in some time
(since 1.2.0-1.2.2).  I wanted to share my thoughts with people here
about what to expect in the future, what we are doing to minimize such
problems as we move forward, and get ideas for ways to help get
community involvement in QA.

As of the release of 1.2.5, current policy was to do a review for the
most common (mostly packaging) issues to date.  The review took no
less than two days, and could take longer.  1.2.5 was in and out of
validation at least twice.  And several people had been beta testing
it for far longer than the validation problem before the particular
problems were discovered.

Given the history of 1.2.x and the lengthy feature-freeze that release
unerwent, I personally do not think that testing alone will resolve
this problem.  I think it is inherent in the codebase we inherited.  I
also think that we are going to have occasional problem releases until
we get all of the inherited code out of our codebase.  To give you an
idea, 1.2.0 went through about a month of active development and
nearly six months of post-feature-freeze testing and bugfixes.  No
codebase should require that sort of overhead in QA.

A review of the SQL-Ledger changelog indicates that SQL-Ledger runs
into the same problem from time to time.  The larger issue is that we
tend to loudly announce and discuss when this sort of thing happens.

First, I would like to suggest that problem releases may occasionally
occur through the development of 2.0 but that they should become less
and less frequent both between major releases and in major trees.  Due
to the code that has been inherited, this is something we are going to
have to live with for a while longer.  The simple fact is:  this
codebase is both disorganized and brittle and it is fundamentally
impossible to fully predict every way a change could break things
(which leaves us with Murphy's Law).  Testing alone is not going to
get us out of these sort of problems.  In fact one of the reasons we
are moving away from this codebase is that it scores pretty high on
the unmaintainability sale.

Secondly, I would like to update you on the attempts we are making to
minimize these problem releases and solicit specific contributions in
helping make this happen.

1)  Seneca has been working very hard to ensure that we get all new
code in 1.3 covered by automated tests.  As time goes on, we expect
full coverage of the application though this may not happen before
2.0.

2)  There is a general concensus that it would be helpful to form a
list of testers who would have access to -rc releases prior to general
release.  We are also discussing the exact minimum hold on an -rc
within a stable branch before it can be released (too long and we fix
fewer bugs, too short and we increase the risk of introducing bugs).
More contribution from users in this area would be of great help.

3)  We are working hard to move ourselves away from the current
codebase towards a structure less condusive of errors.  THis is
probably the most important aspect of the solution.