[ledgersmb-devel] Test coverage milestone

Hi all,

Roughly 4,5 years after the first coverage test run (12 September 2014) and little over 3 years after the first BDD test runs with coverage testing, on April 12th, we have finally reached the 40% coverage milestone demonstrating the effectiveness of the BDD tests to increase coverage.

To get to this point, we contributed to or even created various other Perl project:

* Test::BDD::Cucumber (bugfixes, plugin-capability, "prove" integration)

* Weasel (Web-app and Single Page App testing)
* Pherkin::Extension::Weasel (Weasel integration with Test::BDD::Cucumber)

The most recent addition here - added in March - was the ability to generate an HTML log with screenshots of the web session; for an example of such logs, including screenshots of the application after execution of the various test steps, see https://132-18471236-gh.circle-artifacts.com/0/tmp/artifact/logs/index.html

All of these contributions helped to accelerate the release schedule of LedgerSMB with fewer reported new issues on each release.

With the size of our current test suite, we were running into limitations of Travis CI, with our test suite now approaching maximum run time for a test job. Over the past months, we've therefore been working to move part of our test suite to CircleCI. As part of this migration, we've been confronted with many timing issues which made our test suite produce more than acceptable false-negatives (but never false positives, fortunately).

We've addressed all known instability issues now and moved testing of our master branch to run both on TravisCI and CircleCI. Our test coverage measurements (the slowest part of our tests) is produced on CircleCI and published on Coveralls.io (https://coveralls.io/github/ledgersmb/LedgerSMB?branch=master).

With the improved testing in place, I've been able to clear my backlog in old and pending changes. In the immediate future, new releases of Pherkin::Extension::Weasel and various Weasel::* modules are to be expected -- improvements to further help developer efficiency.

Apart from work on expanding the test suite itself, Yves and I are likely to start moving our test suite from 'prove' to 'yath' (https://metacpan.org/pod/distribution/Test2-Harness/scripts/yath), because it deals with UTF-8 from the start, where we're currently seeing warnings from 'prove' relating to UTF-8.

Regards,

Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.

Robust and Flexible. No vendor lock-in.