Panellizing BDD tests: feature file execution

Hi Yves,

On IRC, we were talking about making the BDD tests be executed in parallel. At the time, I objected to declare them 'parallel' as of now, because there are interactions between the feature files. (Mind you that I'm deliberately not saying that the scenarios can be parallellized at all, which is a completely different problem.)

Let me take this opportunity to explain what the problem for the BDD tests and parallellism is: all feature files use a "template" database to speed up the execution of the scenarios: for each scenario a new database is created, unless specified otherwise. However, database creation takes a *lot* of time. So, instead of creating a new database from scratch, each feature creates a template database once and copies that database later. Which is *much* faster. Also, each feature file creates the template to ensure tests are being run against the latest version of the schema. Especially when running these tests on a developer's local system, this is important. It's much less important on Travis, as that starts out with a clean system anyway.

Taking a tour around the code, here is where the name of the template database is being set:
https://github.com/ledgersmb/LedgerSMB/blob/master/t/lib/Pherkin/Extension/LedgerSMB.pm#L32

This is the code to create the template database (and an initial user):
https://github.com/ledgersmb/LedgerSMB/blob/master/t/lib/Pherkin/Extension/LedgerSMB.pm#L117

This is where the template creation is triggered, unless the template has been created already:
https://github.com/ledgersmb/LedgerSMB/blob/master/t/lib/Pherkin/Extension/LedgerSMB.pm#L173

And this is the step definition which triggers the creation of the template from the feature files:
https://github.com/ledgersmb/LedgerSMB/blob/master/t/lib/Pherkin/Extension/ledgersmb_steps/common_steps.pl#L13

When the template exists and a new test requires a new company database, it's created through this step implementation:
https://github.com/ledgersmb/LedgerSMB/blob/master/t/lib/Pherkin/Extension/ledgersmb_steps/common_steps.pl#L13

Note that it uses the global variable `$company_seq` right above the step definition to create sequentially numbered (unique) test databases. These test databases are all called "standard-<sequence-number>". This is a problem under the `prove` execution of the feature files, because the features are executed as separate processes (forks) and by consequence the sequence numbers will be re-used.

Today I was thinking what we can do to speed up the testing of the feature files under `prove` with parallellism. This is what I came up with:

* If we change the 'ensure_template` function to create a template unless the environment variable LSMB_NEW_DB has a value, in which case we assume that to be a valid template.

* If we change the name of the test database instances to include the $$ (process id) (next to the sequence number, that is), we will have unique test database names, at least for the duration of the test run.

The second point should probably include removing the database when the test completes successfully (because otherwise the local developer's database gets poluted with loads of test remnants).

Comments?

Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.

Robust and Flexible. No vendor lock-in.