Deciding on a default company setup for BDD tests

Hi all,

Chris, John and I have been slowly working our way to creating infrastructure on which we can base browser-based BDD tests. We had some problems with race conditions between the HTML/JS renderer (PhantomJS) and the expectations being tested in the test-driver (Selenium::Driver). However, these have been fixed as of this morning.

Earlier today, I merged the first feature file (2 tests) to 'master'. This feature file does nothing more than just navigate to /setup.pl and /login.pl and verify that the credentials text boxes are displayed.

Now that we're able to create feature files and write step files (and we know what we need to do to prevent these race conditions), I'm thinking that we need to devise a generally applicable structure on how tests are initialized, torn down, cleanup takes place, etc.

John and I were talking how we'd like tests to clean up behind themselves, removing database objects that have been added in the testing process, such databases, (super/login) roles, etc...

To start with the first and foremost question: do we want our tests to run succesfully on a copy of *any* company (as John stated he would like, on IRC) or do we "design" the company setups we want to run our tests on, from scratch, as I was aiming for? (Note that I wasn't aiming for regenerating all setup data on each scenario or feature; I'm just talking about making sure we *know* what's in the database -- we'd still run on a copy of a database set up according to this plan).

I'm thinking a lot of next questions depend on the answer to this one, so, I'll leave it at this for now, with respect to test definitions.

Additionally, John and I were talking about supporting test infrastructure and we agree that it would be tremendously helpful to be able to see screenshots of failing scenarios and maybe to be able to see screenshots of various points in non-failing tests too. Since Travis storage isn't persistent, we were thinking that we'd need to collect all screenshots as "build artifacts" and upload them into an AWS S3 account for inspection.

Comments?

Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.

Robust and Flexible. No vendor lock-in.