Add BDD/Selenium tests (#1251)

Hi John,

Thanks for the elaborate description/review of the available options for BDD testing in #1251 (https://github.com/ledgersmb/LedgerSMB/issues/1251)! (Verbatim copy below for those who can't get to the issue easily).

I'll put my comments in a top posting, because that makes the response more compact. Also, since your text isn't marked as quoted, it makes more clear what's yours and what is mine.

I like how you run down each of the options we have at the various layers we need. To summarize (and verify my understanding), these are the layers:

* Testrunner

* Test framework/browser-driver-abstraction [not available for Perl, you say?]

* Browser driver(s)

* Step definitions

All of these are available for various languages (but not all for Perl), if I understand you correctly. So, that brings me my first assumption: that we need to use Perl for our testing. The reason I was operating under the assumption that we need to use Perl for our testing has a number of reasons (which I write down here for others to challenge them):

1. We use Travis-CI specifically configured for Perl; I'm not sure which other languages are available and to what extent

2. With the developers we have at the moment, I'd expect to get up and running with a few basic scenarios most quickly in Perl

3. Our only option for testing seemed to be Selenium (because of SauceLabs), which looked "complete" at the Perl side

The reason that we seemed limited to Selenium testing was that I didn't understand from the Travis-CI docs that we might be able to run other platforms like PhantomJS. Re-reading the docs with the background you gave me below, I now think they actually *do* provide PhantomJS on all images: https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-PhantomJS .

Your remark that 15 minutes go down to 1minute by switching from Selenium to PhantomJS caught my attention: I really like the fact that our test suite provides its feedback as quickly as it does now. Given that we have loads of tests to add for the in-browser testing, I'd be very much for using the fastest test platform available.

As far as using PhantomJS with Perl, this article suggests we could simply drive PhantomJS with the "standard" Selenium driver: http://blogs.perl.org/users/brian_medley/2013/02/headless-selenium-testing-with-phantomjs.html

Could it be that that driver (or the phantom-framework) provides enough of an API? (Actually, it looks like PhantomJS integrated the webdriver protocol in its 1.8 release in 2012?)

That's the technology side of things. Then you raise the question of why we need BDD, especially since we don't have a library of step definitions readily usable.

There are multiple reasons to want BDD, even though - as you point out correctly - it's perfectly fine for the current developers to write tests directly driving the Selenium or Poltergeist API.

My reasons to want BDD are:

1. If we have BDD, we can solicit contributions to our testing framework from a much broader audience

2. If we have BDD, we can use that as discussion document format for new features to be implemented

3. BDD forces the creation of libraries for our testing routines. If you look at the t/*.t files, you'll find they get hard to follow quite quickly -- I'd expect the forced separation between step definitions and the actual steps to greatly help tests keep clean

4. Driving the web-tester APIs can get pretty tedious to do manually (error-prone); I'm thinking (hoping) that using BDD helps me gain developer efficiency here.

What about my expectations (3) and (4)? Can you confirm those from your own experience?

-----

While writing the above, I've tried setting up PhantomJS testing on Travis CI. Actually, that worked and I've successfully connected to it with the selenium webdriver!

So, we have a choice now for our testing framework to run against SauceLabs (real Selenium Browser-based) or against PhantomJS, it seems.

One thing that's built into SauceLabs which I don't know how to do with PhantomJS is to capture images of a page at the point of test failure. Can you help there?
-----

Thanks again!

Regards,

Erik.

================== original posting

In IRC, we've been discussing options for adding browser-based tests, especially in 15 where so much is driven by _javascript_

Erik mentioned using Test::BDD::Cucumber for this, based on my enthusiasm for Behavior Driven Design tests based on what we're doing in Drupal He was asking me to help get this framework in place

After spending some time evaluating options, it looks like what is available for us to use in Perl is far less mature than what's available for Drupal

TL;DR - check out Test::Selenium::Remote::Driver (part of Selenium::Remote::Driver) That's what we need in place first

Layers/components needed for BDD

The huge head start Drupal has with BDD is a bunch of driver/support modules that provide > 120 step definitions right out of the gate, along with all of the integration of the various layers of tools

Test runner - pherkin, cucumber, behat

The top level is there for Perl -- at least at a basic level Test::BDD::Cucumber provides "pherkin" to run tests, and supports integration with Test::Builder for running them in an automated way

Cucumber is sort of the gold standard here A Ruby tool, Cucumber seems to have the most robust functionality available, and supports running tests in several languages -- though I did not see exactly how you would hook this up to step definitions written in Perl, and the code templates it outputs are in Ruby (though I did find references online suggesting it could create step definition stubs in other languages, but I did not find exactly how to do this)

Behat is PHP

Really, we could potentially use any of these tools -- if we're doing browser-side testing, the only reason we would need Perl is for the scaffolding around more easily inserting test data (eg by calling our own internal functions)

I actually tried setting up a simple feature to test the loginpl screen, and all 3 executed this file, with slightly different output:

All three used the same color-coding to indicate incomplete steps
Pherkin provided little data about the test run -- only the steps themselves
Cucumber and Behat provided more info about line numbers of sources, and output step definition templates (in Ruby or PHP)
Behat provided the line number of the beginning of each scenario Cucumber provided the line number of each individual step
Behat provides a "behat -dl" option that lists all available, pre-defined step definitions from all loaded sources (including your project's Feature Context, where you define your own)

Test framework

The next layer is where Perl seems to fall short

Behat ships with a driver called "mink" which essentially drives the web browser, and supports a bunch of actual drivers -- Selenium, Goutte, Zombie, BrowserKit, Sahi, and a couple others Mink handles starting up the browser, providing an abstraction layer across the various web drivers and integrating right into Behat

It sounds like the main solution for Cucumber here is "Capybara", which mainly uses Selenium or PhantomJS (via "Poltergeist"), and provides a similar role as Mink

For Perl, I didn't find anything on CPAN that looks quite equivalent -- it looks like much of this is done project-by-project The closest thing I found is here: https://githubcom/motemen/Wight, which connects to PhantomJS

This layer provides most of the convenience for writing tests that drive the web browser, but not supporting Gherkin directly -- rather, Cucumber/Behat sits on top of this layer to drive a variety of actual browser engines

Browser drivers

Here Perl has plenty to choose from, in the Mech:: namespace, and Selenium::Remote::Driver At this level you're writing tests for a specific browser driver, no abstraction layer (other than Selenium itself offers some cross-browser testing, grid testing, etc)

Here the main decision might be headless or not -- Selenium drives fully functional browsers, and that makes it slow -- although it also makes it visible, something you can watch in action

We've started using a new screen-comparison tool that uses PhantomJS, and it's so much faster -- tests that used to take 15 minutes to run now finish up in < 1 minute

Step definitions

This is where Drupal is so far ahead The DrupalExtension provides a bunch of step definitions already available for immediate use, for finding text on a page, filling out forms, finding text in particular rows of a table, generating user accounts, auto-logging a user in, creating content, users, taxonomy terms, and more

You can start writing a feature file in Gherkin and immediately execute it with much of it passing, not needing any step definitions written

What do we really need?

BDD and Gherkin is very useful for defining the user experience, mainly as a communication tool that can be tested very quickly -- if you already have an extensive library of step definitions

Since we don't have that library, there's quite a gap here in implementing all the pieces necessary to get this working

And given that the main goal here is testing the functionality of the web side of our application, and most of us (at least for the moment) are capable of creating these tests in code (especially with some examples to start with) I think we should start out with tests written using the Test::Selenium::Remote::Driver module directly, and not attempt to engineer a whole BDD testing stack to drive those Selenium tests

The alternatives:

Build out Wight or equivalent, and get it integrated with Test::BDD::Cucumber to a sufficient point -- probably quite a bit of effort to do this
Use Cucumber/Capybara/Ruby -- need to set up data manipulation using Ruby, or figure out how to get Cucumber to run Perl steps for when we want to enter data Introduces test dependency on Ruby (though if we're considering Sass, we may end up with that anyway -- Compass seems to be the de-facto Sass compiler)
Use Behat, and write steps in PHP -- we would be able to use many of the pre-defined steps from DrupalExtension, because they are simply targeting the DOM in the browser But now we have a PHP/Composer dependency, and any data population we want to do would need to go through a PHP binding (or escape to a shell)
Possibly try Cucumberjs -- I did not dig deep, but it does look like there are a ton of tools built on nodejs for this kind of processing, and it seems like the other vast ecosystem used across languages -- Ruby and Nodejs seem to be on par for processing tools for testing, CSS/Sass processing, etc And the dojo build step does introduce a dependency on nodejs for doing the _javascript_ build

None of these seem like the right option for a Perl project I've been scouring the web, and I'm finding very little on actually doing any tests driving browsers for testing using Perl Just a handful of modules that don't really talk to each other, relying on a lot of code from the project (us) to actually get working

—
Reply to this email directly or view it on GitHub.

Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.

Robust and Flexible. No vendor lock-in.