Re: Add BDD/Selenium tests (#1251)

[ snip ]

> The reason I was
> operating under the assumption that we need to use Perl for our
> testing has a number of reasons (which I write down here for others to
> challenge them):
>
> 1. We use Travis-CI specifically configured for Perl; I'm not sure
> which other languages are available and to what extent
> 2. With the developers we have at the moment, I'd expect to get up and
> running with a few basic scenarios most quickly in Perl
> 3. Our only option for testing seemed to be Selenium (because of
> SauceLabs), which looked "complete" at the Perl side
>

I think the main reason to use Perl, at least for part of this stack, is
so we can reuse internal functions for creating test data, instead of
rewriting that functionality in another language.

Ok. Agreed. It might be really practical to have a ways of inserting data through known-good methods. Though since we're only starting out now, I don't think we'll need this soonish :-)

However, I know we're on a path towards moving the essential business
logic to the database and providing other language bindings (and there's
already some for PHP) so that doesn't sound like a complete blocker.

True, but that depends on access through web services on the same server.

What I'm wondering though is the impact of mixing languages in a project. That is, we're now mainly a Perl project, with some JS and a bit of PL/pgSQL. Now Perl programmers working on bigger projects are likely to be familiar with all of JS, CSS, HTML(5) and PL/(pg)SQL.

Lets say that we end up deciding to go with Behat or Cucumber instead of Pherkin -- due to their better starting position. What will that do to our ability to attract developers? My thinking was: the languages and technologies, the better -- the more attractive the project to both developers (need fewer skills) and end-users (doesn't look too scary).

[ snip ]

> As far as using PhantomJS with Perl, this article suggests we could
> simply drive PhantomJS with the "standard" Selenium driver:
> http://blogs.perl.org/users/brian_medley/2013/02/headless-selenium-testing-with-phantomjs.html
> Could it be that that driver (or the phantom-framework) provides
> enough of an API? (Actually, it looks like PhantomJS integrated the
> webdriver protocol in its 1.8 release in 2012?)
>

Yes, I've seen that, too -- PhantomJS can be run as a server, and
supposedly talks the same wire protocol as the Selenium driver, so
should be able to swap this out interchangeably. In either case, you
just need to start up the driver before running your tests.

Agreed. There's one major advantage for PhantomJS over Selenium-with-SauceLabs: Selenium-with-SauceLabs can only be used on my local system with the same (hairy) bridge configuration that we now have for Travis-CI as well. (Which took quite a bit of experimenting to get set up correctly!). PhantomJS can run locally on the developer's machine without the need to set up any external connections or logins.

[ snip ]

> There are multiple reasons to want BDD, even though - as you point out
> correctly - it's perfectly fine for the current developers to write
> tests directly driving the Selenium or Poltergeist API.
> My reasons to want BDD are:
>
> 1. If we have BDD, we can solicit contributions to our testing
> framework from a much broader audience
> 2. If we have BDD, we can use that as discussion document format for
> new features to be implemented
> 3. BDD forces the creation of libraries for our testing routines. If
> you look at the t/*.t files, you'll find they get hard to follow quite
> quickly -- I'd expect the forced separation between step definitions
> and the actual steps to greatly help tests keep clean
> 4. Driving the web-tester APIs can get pretty tedious to do manually
> (error-prone); I'm thinking (hoping) that using BDD helps me gain
> developer efficiency here.
>
>
> What about my expectations (3) and (4)? Can you confirm those from
> your own experience?
>

I would agree with #1 - 3. Not sure about #4, though. I think it adds
yet another layer of abstraction, and complexity -- not reducing it.

Ok. I can see why you say that. My personal feeling is that the bits we need to realize the individual steps are probably much more contained/ simple than larger test programs that our application would make up otherwise. But, even if #4 turns out the way you say, then I think that #1 - #3 add more value than #4 will cost.

[ snip ]

After all my scouring of stuff yesterday, I think #2 is the critical
thing -- it frames your development in terms of user experience, so if
you're working on something you can't easily describe using BDD, it
raises the question "Should I really be working on this?"

Agreed. And at least for me, it will help me understand where other developers want to go when they announce to start working on a new feature or behaviour.

> -----
> While writing the above, I've tried setting up PhantomJS testing on
> Travis CI. Actually, that worked and I've successfully connected to it
> with the selenium webdriver!
>
> So, we have a choice now for our testing framework to run against
> SauceLabs (real Selenium Browser-based) or against PhantomJS, it seems.
> One thing that's built into SauceLabs which I don't know how to do
> with PhantomJS is to capture images of a page at the point of test
> failure. Can you help there?

I know that's a basic functionality of PhantomJS.

Actually, I found this to be a function of the Selenium Webdriver too. So, probably we can download the images from the test runs, as long as we figure out how to upload them to some public infrastructure that allows people to look at them.

In summary: my preference just moved from Sauce to Phantom, with the exception maybe if we want to test a specific browser version that's available on Sauce, but not based on Phantom's base library.

Bye,

Erik.

http://efficito.com -- Hosted accounting and ERP.

Robust and Flexible. No vendor lock-in.