Re: Add BDD/Selenium tests (#1251)

[John Locke <..hidden..>]

Hi, Erik,

Notes below...


On 01/10/2016 06:46 AM, Erik Huelsmann wrote:
> Hi John,
>
>
> Thanks for the elaborate description/review of the available options 
> for BDD testing in #1251 
> (https://github.com/ledgersmb/LedgerSMB/issues/1251)! (Verbatim copy 
> below for those who can't get to the issue easily).
>
> I'll put my comments in a top posting, because that makes the response 
> more compact. Also, since your text isn't marked as quoted, it makes 
> more clear what's yours and what is mine.
>
> I like how you run down each of the options we have at the various 
> layers we need. To summarize (and verify my understanding), these are 
> the layers:
>
>  * Testrunner
>  * Test framework/browser-driver-abstraction [not available for Perl, 
> you say?]
>  * Browser driver(s)
>  * Step definitions
>
> All of these are available for various languages (but not all for 
> Perl), if I understand you correctly. So, that brings me my first 
> assumption: that we need to use Perl for our testing. The reason I was 
> operating under the assumption that we need to use Perl for our 
> testing has a number of reasons (which I write down here for others to 
> challenge them):
>
> 1. We use Travis-CI specifically configured for Perl; I'm not sure 
> which other languages are available and to what extent
> 2. With the developers we have at the moment, I'd expect to get up and 
> running with a few basic scenarios most quickly in Perl
> 3. Our only option for testing seemed to be Selenium (because of 
> SauceLabs), which looked "complete" at the Perl side
>

I think the main reason to use Perl, at least for part of this stack, is 
so we can reuse internal functions for creating test data, instead of 
rewriting that functionality in another language.

However, I know we're on a path towards moving the essential business 
logic to the database and providing other language bindings (and there's 
already some for PHP) so that doesn't sound like a complete blocker.



> The reason that we seemed limited to Selenium testing was that I 
> didn't understand from the Travis-CI docs that we might be able to run 
> other platforms like PhantomJS. Re-reading the docs with the 
> background you gave me below, I now think they actually *do* provide 
> PhantomJS on all images: 
> https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-PhantomJS 
> .
>
> Your remark that 15 minutes go down to 1minute by switching from 
> Selenium to PhantomJS caught my attention: I really like the fact that 
> our test suite provides its feedback as quickly as it does now. Given 
> that we have loads of tests to add for the in-browser testing, I'd be 
> very much for using the fastest test platform available.
>
> As far as using PhantomJS with Perl, this article suggests we could 
> simply drive PhantomJS with the "standard" Selenium driver: 
> http://blogs.perl.org/users/brian_medley/2013/02/headless-selenium-testing-with-phantomjs.html
> Could it be that that driver (or the phantom-framework) provides 
> enough of an API? (Actually, it looks like PhantomJS integrated the 
> webdriver protocol in its 1.8 release in 2012?)
>

Yes, I've seen that, too -- PhantomJS can be run as a server, and 
supposedly talks the same wire protocol as the Selenium driver, so 
should be able to swap this out interchangeably. In either case, you 
just need to start up the driver before running your tests.

>
> That's the technology side of things. Then you raise the question of 
> why we need BDD, especially since we don't have a library of step 
> definitions readily usable.
>
> There are multiple reasons to want BDD, even though - as you point out 
> correctly - it's perfectly fine for the current developers to write 
> tests directly driving the Selenium or Poltergeist API.
> My reasons to want BDD are:
>
> 1. If we have BDD, we can solicit contributions to our testing 
> framework from a much broader audience
> 2. If we have BDD, we can use that as discussion document format for 
> new features to be implemented
> 3. BDD forces the creation of libraries for our testing routines. If 
> you look at the t/*.t files, you'll find they get hard to follow quite 
> quickly -- I'd expect the forced separation between step definitions 
> and the actual steps to greatly help tests keep clean
> 4. Driving the web-tester APIs can get pretty tedious to do manually 
> (error-prone); I'm thinking (hoping) that using BDD helps me gain 
> developer efficiency here.
>
>
> What about my expectations (3) and (4)? Can you confirm those from 
> your own experience?
>

I would agree with #1 - 3. Not sure about #4, though. I think it adds 
yet another layer of abstraction, and complexity -- not reducing it. 
Perhaps after we build up a large library of step definitions that we 
can re-use, there might be some benefit here -- but until we have that, 
we're writing the tests twice, once in Gherkin, once in whatever 
language we end up writing the step definitions.

After all my scouring of stuff yesterday, I think #2 is the critical 
thing -- it frames your development in terms of user experience, so if 
you're working on something you can't easily describe using BDD, it 
raises the question "Should I really be working on this?"
>
>
> -----
> While writing the above, I've tried setting up PhantomJS testing on 
> Travis CI. Actually, that worked and I've successfully connected to it 
> with the selenium webdriver!
>
> So, we have a choice now for our testing framework to run against 
> SauceLabs (real Selenium Browser-based) or against PhantomJS, it seems.
> One thing that's built into SauceLabs which I don't know how to do 
> with PhantomJS is to capture images of a page at the point of test 
> failure. Can you help there?

I know that's a basic functionality of PhantomJS.

http://phantomjs.org/screen-capture.html

Cheers,
-- 
John Locke
http://www.freelock.com


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Ledger-smb-devel mailing list
..hidden..
https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel