Task #48241

Work Package #48367: [WIP] End-To-End Testing of Neos with Behat

Create sandboxed Jenkins testing environment

Added by Christopher Hlubek about 2 years ago. Updated about 2 years ago.

Status:New Start date:2013-05-16
Priority:Should have Due date:
Assigned To:Robert Lemke % Done:

0%
Category:- Spent time: -
Target version:-

Description

Running tests (pre-merge / post-merge) should be separated from release management and deployment.

Some notes about the additional setup for running Behat tests with headless Selenium:

CI Setup for Selenium

  • Install Firefox
  • Run Apache on 8081 (parallel to Nginx)
  • Create VHost in Apache
  • Install MySQL

Ideas:

  • Post only message on failure (instead of -1)
  • Start Xvfb using Jenkins Xvfb plugin (run in parallel in start / stop after tests)
  • Capture video (headless gem)
    • Start capturing video before each scenario

History

#1 Updated by Sebastian Kurfuerst about 2 years ago

  • Parent task set to #48367

#2 Updated by Christopher Hlubek about 2 years ago

We created an account on Sauce Labs (http://saucelabs.com/) for Neos (it's free for Open Source projects). They provide excellent browser testing VMs and support for Selenium WebDriver, so we can implement a build matrix for the Behat tests on different device / browser combinations. It's also much easier to debug errors or problems in failing tests since they provide a detailed log and captured video for each test.

I already installed a Jenkins plugin for Sauce Labs, but we still have to split the job into at least one downstream job for running the Behat tests. A build pipeline for review could look like this:

  • Gerrit Trigger for Neos package dependencies (on patchset created)
    • Run Unit Tests -> publish results
    • Run Functional Tests -> publish results
    • Run Behat tests on different browsers (if Unit / Functional tests are okay)
    • Publish final results to Gerrit (Set verified +1, but do not set -1 for failure)

Also available in: Atom PDF