How to execute specflow features without recompilation or visual studio

June 28, 2018 by rdagumampan

If you are new to BDD/TDD/Specflow/Cucumber/Gherkin buzz, its worth spending few minutes reading this. Intro to BDD

How about working with team from business that created 100+ Specflow feature scenarios before we even start writing a single C# code! It’s astonishing, TDD in its own form, and I love it! So we supported all behaviours and make them all passed over time. Deployed the services. Done.

As we get better, we receive more feature scenarios: like 5 today, and maybe 10 more next week. To support this, we need to continously recompile the test project and sometimes only to realize the tests failed due to faulty feature files or bad test data. This feedback loop has been working but could certainly be better.

What if our analysts can:

  1. Create new or modify existing feature files on his/her computer
  2. Run tests by issuing commands like these

    # execute a specific feature
    / SpecRunner.exe 
        /feature: FoobarFeatureFile.feature 
        /assembly: FoobarScenarioBindings.dll
    
    # execute all features from a drop folder
    / SpecRunner.exe 
        /feature: C:/FoobarFeatures 
        /assembly: FoobarScenarioBindings.dll
    
  3. Get a report and start discussions with dev team

Solution Demo (3-mins, best viewed in 1080p HD)

This vid shows a PoC where I used Specflow and xUnitAdapter to execute external feature files. The first run executes all embedded feature files in the DLL. Next run, I added a feature file and re-executed the tests. No recompile, runs on console.

Solution Details

It’s inspired by Gaspar Nagy’s (Specflow creator) sample project for the xUnitAdapter plugin for VS. I thought it may just work out well with a console runner.

Pre-requisites:

  • VS 2017 + Specflow for VS2017
  • .NET Framework 4.5.2 Developer Pack

On a fresh Specflow project in VS2017, add feature files and run the ff steps:

  1. Install Specflow xUnit adapter Install-Package SpecFlow.xUnitAdapter.
  2. Install xUnit console runner Install-Package xunit.runner.console.
  3. Remove all code behind files from .feature files,
  4. On each FoobarFeature.feature Properties, set Custom Tool empty.

    specflow-vs-2017

  5. Compile and run from console

    / cd C:\play\docker-sql-sandbox\specflow\packages\xunit.runner.console.2.3.1\tools\net452
    / xunit.console.exe 
        "C:\play\docker-sql-sandbox\specflow\SpecflowTests\bin\Debug\SpecflowTests.dll" 
        -html c:\temp\specflow-test-results.html
    

    specflow-vs-2017

  6. Add more feature files in the ExternalFeatures folder and re-run. Check out if new feature files are executed.

Alternatively, checkout full source code here: https://github.com/rdagumampan/docker-specflow

Limitations

  • The adapter only works with xUnit.

Solution alternatives

  1. Analysts uploads into repo. Auto-trigger CI/CD and run Specflow regression tests (as-is).
  2. Analysts runs in local machine, fixed local, deliver to dev when its good enough (preferred, but we keep #1).

Next steps

In real-world, these regression tests cannot run on its own. It needs all the interacting services and datastores. So I will create a portable container containing all the services and datastore that data analysts can orchestrate on their own while making more feature files or refinining the models to suite the tests scenarios. Watch out :)

References

Gaspar Nagy (Specflow creator) shares his story behind the Specflow code-behind
http://gasparnagy.com/2017/04/specflow-without-code-behind-files/
https://github.com/gasparnagy/SpecFlow.xUnitAdapter

Gherkin Overview
https://docs.cucumber.io/gherkin/reference/

© 2017 | About | Contact | Follow me on Twitter | Powerered by Hucore & Hugo