I’m a big believer in the benefits that Continuous Delivery provides to help teams ship software. An important component of this is the delivery pipeline, which aims to automate the entire process from checking in code, running a build, testing, to ultimately deploy the code into a customer facing environment. A critical part to getting this pipeline to work is to ensure the quality of the code that gets delivered - which is done by running automated tests to ensure that bugs don’t occur. However, one problem with automated tests (specifically unit tests) is that it is easy to miss testing certain parts of your codebase (e.g. hitting all cases for an if statement). Fortunately, there are many code coverage tools that exist for languages to help determine if your tests hit all parts of your code. However, this type of tooling is still a bit young in the javacript space and I want to share my findings on the tools I evaluated while incorporating code coverage into my team’s automated javascript tests.
As I said before, the tooling for code coverage in Javascript is still pretty young in my opinion (especially when compared to java). Fortunately, the introduction of node.js and growth of the javascript community in the past few years has helped this problem tremendously. However, this popularity (and youth) actually created a few problems for me during my evaluation and adoption of tools. Before I get into this, I want to describe the details of how my team’s automated tests run and the requirements I needed from the eventual tool:
Test Environment
- We use jasmine to write our test cases (and also incorporate RequireJS)
- Automated tests need to run in the browser from a special Test Runner page that basically sets up RequireJS and a special jasmine.phantomjs-reporter.js file[1]
- PhantomJS is used to run tests in our builds and from the CLI on our local machines
- Builds are executed on Jenkins
- We report code quality metrics for all of my team’s projects to Sonar
Requirements from a Code Coverage Tool
- It needs to be easy for developers to continue to run tests
- It needs to be easy to access its reports, otherwise developers will ignore it
- It needs to run with my team’s special “Test Runner” page in the browser (and handle RequireJS)
- It needs to be able to generate a report format that can be easily consumed by Sonar (specifically LCOV)
From these requirements, I ended up spending a significant amount of time evaluating the following tools:
I started with evaluating istanbul thanks to this blog post, which sold me on how “easy” it was to incorporate code coverage into your javascript tests. The one problem I had with this article is that it is specifically targeted at Jasmine tests that are NOT running in a browser. This detail ended up wasting a good bit of my time in trying to get istanbul to work because this nuance was not documented well.
I then evaluated Karma, which is just a test runner for javascript that makes it easy to boot all of the browsers on your local machine, run tests, and then report the results to the command line. I really liked Karma, it incorporated istanbul’s functionality and it was really easy to get its initial set up to work. However, I ended up dropping this tool because I was unable to get it to run our special “Test Runner” page, which mostly does some library and RequireJS setup while ensuring that each test runs in its own RequireJS context. From what it looked like to me, Karma didn’t handle running our specific test runner page and instead wanted to automagically inject our tests into its own runner page.[2]
Next up in this process was Blanket.js, which I must thank asciidisco for introducing to me since I didn’t know it existed until I saw his comment on github. Blanket.js was finally the right tool I was looking for in this process; it was REALLY easy to incorporate into my special test runner page. All I had to was download blanket.js, its special jasmine bootstrap file, and add the following lines of HTML to my page:
<script type="text/javascript"
data-cover-only="/"
data-cover-never="['/lib','/test']"
data-cover-flags="branchTracking"
src="lib/blanket.min.js"></script>
<script type="text/javascript" src="lib/jasmine-blanket.js"></script>
Blanket.js had a nice report design, and it helped guarantee that all developers that executed tests in the browse would see the coverage results since it ran in the browser alongside jasmine. I was really really close to adopting this tool, but it turns out that it currently lacks reporters that can easily generate the coverage data in a format that I can send to Sonar (specifically the LCOV format). I would have loved to have written a reporter to add this feature, but the demands of my job did not offer me this time.
So after all this, I came around to using JSCover. This is a code coverage tool that is the child of JSCoverage and utilizes a java server to take your javascript files, dynamically inject coverage instrumentation code, and report the results in its UI. I first disregarded this tool when I started this research because I did not like adding another runtime dependency to my team (i.e. requiring Java to be installed and starting the server before running the tests). However, at this point, I didn’t care and just wanted something that worked that also supported LCOV reports. Fortunately, it was easy to get my automated tests running in this tool (even easier than blanket.js because of some bugs I encountered). All I had to do was start the server, have it look at the right source directory, and then run it against a special URL. On top of this, it also had built in support for integrating with PhantomJS and it was super easy to convert its reports into various formats.
Now that I had a code coverage tool that met all of my requirements, the last part was to get this code to run as part of our Jenkins build (which utilizes a grunt script). This was easy to get running, but I encountered two errors that consistently broke my builds:
- Sometimes phantomJS would fail to connect to the JSCover server
- Sometimes phantomJS would connect to the server, but then give up executing my tests at a random point during the run.
These were really weird issues that only occurred on my team’s Jenkins nodes and were hard to diagnose - even though they turned out to be simple fixes. For issue 1, that error was the result of my grunt script not waiting for JSCover to start before I executed phantomJS. For the second issue, it turns out that my team was using a special jasmine test runner to help with producing XML files after tests completed. The problem with this file was that it had a function that waited for Jasmine to complete its execution, but utilized an extremely short timeout before it gave up running the tests. This was a problem with Jenkins + JSCover because it took a longer time for the tests to load and run now that they had to be loaded from a web server instead of straight from the file system. Fortunately, this fix was as easy as increasing the timeout.
Conclusion
After all this, I am pretty happy with using JSCover because it was able to meet all the requirements that I noted above and was easy to get working. From my initial experiences with tools like JSCover and Blanket.js, they meet my need of generating coverage reports and are simple to get running, but there is a lot of room to grow in terms of integrating with toolsets that already exist in the market. In conclusion, I want to provide the following recommendations/feedback for anyone that is working on a code coverage tool:
- Be aware of the quality management and continuous integration tools that many software engineers use and try your best to easily integrate with them [3].
- Be explicit in your documentation as to whether you report the coverage for code that runs in node.js, the browser, or both (and provide docs as to how to get your tool to work).
- Try to simplify your set up process as much as possible and aggresively document it.
TL;DR
If you want to get code coverage reports for Javascript tests that run in the browser, your best choices are Blankt.js or JSCover. If you also need to integrate these reports into your team’s continuous integration and quality management tools, JSCover is the best choice as it has support for generating reports in various formats.
Notes
- This file was created to help output Jasmine’s test results from Phantom into XML files that can be consumed by Jenkins. This file was first introduced in this article.
- I may be completely wrong in this understanding of how Karma runs tests that need a specific test runner page.
- From my experience in the DevOps/Continuous Delivery space, Jenkins is one of the most popular CI tools in the marketplace. I’m really impressed with Sonar right now, but only recently discovered it so I am not sure how popular it really is.