- We use jasmine to write our test cases (and also incorporate RequireJS)
- Automated tests need to run in the browser from a special Test Runner page that basically sets up RequireJS and a special jasmine.phantomjs-reporter.js file
- PhantomJS is used to run tests in our builds and from the CLI on our local machines
- Builds are executed on Jenkins
- We report code quality metrics for all of my team’s projects to Sonar
Requirements from a Code Coverage Tool
- It needs to be easy for developers to continue to run tests
- It needs to be easy to access its reports, otherwise developers will ignore it
- It needs to run with my team’s special “Test Runner” page in the browser (and handle RequireJS)
- It needs to be able to generate a report format that can be easily consumed by Sonar (specifically LCOV)
From these requirements, I ended up spending a significant amount of time evaluating the following tools:
Next up in this process was Blanket.js, which I must thank asciidisco for introducing to me since I didn’t know it existed until I saw his comment on github. Blanket.js was finally the right tool I was looking for in this process; it was REALLY easy to incorporate into my special test runner page. All I had to was download blanket.js, its special jasmine bootstrap file, and add the following lines of HTML to my page:
Blanket.js had a nice report design, and it helped guarantee that all developers that executed tests in the browse would see the coverage results since it ran in the browser alongside jasmine. I was really really close to adopting this tool, but it turns out that it currently lacks reporters that can easily generate the coverage data in a format that I can send to Sonar (specifically the LCOV format). I would have loved to have written a reporter to add this feature, but the demands of my job did not offer me this time.
Now that I had a code coverage tool that met all of my requirements, the last part was to get this code to run as part of our Jenkins build (which utilizes a grunt script). This was easy to get running, but I encountered two errors that consistently broke my builds:
- Sometimes phantomJS would fail to connect to the JSCover server
- Sometimes phantomJS would connect to the server, but then give up executing my tests at a random point during the run.
These were really weird issues that only occurred on my team’s Jenkins nodes and were hard to diagnose - even though they turned out to be simple fixes. For issue 1, that error was the result of my grunt script not waiting for JSCover to start before I executed phantomJS. For the second issue, it turns out that my team was using a special jasmine test runner to help with producing XML files after tests completed. The problem with this file was that it had a function that waited for Jasmine to complete its execution, but utilized an extremely short timeout before it gave up running the tests. This was a problem with Jenkins + JSCover because it took a longer time for the tests to load and run now that they had to be loaded from a web server instead of straight from the file system. Fortunately, this fix was as easy as increasing the timeout.
After all this, I am pretty happy with using JSCover because it was able to meet all the requirements that I noted above and was easy to get working. From my initial experiences with tools like JSCover and Blanket.js, they meet my need of generating coverage reports and are simple to get running, but there is a lot of room to grow in terms of integrating with toolsets that already exist in the market. In conclusion, I want to provide the following recommendations/feedback for anyone that is working on a code coverage tool:
- Be aware of the quality management and continuous integration tools that many software engineers use and try your best to easily integrate with them .
- Be explicit in your documentation as to whether you report the coverage for code that runs in node.js, the browser, or both (and provide docs as to how to get your tool to work).
- Try to simplify your set up process as much as possible and aggresively document it.
- This file was created to help output Jasmine’s test results from Phantom into XML files that can be consumed by Jenkins. This file was first introduced in this article.
- I may be completely wrong in this understanding of how Karma runs tests that need a specific test runner page.
- From my experience in the DevOps/Continuous Delivery space, Jenkins is one of the most popular CI tools in the marketplace. I’m really impressed with Sonar right now, but only recently discovered it so I am not sure how popular it really is.