How Much Code Coverage is Enough?
Measuring code coverage allows you to see exactly what percentage of your codebase you're testing. What should we be aiming for: 90%, 100%, or something else entirely?
We value automated testing at Viget, but we don't have a specific rule on code coverage from client to client. Between personal projects and Viget clients, my testing habits vary widely from scrappy MVP code with minimal tests, to robust end-to-end test suites with 100% coverage. The latter got me thinking: how much does 100% coverage really tell us? I'd argue that, like most things in life, what we should really be striving for here is quality and not quantity. I should note that I'm writing particularly about using Rails, RSpec, and SimpleCov here.
Pros of Testing
- Well written tests communicate intent to fellow developers.
- Well written
itblocks lay out a map for other developers to come in and understand how the code is expected to work.
- Well written
- Testing gives you confidence in edge cases and error handling.
- Errors will happen. While you can't realistically account for every possible edge case, you can write tests for sad paths to be confident they're handled appropriately.
- Integration tests make refactoring less scary.
- With thorough integration tests, you can rip apart a large feature X and your test suite will ensure there aren't any unintended side effects on feature Y or Z.
- Unit tests makes refactoring less scary.
- Writing that
it should have_many :photosspec may seem trivial, but you'll be happy to have it when you change the
photostable name to
images. You'll catch that the relationship name needs to change.
- Writing that
Cons of Testing
- They take 5ever to write.
- The more you're in the habit of writing integration specs, the faster you'll become at writing them. Even so, it's always going to be a long, sometimes tedious task to test. However, the hours spent writing tests before launch feel a lot better than the hours spent trying to hot fix a bug that's gone to production.
- If you wish to make an apple pie from scratch, you must first create the universe.
- Let's say you need to test the order confirmation page. You need an order. To create an order, you need a user and some products. You need a way to mock payment processing. The setup bloat can get very big, very fast.
- Tools like FactoryBot as well as spec helper files that break out common flows will make your life easier. Taking the time to create methods like
submit_successful_paymentthat are accessible by multiple specs will make your life (and a future developers life!) easier.
- API calls are tedious.
- You're writing an integration test that requires making an API call to an external dependency. You don't want to hit your rate limit or be charged just for running tests.
- VCR is your friend here. Make the call once and store the request/response to use every time the test runs.
100% coverage does not mean your code is flawless. Firstly, it's easy for that 100% number to become meaningless quickly. You could theoretically put
#:no-cov#s all over the place and claim 100% coverage without having even 50% coverage (don't do this!!).
On the other hand, you can "cover" a line of code without testing anything meaningful at all. Consider a method
def display_text "Your order has been received." end
You could write a unit test showing that the
display_text method matches a string. That method would be 100% covered. A more meaningful test would be to write an integration test that this
display_text shows up on the order confirmation page. 100% covered doesn't necessarily mean a codebase is error-proof.
My Two Cents
Use a tool like SimpleCov and set a coverage goal, somewhere between 90-100%. You'll want to consider the level of QA engagement, development time available to get a feature out, and how risk averse you are. Mostly, don't get too caught up in that specific number. High coverage != well tested. If you're focused on the number, you could end up with perfunctory tests that don't provide much value. Instead, focus on writing a solid mix of unit tests and integration tests that account for happy and sad paths, and the coverage will come.