| Subcribe via RSS

Test Suites: To Critic or Not?

February 19th, 2014 | 1 Comment | Posted in CPAN, Perl

Over the holiday weekend, I spent a good amount of time cleaning up my RPC::XML test suites with regards to Perl::Critic. It wasn’t quick work, either: Much of the suite was written when RPC::XML was first under development, which makes a lot of it well over 10 years old. But I had an extra day, and plenty of time to kill. So 25 files and one git commit later, my test are all clean with respect to my chosen Perl::Critic configuration. The exceptions are a couple of Modules::ProhibitExcessMainComplexity complaints, and most of them also chirp about Modules::RequireVersionVar because the policy doesn’t allow for exclusion of files like test suites.

But all through this, I was wondering: Does it really matter if your test suites are as clean as the code itself?

How often does someone else look at your test suites as closely as they look at the modules that those tests support? I’m not setting up a life-lesson here, I’m genuinely curious. I spent some time thinking about it, and I can say that the only times I’ve looked at test suites in other distributions has been when I was trying to figure out how they did something, or how a particular testing module is used. I’ve very rarely looked for examples of how to write my own tests, and I probably should. I’m pretty sure I could learn from some (most?) of them.

See, I look at Perl::Critic as at least partly geared towards the readability of code; using Perl::Critic helps me make my code more consistent in style and structure, which in turn makes it more readable. I introduced it into my day job (along with a web front-end) largely to try and encourage certain consistencies in style, uniformity even, in the name of corralling Perl coders who had no collective style. Yes, there are a lot of other benefits to critic that are there to be had. It has led me directly to bugs, or to code that could potentially fail, on numerous occasions. But for me, myself, style and structure are the order of the day. So, if few people look at your test suites, is there any reason to run them through the critic?

After a day or two of idle background-thought on it, I’ve decided that it is worth it. Even if the only person who looks at your tests is you. Because you do eventually have to look at those tests. And RPC::XML is not even remotely the oldest CPAN module, so there are distributions out there whose tests are much older than my 10+ years. And that’s plenty of time to forget WTF you were thinking when you wrote the test. The least you can do is try to make it readable for the next time you have to look at it.

Tags: , , ,

Perl Module Release: Test-AgainstSchema 0.100

January 17th, 2012 | 2 Comments | Posted in CPAN, Perl, Software
Version: 0.100

Released: Monday January 16, 2012, 02:00:00 PM -0800

Changes:

  • lib/Test/AgainstSchema.pm (added)
  • lib/Test/AgainstSchema/XML.pm (added)
  • t/10_xml/05_xml_parsing.t (added)
  • t/10_xml/10_sgmldtd.t (added)
  • t/10_xml/20_xmlschema.t (added)
  • t/10_xml/30_relaxng.t (added)
  • t/10_xml/basic_tests.pl (added)
  • t/10_xml/simple-bad.dtd (added)
  • t/10_xml/simple-bad.rng (added)
  • t/10_xml/simple-bad.xsd (added)
  • t/10_xml/simple.dtd (added)
  • t/10_xml/simple.rng (added)
  • t/10_xml/simple.xsd (added)
  • t/10_xml/xml-simple-bad.xml (added)
  • t/10_xml/xml-simple.xml (added)
  • t/90_integration/10_with_xml.t (added)
  • t/90_integration/90_empty_subclass.t (added)
  • t/90_integration/MyTestAgainstSchema.pm (added)
  • t/util.pl (added)
  • xt/00_load.t (added)
  • xt/01_pod.t (added)
  • xt/02_pod_coverage.t (added)
  • xt/03_meta.t (added)
  • xt/04_minimumversion.t (added)
  • xt/05_critic.t (added)

Initial commit of code, tests and test files.

Tags: , , , ,

Continuing Adventures with Devel::Cover

July 17th, 2011 | No Comments | Posted in CPAN, Perl

My excursions into code-coverage testing continue. Since my last post on the subject, I’ve released a new version of one my smaller modules. This release of the code, besides fixing a few small bugs, marks a full 100% coverage-complete test suite: statements, branches, conditionals and subroutines. (Though to be fair, there is only one subroutine in the module.)

Meanwhile, work on RPC::XML continues at a reasonable pace. I have 3 modules at 100% statement coverage (2 that I think might be, but aren’t due to what I suspect are bugs in Devel::Cover), 5 modules at 100% subroutine coverage (plus 1 that should be, but for the same bugs) and 1 module at 100% branch coverage. Presently, the test suite has 354 more tests than were in the 0.74 release— an increase of just over 57%. There were other things than this that I had hoped would be in the 0.75 release, but my change-list is getting awfully long. So I may cut the release after I finish work on extending the code coverage on the remaining modules, and leave the other work for the next cycle (when I have much better coverage in my test suites).

I’m really happy with what this tool has brought to my efforts. I’ve opened a couple of bug-tickets in RT, for issues that I’ve found. I hope that once I have RPC::XML 0.75 ready to go, I can open some more on the issues I’m seeing and be able to point them to specific lines of specific modules that exhibit the behavior.

Tags: , ,

I (Apparently) Suck at Writing Tests

July 3rd, 2011 | No Comments | Posted in CPAN, Perl

(This probably looks familiar. Somehow, my WordPress install got rolled-back a few days. I don’t know how or why. I only know that this post disappeared, as did changes to a plug-in and changes to my “On Javascript and Flash” page. So apologies for seeing this post twice. I’m restoring it from Google’s cache. Clearly I need to spend what remains of the long holiday weekend setting up automated regular backups of my WP installation, something I should have already done were I not so lazy.)

As I go in to what is (for me) a 4-day weekend*, my plans are largely based around writing tests. Lots of tests, most likely. Because, it seems, I’m not very good at it.

That’s a pretty harsh statement for me to make, especially considering that I work in a QA organization. (Though, to be fair, I don’t actually write tests here— I work on the framework that my organization uses to develop their automated test suites.) And (to be fair) I’m probably being overly hard on myself. But I recently started playing around with Devel::Cover, and I was shocked to find out how much of my code is not covered by my tests.

The gist I show above is the sequence of commands that I use to generate coverage data. I adapted this from someone else’s blog post about Devel::Cover. It’s simple, but the part that had kept me from using this before now was not knowing how to apply it to the test suite as a whole, which is what setting HARNESS_PERL_SWITCHES does. (Turns out this is also covered in the man page for Devel::Cover, but for some reason I hadn’t seen that…)

The results have not been pretty.

My most-used module, Image::Size? Overall coverage is 57%, with statement coverage at 67.7%. My RPC::XML module is currently at 78.1% overall (84.1% statement) in my development copy, but that’s after many hours of work over the last week or so. It started out in the low 60s. And I haven’t even applied this to all of my modules.

It seems that my weakest area is in negative testing— I’m just not covering all of my error cases. In one module from RPC::XML, the statement coverage was barely over 50% because the module itself (RPC::XML::Parser) is meant as a common base-class with some code in new() to allow for backwards-compatibility with older versions of RPC::XML. I define the other methods that child classes should override, but I never tested the error-catching code that ensures that the methods are overridden. An hour and a new test-suite later, and this is one of only two modules that has 100% statement coverage (but not 100% overall… it’s only 66.7% covered on conditionals).

Of course, I’m being overly hard on myself. And I’m not really that serious about considering myself so poor at test development. What this (ongoing) exercise has shown me, is how much more I have to learn about it. I do try to practice test-driven development where I can, but I’m not sure that has yielded good coverage in all cases. I tend to practice defensive coding, trying to cover all the potential error cases where I can. But that is no guarantee that I have written tests for all of the potential error cases. Indeed, so far it’s pretty clear that I haven’t.

So if you haven’t looked in to Devel::Cover yet, give it a go. It’s been a great help to me!

* My employer is celebrating our ranking on Fortune’s Great Places to Work list by giving us an extra day off for the holiday weekend. There’s a reason why I like this place!

Tags: , ,