| Subcribe via RSS

CPAN Testers Follow-Up: Progress Is Made

February 13th, 2014 | 4 Comments | Posted in CPAN, Perl

Or, How I Learned It (Mostly) Wasn’t My Fault

So, in my previous post I noted how my reward for getting back on the horse had been a slew of failure reports from the CPAN Testers service. This is a follow-up on what I’ve learned these last 6 days.

First, Do Not Panic

You’ve just gotten a barrage of FAIL reports from CPAN Testers. Don’t get yourself agitated over this. Think about it: You’ve just been handed (free of charge) the results of someone else taking the time to test your product for you, probably on a combination of architecture/OS/Perl that you didn’t have available to test yourself. This is a good thing for you. Once upon a time, you wouldn’t have had this until someone was actually using your code in the field. Then the report you got would have been from someone annoyed and/or angry over breakage. Instead, you’ve gotten some number of reports that look like this, which can hopefully help you zoom in on the issue.

I (kind of) panicked. Fortunately, after someone responded to my previous post I was able to calm down and start looking at the content of the reports more rationally.

Second, Figure Out What’s Gone Wrong

Look, you wrote the module in question. This means that you are almost certainly the best-suited to determine what went wrong. (Not always, as I’ll point out a little later, but probably.) Look at the reports carefully, see what tests failed. Look at those tests yourself and see what could have tripped them up. Look for counter-examples, or noteworthy differences. In my case, the majority of my test failures were caused by my server class (RPC::XML::Server) failing to instantiate because (it claimed) the address it was told to bind to was in use. This didn’t make sense, as I had code I was using that actually scanned for a port that wasn’t in use. The port that the server was trying to bind to had, just moments before, been judged available by virtue of the fact that a raw socket connection to it had failed. So what was wrong? Why was it failing? And more interestingly, why was it failing in these cases, but not in the dozens of PASS reports that I had also racked up?

I noticed that all the failing cases had something in common: a comment in the test output stream that a package called Acme::Override::INET was monkey-patching IO::Socket::IP in place of IO::Socket::INET. I had my first big clue.

Third, Prepare an Environment for Testing

If you aren’t using perlbrew, then either you aren’t testing as well as you could be or you’re spending way more effort on your testing than you need to be. I first started using it because the Perl that comes with Mac OS is older, and I wanted to be able to install and update CPAN modules more easily. Then I started keeping a half-dozen or so different additional installations of Perl around, at various versions with and without threading. If you’re developing modules, it’s an invaluable tool for testing your modules across different versions. It also meant that I could install Acme::Override::INET and IO::Socket::IP in a Perl other than my “main” one, so that I didn’t upset my primary work/development environment.

So I did just that; I picked a version of Perl that I had that was identical to one that had failed in one of the reports, and installed those two modules.

Fourth, Learn What You Can

Suddenly, I too was getting the exact same failures in the exact same places. This was great! I started throwing in some debugging statements, and verified that in each case a previous server allocation had succeeded but the next allocation (which was re-using the same port) would fail. For some reason, IO::Socket::IP seemed to be causing sockets (or at least the port allocations) to hold on beyond the life of the socket. I even managed to show that the error occurred over the process boundary, when I ran a test twice in rapid succession and the second run failed in the first server allocation! When I removed Acme::Override::INET and IO::Socket::IP, it immediately started working again. Same tests, same code, same Perl executable… the only difference was IO::Socket::IP.

(I should note here that un-doing the installation of Acme::Override::INET is tricky. It installs a “fake” IO/Socket/INET.pm file, but it uses Module::Build to do so. Unless you have M::B configured to use the same installation locations as ExtUtils::MakeMaker, M::B will install in a directory that is earlier in your search path than the directory that E::MM uses. So re-installing the IO module will not overwrite the IO::Socket::INET fake that Acme::Override::INET installed. I had to delete all four files that were installed.)

Fifth, Do Something with Your New Knowledge

So I opened up an RT ticket for IO::Socket::IP, and explained my situation at length in it. The module’s author replied and pointed out that the way I was selecting a port was much more work than I needed to do in that case. See, I’m not actually very adept at network programming. That’s actually an understatement… if it weren’t for HTTP in general, and HTTP-related Perl modules like LWP in particular, I would probably not have a CPAN module to my name that is so network-centric. Anyway, I at least felt at this point that there wasn’t anything wrong with RPC::XML::Server. But based on the suggestions from the IO::Socket::IP author, I changed most of my server-oriented tests to let the kernel allocate a random port instead of me selecting one. Now the tests all pass, even with the newer module and the monkey-patch installed. Mind you, there is still an issue with IO::Socket::IP, but I can’t help there. I can only hope that the information I’ve provided helps him out.

Sixth, and This Is Important, Be Content With It

It’s frustrating to know that there will still be some FAILs coming in, but I can’t help that. I can be happy with the fact that my tests are fixed, that there wasn’t actually anything wrong with my networking code (because I’d be close to lost if I had to go into a deep dive to troubleshoot that part). There are still things to fix and add to the RPC::XML package, but my server tests are all now a bit more robust than they were this time last week. I’ve found and reported an issue that will hopefully help another CPAN author. And, importantly for me, I’ve stayed engaged in my code for this last week. That’s been hard for me to do for a long time now, and it feels really good to have done it. Even though it’s only been about 5-6 hours of effort spread across that many days, I’m happy about it, and that will hopefully mean that the next 5-6 hours will come more easily.

Tags: , ,

CPAN Testers and RPC-XML: Well, Crap

February 7th, 2014 | 1 Comment | Posted in CPAN, Perl

So I’m starting to break my drought on personal-project coding, and yesterday I cut a new release of RPC-XML, the first one since September of 2012. I’m pretty happy about this; besides the fact that I was able to clear out my pull-request queue on GitHub, this means that I’m (slowly) getting my “old self” back. I’m (slowly) starting to work on my hobby-programming, and no longer seem to be limited to just doing my work for my day job.

Then this morning I checked my RSS feed for CPAN Testers Reports. NINE failures, all of them tests that passed just fine on my dev platform and on my Linux server. Tested with multiple Perl versions, etc. Granted, I don’t test other versions/platforms exhaustively, not compared to some people at least. But the things that seem to be failing are tests that have always passed before. Like, suddenly some of the tests that actually create server objects are failing to bind to their given address, saying that it’s already in use. Except that I programmatically select a port by first establishing that it is not in use. I suppose that theoretically it is possible that some other process stepped in between the time I do the port-detection and the time I actually allocate the server object. But like I said, these are tests that were passing just fine before. There are some others, as well, that seem to be consistently problematic for people who are not me. But I don’t get a lot of information from the CPAN Testers reports, just the output from “make test”, really. So until/unless I can replicate those failures, I’m not sure how to address them.

This isn’t really what I pictured, when I finally started getting back in the saddle…

Tags: , ,

And A Heavy Sigh Is Given

February 6th, 2014 | No Comments | Posted in Meta-Posts

I really don’t know which is worse; that it has been over a year since my last blog post, that it has been that long since my last CPAN upload, or that my last blog post was due to my last CPAN upload.

I need to work on this, somehow.

Tags: ,

Perl Module Release: RPC-XML 0.78

February 6th, 2014 | No Comments | Posted in CPAN, Perl, Software, XML
Version: 0.78
Released: Thursday February 6, 2014, 08:00:00 PM -0800

Changes:

  • lib/RPC/XML.pm

A patch to loop detection in smart_encode from Dag-Erling Smørgrav. Some other minor bits.

  • lib/RPC/XML/Procedure.pm

RT #83108: Fixed a spelling error. Some other fixes, too.

  • lib/RPC/XML.pm

RT #86187: Force key-ordering in struct as_string and serialize. Was getting some intermittent bug reports of failures in t/15_serialize.t that amounted to the keys in a fault struct not being in consistent order.

  • lib/RPC/XML.pm
  • t/15_serialize.t

Undo the previous change and fix the test. The previous change didn't feel right, so this rolls it back and fixes the problem at the level of the test, instead.

  • Makefile.PL
  • lib/RPC/XML.pm
  • lib/RPC/XML/Client.pm
  • lib/RPC/XML/Server.pm

Replace direct evals for loading optional modules with Module::Load. Required adding this to Makefile.PL because Module::Load is not core in 5.8.8. Also did some slight doc tweaking.

  • lib/RPC/XML.pm
  • lib/RPC/XML/Client.pm

Merge pull request #5 from alexrj/utf8-encode. Use utf8::encode() instead of utf8::downgrade().

  • lib/RPC/XML.pm
  • lib/RPC/XML/Client.pm
  • lib/RPC/XML/Server.pm

Finish the uft8 encode vs. downgrade change from the previous commit. Changed in places that were overlooked, and adjusted the version number in all three modules.

  • lib/RPC/XML.pm
  • lib/RPC/XML/Parser/XMLLibXML.pm

Merge pull request #6 from dctabuyz/master. Added 'no_blanks' libxml option to skip blank XML::LibXML::Text nodes.

  • lib/RPC/XML/Server.pm

Merge pull request #7 from kvar/master. Initialize $do_compress in RPC::XML::Server between requests.

  • lib/RPC/XML.pm
  • lib/RPC/XML/Parser/XMLLibXML.pm
  • lib/RPC/XML/Server.pm

Bump version numbers on modules changed in github pulls.

Tags: , , ,

Perl Module Release: RPC-XML 0.77

September 3rd, 2012 | No Comments | Posted in CPAN, Perl, Software, XML
Version: 0.77
Released: Monday September 3, 2012, 12:00:00 PM -0700

Changes:

  • t/15_serialize.t

Fix a test failure on Windows.

  • lib/RPC/XML.pm

RT #70408: Fix spelling error in man page, reported by Debian group.

  • t/90_rt54183_sigpipe.t

Fix to handle cases where server creation fails. Now skips the tests rather than dying.

  • lib/RPC/XML/Client.pm

RT #67486: Add port to Host header in client requests.

  • lib/RPC/XML/Server.pm

RT #65341: Added “use” of forgotten library File::Temp. This was causing failure when “message_file_thresh” kicked in.

  • t/10_data.t

RT #78602: Changed 64-bit test from use64bitint to longsize. On some systems (such as OS X), use64bitint can be true even when in 32-bit mode.

  • t/21_xml_libxml.t

Fix from Christian Walde, skip passed test on Windows.

  • lib/RPC/XML/Server.pm
  • t/40_server.t

Checkpoint refactoring and additional tests. Work is not complete here, but the Net::Server changes demand immediate attention

  • t/20_xml_parser.t

RT #72780: Check for a possible parser failure. One instance of XML::Parser failing to parse the extern entities test. Cannot reproduce, so wrap it in a “skip” block for now.

  • lib/RPC/XML/Procedure.pm
  • t/30_method.t

RT #71452: Correct handling of dateTime parameters. Existing code in lib/RPC/XML/Procedure.pm did not properly handle parameters of the dateTime.iso8601 type. Also, there were no tests for these.

  • MANIFEST
  • t/30_method.t (deleted)
  • t/30_proceudre.t (added)

Renamed t/30_method.t to t/30_procedure.t.

  • lib/RPC/XML/Server.pm

RT #77992: Make RPC::XML::Server work with Net::Server again, after the API changes of Net::Server 2.x.

Tags: , , ,

Perl Module Release: Image-Size 3.231

May 23rd, 2012 | No Comments | Posted in CPAN, Perl, Software
Version: 3.231
Released: Wednesday May 23, 2012, 11:30:00 PM -0700

Changes:

  • lib/Image/Size.pm

Small change to swfmxsize for short-buffer issues.

  • lib/Image/Size.pm

Multiple changes in this commit: Abandoned AutoLoader; moved POD to the end of the file now that all the per-format subroutines are part of the file-proper; fixed cases where lack of ImageMagick would cause the error message to be overwritten.

  • t/all.t
  • t/kazeburo-bar.jpg (added)

New test (and image) for JPG tag-offset issue.

  • lib/Image/Size.pm

Applied patch from github user kazeburo: bug fix for some jpeg files

  • imgsize

Fixed so that default output now catches errors. Previously, you would get an “undefined” warning if there was an error in parsing/sizing the input file.

  • lib/Image/Size.pm

Turns out the 4 bounding-box ints for EMF are signed.

Tags: , , ,

Perl Module Release: Test-AgainstSchema 0.100

January 17th, 2012 | 2 Comments | Posted in CPAN, Perl, Software
Version: 0.100
Released: Monday January 16, 2012, 02:00:00 PM -0800

Changes:

  • lib/Test/AgainstSchema.pm (added)
  • lib/Test/AgainstSchema/XML.pm (added)
  • t/10_xml/05_xml_parsing.t (added)
  • t/10_xml/10_sgmldtd.t (added)
  • t/10_xml/20_xmlschema.t (added)
  • t/10_xml/30_relaxng.t (added)
  • t/10_xml/basic_tests.pl (added)
  • t/10_xml/simple-bad.dtd (added)
  • t/10_xml/simple-bad.rng (added)
  • t/10_xml/simple-bad.xsd (added)
  • t/10_xml/simple.dtd (added)
  • t/10_xml/simple.rng (added)
  • t/10_xml/simple.xsd (added)
  • t/10_xml/xml-simple-bad.xml (added)
  • t/10_xml/xml-simple.xml (added)
  • t/90_integration/10_with_xml.t (added)
  • t/90_integration/90_empty_subclass.t (added)
  • t/90_integration/MyTestAgainstSchema.pm (added)
  • t/util.pl (added)
  • xt/00_load.t (added)
  • xt/01_pod.t (added)
  • xt/02_pod_coverage.t (added)
  • xt/03_meta.t (added)
  • xt/04_minimumversion.t (added)
  • xt/05_critic.t (added)

Initial commit of code, tests and test files.

Tags: , , , ,

Looking Back, Looking Forward: 2011 & 2012

January 1st, 2012 | No Comments | Posted in Meta-Posts, Perl

So here we are, starting yet another new year. Seems like I was just here, but when I look back at my archives I see that my only post in January of 2011 was for the 0.74 release of RPC-XML. So I wasn’t even as on-the-ball a year ago as I thought I was. And over the years I’ve gotten out of the habit of making elaborate resolutions for each new year. So this time around, I’m going to reflect a bit on high points of 2011, and ponder a bit about what I hope to do in 2012…

2011 was overall a pretty good year. It was my first full year at NetApp, after having done more job-hopping than I particularly liked to do in the years from 2006 to 2010. NetApp has been a really good place to be, both stable and challenging. I have good co-workers, and good management. Some high points of 2011 included:

  • I snagged some kudos in the form of winning an internal friendly competition at NetApp for my work with Perl::Critic (a web interface similar to the one at perlcritic.com, but with some additional features, bells and whistles specific to NetApp’s needs).
  • I was once again supported by my management to attend OSCON this past summer, which not only meant learning many new bits of tech but also meant seeing many friends I only see at the con.
  • I made the leap from being a strictly Linux guy, to obtaining my first Apple Macbook (Pro). It has been (and continues to be) a quirky learning curve, but I’m happy with it. I don’t know if I’m more productive with it than I was with my Linux laptop, but I’m at least as productive, so (in theory) it can only get better as I get more accustomed to it.
  • After being familiar with the module for some time, I finally made the leap of starting to use Devel::Cover on some of my code, which lead to some vast improvements in my testing suites (as well as flushing out numerous bugs along the way).
  • I completed the online Introduction to Artificial Intelligence class that Stanford offered this fall. It was an experiment in online learning that the Stanford engineering school was conducting, one of three courses offered during that time. As an experiment, it must have gone well as they are offering ten courses this coming term: CS 101, Machine Learning, Software as a Service, Human-Computer Interaction, Natural Language Processing, Game Theory, Probabilistic Graphical Models, Cryptography, Design and Analysis of Algorithms I, and Computer Security. I plan on following up the AI class with the ML class. I’d love to take about half of them, but I have to be realistic about the free time I have.

So that was 2011. What do I plan for 2012?

  • Release my CPAN modules more frequently, which means working on them more than I currently do. Over on his blog, Mark Fowler has resolved to release a distribution to CPAN once a week, every week, throughout 2012. I won’t be doing that. But I can take from his thoughts on the matter some good direction and ideas, and I can apply those to how (and when) I choose to release.
  • Do OSCON again. This may be tricky, as there has been a managements change in my organization. I don’t know yet if the new director of my org will feel the same way about education and training as the previous person did.
  • Related (slightly) to the first point: Release at least two new CPAN distributions. I have the specific ones in mind; one is a complete re-write/re-organization of an existing distro of mine, the other is completely new.
  • Finally get around to learning Clojure. I’ve been toying with it and tinkering with it to a very light degree, but this year I will buckle down and actually work my way through the entirety of one or more books on the language. Most likely starting with The Joy of Clojure.
  • Oh, and of course write more often here. The AI class effectively killed my blogging for the last part of 2011, but judging from people’s reviews and feedback on the ML class I don’t expect it to so thoroughly take over my life as the AI class did. So even if some weeks I only manage to eke out a “module Monday” post, I hope to at least accomplish that much.

Not necessarily lofty goals there, I will admit. But I have also resolved to spend more time on my non-computing hobby, so I am not going to set myself up with resolution expectations that require me to practically sleep with the laptop to accomplish them. I’d rather set my expectations at a challenging-yet-reasonable level, and actually achieve them.

Here’s to the new year…

Tags: ,

Perl Module Monday: HTTP::Tiny

October 17th, 2011 | 2 Comments | Posted in CPAN, HTTP, Perl

I’m still deep in the Stanford AI class, so this will be a light-weight posting. And since it’s going to be light-weight anyway, I’ll cover a module in the *::Tiny namespace: HTTP::Tiny.

HTTP::Tiny is a simple HTTP/1.1 client library with plenty of options. It handles HTTPS (if you have IO::Socket::SSL available) as well as HTTP requests, and does all the basic HTTP verbs. As is the case with most *::Tiny modules, the goal is to do as much as one can, without the overhead or dependency chain of a larger module. In this case HTTP::Tiny stands as a replacement for LWP::UserAgent, for those cases when you don’t need the full functionality that LWP provides.

The main methods of HTTP::Tiny that you’re likely to utilize (besides the constructor) are request() and get() (which is just a front-end to request(), with the ‘method’ argument set to GET). There is also a method called mirror(), which is handy for making a local copy of a web resource on your filesystem. mirror() even sets an “If-Modified-Since” header on the request, if the file already exists. A nice touch to have added! The request() method allows for a very useful range of options, that make it easy to pass specific headers, use call-back subroutines for either (or both) of the request body or the processing of the response, and provide trailer headers for chunked transfer-encoding. One thing I find curious, though, is why the author provides a short-hand method for the GET request, but not for the other verbs. Since all are called using the same semantics, it seems to me like it would have made as much sense to provide head(), put(), etc.

Still, it’s a nice little approach to HTTP communication, that doesn’t require as much setting-up of resources as LWP generally does. It doesn’t have the flexibility that LWP does, either, but sometimes you just don’t need that. You just need to get going in a few lines:

use HTTP::Tiny;

my $http = HTTP::Tiny->new();

for my $url (@ARGV)
{
    (my $file = $url) =~ s{^.*/}{};
    if (! $file)
    {
        warn "Skipping $url (no file component)\n";
        next;
    }
    $http->mirror($url, $file);
}

The above just mirrors all the URLs passed in via @ARGV, using the last file element of the URL as the file name to save to. It doesn’t have the progress-bar and summary that LWP’s “lwp-download” has, but it gets the job done.

So have a look, this could be a useful addition to your toolkit, sitting beside LWP and handling some of the simpler tasks for it.

Tags: , , ,

No PMM This Week

October 10th, 2011 | No Comments | Posted in Perl

Alas, I didn’t get this done earlier in the day, and now I need to spend the remainder of my evening working on the first units in the Stanford on-line AI class. These materials were only just posted, but I’m already behind the curve because I’ve not reviewed all the pre-class material. Hopefully I’ll be able to get a PMM candidate picked out for next week and get the post written before it gets this late in the day.

Tags: ,