| Subcribe via RSS

AutoLoader Considered Harmful?

July 30th, 2011 Posted in Perl

Lately, I’ve been doing a lot of work on some of my modules using the fine Devel::Cover package. And for about two years now, I’ve also been using Perl::Critic almost religiously (to the point where I’ve introduced it around $DAY_JOB, and even built them a snazzy web interface for it). But these two share a common failing: both fall down when they encounter a module that uses some form of external-file-based auto-loading.

First, let me explain what I’m referring to, for those who might not be familiar with auto-loading. Perl has this nifty feature called auto-loading, in which a function that doesn’t exist when it is called might have the chance to spring into life dynamically. Perl will look for and invoke a special subroutine called AUTOLOAD in your current namespace, giving it the name of the subroutine that was just requested in the special variable $AUTOLOAD (and passing that call’s args list in @_). You can get the routine by any means that Perl allows: you can create it on the fly, you can on-demand load a module and add it to your @ISA search path, etc. One of the most clever uses I saw early on in Perl 5’s life, was Lincoln Stein’s CGI module. In it, he used auto-loading to define all the HTML shortcut routines from a template, eval‘ing them on the fly as they were each first called.

In short, this amounts to Perl’s version of C’s dynamic loading, deferring the loading and compilation of code (well, in the C world you were just deferring the loading and linking, it was already compiled) until you needed it. Into to this we introduce the core modules AutoLoader and AutoSplit. AutoLoader provides for you an AUTOLOAD method that uses the name of the requested routine and the package it is in to look for a file by that name under a directory hierarchy within your @INC path. To compliment this, AutoSplit is used by the build/install process (be it ExtUtils::MakeMaker or Module::Build, etc.) to look at a module, determine if it uses AutoLoader, and splits out the routines that were defined as dynamically-loading into these per-name files. In fact, I got my break into the world of writing about Perl by writing an article explaining these two modules for the Perl Journal (alas, I cannot find a link to the article online). I also made my first contributions to Perl’s core by fleshing out the docs for these two, as I had been using them at my then-job to make a 20,000-line module more manageable.

But! I’m clearly not hear to sing the praises of AutoLoader, or I would have titled this post something else entirely. I don’t necessarily come here to bury Caesar in lieu of praising him, but introducing AutoLoader (or SelfLoader, for that matter) gums up the works when you use either of Devel::Cover or Perl::Critic. And while Devel::Cover might be able to fix this (I’ve filed a bug, but I don’t know the internals of D::C so I don’t know if it is addressable), Perl::Critic definitely won’t be able to.

See, AutoLoader/AutoSplit (and SelfLoader, though it uses a slightly different model) work by having you put the code you want to delay loading after the __END__ token in your module (__DATA__ in the case of SelfLoader). AutoSplit can then find it and split it out into individual files, and as the compiler stops compiling once it hits that token nothing at that point onward contributes to the start-up compilation phase. But while this successfully hides the code from the compiler, it also very successfully hides the code from Perl::Critic and Devel::Cover as well.

P::C is built on the PPI package by Adam Kennedy, and in terms of a Perl module as a document anything after __END__ is simply not code and not going to be processed as such. So your code that uses AutoLoader is not being fully analyzed by P::C. I had released a version of RPC::XML that I thought was critic-clean, without having thought about this caveat. When I later went and commented-out the “use AutoLoader” and “__END__” lines, I found a whole new set of violations to clear up.

Likewise, I had a similar problem with Devel::Cover. Only slightly worse, because I assumed that they had already dealt with this problem. The code gets loaded just like any other Perl module, so I assumed that when it loaded it then got instrumented. And the code uses “#line” directives to associate the code with the correct line in the original *.pm file, so I thought that they would be able to “translate” the line numbers of coverage statistics back to the originating file. But alas, no— code after __END__ is just as hidden from D::C as it is from P::C.

Which leaves me wondering whether AutoLoader (and __END__-based auto-loading in general) might not have run its course of usefulness. I’m left wondering what sort of hoops the authors of syntax-highlighting for Emacs had to jump through, to determine that content after __END__ was actually code rather than data, and to highlight it as such (vim doesn’t, it leaves it in the same font-face as data). Editors aside, here are two extremely useful development tools for Perl programmers (remember, I’m practically religious about P::C, spreading the gospel to my workplace and anywhere else I can), and both are hampered by the use of auto-loading. Given the leaps in memory and CPU over the last 15 years or so since I wrote that first article, do we really need AutoLoader anymore? I mean, I’m not saying do away with auto-loading itself, it has a truly useful place in Perl. I’m just not so sure whether we need AutoLoader (or SelfLoader) like we used to.

4 Responses to “AutoLoader Considered Harmful?”

  1. Gabor SzaboNo Gravatar Says:

    I have other issues with the AutoSplit though it might just show my lack of knowledge. When you are running the already split up code and you encounter a bug it is quite hard to debug and fix the code as that requires mapping it back to the source.

    And I already worked at companies that lost the original file.

    So I don’t recommend any of my clients to use that and when I encounter a piece of code I’ll try to get rid of it while refactoring.

    That does not mean that dynamic loading is bad. I just would keep modules intact. So if you’d like to load some code conditionally, go ahead do it. But put it in its own module. By design. That works well. Splitting up code is IMHO not.

    BTW I love both Devel::Cover and Perl::Critic

  2. Aristotle PagaltzisNo Gravatar Says:

    I advise to get rid of any uses. The POD should be updated to advise against its use too.

    I believe it was not long ago that the last use of AutoLoader in another core module was removed, although I don’t recall the details. There are still some uses in dual-life modules.

  3. Leon TimmermansNo Gravatar Says:

    90% of the time AutoLoader is a premature optimization, and in the rare case it isn’t that probably points out a badly structured module. It’s a source of subtle bugs that should be avoided.

  4. zgrimNo Gravatar Says:

    AutoSplit is evil :) just recently I encountered a weird bug where I was running some forked code and if it needed to chroot, Storable wouldn’t find it’s needed .al files for freeze/thaw, which was solved via a dummy load of freeze/thaw before the fork(). Very unpleasant debugging session though :)

Leave a Reply