| Subcribe via RSS

Well. That Was… Educational.

August 29th, 2017 | No Comments | Posted in CPAN, Perl

I got a great lesson recently in the dangers of making assumptions about other people’s code. One could almost call it an object lesson. (A really bad pun that will become more evident as you read further.)

I’ve had a small flurry of coding activity recently, fixing some issues in YASF and writing a new module, one that I’m calling “JSON-WithComments“. (Note to self: need to update my avatar pic on that site.) The new module is actually very simple; the manual page is longer than the code itself. It sub-classes the JSON module in order to allow you to have comments (either Perl/shell-style, or JavaScript-style) in JSON data (presumably from a file that is being used as the configuration for a larger tool of some sort). I came up with the need for this based on a project at my day job, but decided to flesh it out and develop it at home so that I could share it on CPAN.

(A little while into the project, I discovered that the JSON module will actually support Perl-style comments directly, via the relaxed method. I decided to continue with the development of the module anyway, as I was already supporting multiple comment styles. But if you only need Perl-style comments, you can manage with just the JSON module.)

Anyway, so I spent lots of time studying the main JSON.pm file, determined that the objects were hash-reference-based, and wrote my code to extend the objects with a new property to track which comment-style to use. I wrote some (admittedly sparse) tests and some docs, and Sunday night I uploaded version 0.001 to CPAN.

Monday morning, I woke up to 70+ failure notifications from the automated testing corps.

See, JSON is really just a front-end that chooses between JSON::PP and JSON::XS back-ends. But no big deal, I have both of those installed, so my code must be working with both back-ends, right? Right?

Wrong. I did not have JSON-XS installed, which (being much faster) is the preferred back-end. And this back-end does not, in fact, implement the underlying object as a blessed hash reference. It uses a blessed scalar. So when my code cheerfully tried to store and retrieve a hash key from the reference, it didn’t just not work, it died immediately. Taking the test suite with it.

Once I finally figured out what the cause was (by which I mean, I figured out that I didn’t have JSON-XS installed after all and once installed it broke on my machine too), a fix was pretty easy to implement and last night I uploaded version 0.002. But getting a 70+ flurry of failures on my initial upload was a lesson I’ll not soon forget.

So be cautious about making assumptions when sub-classing someone else’s class. Lest you, too, receive the same “object lesson” that I just did…

Tags: , , , ,

Change on the (Distant?) Horizon

January 8th, 2017 | No Comments | Posted in Meta-Posts

Well, I’m still not sure if this means I’m returning to regular blogging or not. Perhaps only semi-regular activity is the best I’ll be able to manage. But for now, if I can manage to at least write once a week it should keep the Perl Ironman Challenge bot happy.

I’m going to be changing some things around here, if I do in fact start spending time here on a regular basis again. The first thing I’m going to work on changing is the theme. I actually like the current theme, as I’m a fairly minimalist type when it comes to web design. But it isn’t set up to handle mobile devices very well, and that’s just a reality that every blog needs to embrace if it hasn’t already (and I haven’t). In a more perfect universe, I’d have both the time and the know-how to either hack this theme for better mobile support, or just roll my own theme. But I don’t have that know-how (or time), and to be honest I’m not in a hurry to learn WP theme development. But if you read this and prefer mobile devices, rest assured that making the site mobile-friendly is the top priority. Well, trying to write consistently is technically the top priority, but this runs a very close second.

Next, I’m going to do something about the near-astronomical number of spam accounts that have registered here (and continue to do so daily). I direct the mail notifications for this site to my Gmail address, and I do not exaggerate when I say that I average 100 or more new users per day. Of course, they’re (almost) all bots. I have a plugin in place that protects the site against login attacks, but I don’t have anything in place really that tries to prevent bots from signing up. That will have to change; I’m kind of tired of having over 100 new emails from WordPress daily… it makes accessing my Gmail account over IMAP a pain in the ass. (Yeah, I can filter them to a folder, but that doesn’t really solve the problem, it only masks it.)

(A side effect of the registration work is that it might impact legitimate registered users. I would prefer to purge the database of the fake users, but I’m not sure if I’ll be able to without putting real accounts at risk in the process. If I do have to just blow away the whole database, I’ll warn users here on the blog at least a month or so in advance.)

Following that (or perhaps in concert with that effort) I’m going to re-visit the other plugins that I’m using here, and try to trim away those that aren’t really contributing anything useful and possibly add a few new ones that do. While I’m not much of a cheerleader for PHP, I do have a great appreciate for the flexibility and ease-of-use that WordPress provides. It has a hell of an ecosystem behind it, and I need to be taking more advantage of that.

In terms of content (presuming I do start writing more regularly), expect to see less of a focus on Perl. I’m not giving up on it by any measure (Perl has been very good to me over the years), but I don’t want to get typecast as only a Perl programmer. During the first half of last year, I started really seriously working on learning Clojure, which is just a really fun language to program in. (Well, assuming you like functional programming and aren’t irrationally allergic to Lisp.) So far I’ve mostly just been working through problems on Project Euler. I can’t seem to link directly to my “progress” page, but my user-name on PE is also rjray. So far, I’ve solved 87 problems and earned 6 of the problem-solving awards (haven’t gotten any posts in the forum to be made permanent yet, so no forum-post awards). It’s been quite interesting, but by it’s very definition PE is all about the math, so this hasn’t really exposed me to the full range of of Clojure’s capabilities yet. I hope this year to move beyond just PE and do some other things with Clojure that will push me to fill in the missing pieces that I haven’t yet used.

In that same vein, I am hoping to spend a little time dusting off my long-dormant C/C++ skills. I now work for NVIDIA, who is doing just amazing things with deep learning, machine learning, etc. using their GPU technology and the CUDA platform. And CUDA is pretty much all about C++ (well, there is also support in Fortran and Python, to be fair). My C++ is so rusty, that when I last used C++ they (the standards committee) were only just considering templates as a new feature, and hadn’t quite agreed on a standard yet. My role at NVIDIA doesn’t directly involve C++ (or else I’d be in a pretty bad spot in terms of being able to do my job), but CUDA and GPU programming look fascinating to me, so I’m hoping I can at least dip my toe in the water at some point in 2017.

And lastly (for now), you may even see some Python content here. While my NVIDIA role is primarily Perl, there is also a fair amount of Python floating around, and I’ve been slowly acquainting myself with it for the last couple of months in order to understand, enhance and fix things. But don’t expect to see me become a Python convert; most of what I have seen has me scratching my head at how the Python community can really feel so smugly superior to the Perl community. Most specifically, I’ve been listening to Pythonistas brag for YEARS about how much better the object model in Python is when compared to Perl’s. Having gotten a taste of Python’s object model, I can only assume I’m missing something big. Because I’m pretty far from impressed so far.

But that’s a rant for another time.

Tags: , , ,

I Made a New Thing in Perl

January 1st, 2017 | 2 Comments | Posted in CPAN, Perl, Software

(For now, I’m going to casually ignore the fact that it has been literally two years since my last blog posting. If this post heralds a return to regular blogging, I may address that in a future post.)

I made a new thing in Perl: YASF, or Yet Another String Formatter. I’m rather happy with it, even though I suspect that no one else is using it yet. I’m happy mostly because it’s the first new module idea that I’ve been able to actually turn into real code, in several years. The “why” of this is somewhat complicated, but can best be summed up in six words: Deep depression, self doubt, strong medications.

So what is it, and what does it do?

It’s a lightweight string-formatter module inspired by the format method of strings in Python, with a little syntactic sugar based on Python’s % operator for string formatting. It scratches a particular itch I’ve had for some time now: why do I keep writing (basically) the same hash-based search-and-replace regular expressions over and over? By which I mean something like:

my $pattern =
    'This is a string with one {key1} followed by another {key2}.';
my $hash = { key1 => 'value', key2 => 'different value' };

(my $string = $pattern) =~ s/{(\w+)}/$hash->{$1}/ge;

Now, this is a fairly convoluted example, but it gets at the point: this is a very common pattern. And it is usually not this straightforward; you probably have a series of value-sets that you would want to substitute into $pattern in order to get a new $string value, otherwise you would just use the hash values directly when declaring $pattern. Maybe your pattern is a global value made read-only by Readonly or Const::Fast. There are (I imagine) more reasons for wanting to do this than I can think of by myself. But I had already run into enough reasons on my own that I wanted to do something about it.

Coincidentally, around the most-recent time that I found myself writing something for this pattern yet again I was also learning Python for a task I was given at my day-job. I was looking at an existing code base and tracking down the elements and constructs that I didn’t immediately recognize in O’Reilly & Associates’ Learning Python. In the process, I stumbled upon two elements of Python that I thought were rather nifty: the string % operator, and the format method. The operator was the earlier of the two features, and is essentially an implementation of printf that takes the format pattern on the left of the operator and a list of values on the right, and puts them in according to printf-style format specs in the pattern.

But the format method, which came later, it’s a much different beast. It can do quite a bit more than the operator. Place-holders in the pattern can be named keys pointing into a dictionary (hash), or numbered indices into an array. And the range of formatting that can be applied to each interpolated value is quite a bit more extensive than what printf offers. Now, while I realize that the main impetus for Python’s string-formatting is that they don’t do in-string interpolation of variables the way Perl does, I still saw this as a possibly useful micro-templating approach. Imagine if the example above had looked more like this:

my $pattern =
    'This is a string with one {key1} followed by another {key2}.';
my $hash = { key1 => 'value', key2 => 'different value' };

my $string = $pattern % $hash;

OK, that’s still pretty contrived. And I’m not too good these days at making up examples that look like real-world problems. But consider this:

use URI;
use Const::Fast;

const $SPECBLOCK =>
    "Site: {name}\nProtocol: {url.scheme}\nPort: {url.port}\n" .
    "Host: {url.host}\n";

while (($name, $url) = splice @data, 0, 2) {
    my $block = $SPECBLOCK % { name => $name,
                               url  => URI->new($url) };
    print "$block\n";

That has a little more meat to it, yeah? And the use of an object as a value that can have methods called on it, that’s something that the Python format does, so surely a Perl copy should do so as well.

But there’s a catch: you can’t really do the above with Perl strings, because they aren’t objects derived from a class that you can monkey-patch. So I had to come up with a class, instead, to add the functionality to:

use URI;
use Const::Fast;
use YASF;

const $SPECBLOCK => YASF->new(
    "Site: {name}\nProtocol: {url.scheme}\nPort: {url.port}\n" .
    "Host: {url.host}\n"

while (($name, $url) = splice @data, 0, 2) {
    my $block = $SPECBLOCK % { name => $name,
                               url  => URI->new($url) };
    print "$block\n";

That will work. But wait, there’s more! And at no extra charge!

Because the format patterns are now objects, I can do things with them, to them, and on them. And the first thing I did, was introduce the concept of binding a data structure to the object to use as a default source if you had places where explicitly providing a binding wasn’t feasible. This opened up the ability to overload more operators, most notably the stringification operator:

use URI;
use Const::Fast;
use YASF;

const $SPECBLOCK => YASF->new(
    "Site: {name}\nProtocol: {url.scheme}\nPort: {url.port}\n" .
    "Host: {url.host}\n"

my %binding;

while (($name, $url) = splice @data, 0, 2) {
    @binding{qw(name url)} = ($name, URI->new($url));
    print "$SPECBLOCK\n";

Again, even with more meat that is still a fairly contrived example, and a line or two longer than the previous version. But consider the following snippet:

use DBI;
use YASF;

# Set up database connection, then declare $sth as a fetch
# statement

# Declare $str as a YASF object with the pattern you want, with
# each field from the fetch statement available as a value in the
# pattern.

my %row;
$sth->bind_columns( \( @row{ @{$sth->{NAME_lc} } } ));
while ($sth->fetch) {
    print "$str\n";

I based that off of the example in the DBI manual page of binding database columns directly to values inside a hash. I haven’t tested it yet, but I plan to add some tests to the distro soon, runnable when the user installing the module has DBD::SQLite available.

Anyway, there’s a lot more to the module, even though it is currently only at version 0.004. If this seems interesting, feel free to check it out on CPAN and/or GitHub. There are a few current caveats:

  1. Version 0.004 does not have any of the Python-ish formatting support, it only does expression substitution. If you put a format in as with Python, it will be quietly ignored for now. I am working on formatting right now, and hope to have a version 0.005 out before long.
  2. I still consider this alpha-quality software, and as such I may yet change some elements of the interface. In particular…
  3. …I’m debating whether to take the separate bind and binding methods and roll them into a single method. On the one hand, “bind” is more of a verb and thus a little more meaningful for the action of assigning bindings. On the other hand, most users (I suspect) would expect the same method that makes the bindings to return the current bindings. So this may change, especially once (if) other people start using this and give me some feedback.
  4. Lastly, I’m not too fond of the name “YASF”. I’m crap at naming things, and wrestled with this one for well over a month before a friend suggested “Yet Another String Formatter”. I would definitely entertain better suggestions, with the limitation that they need to be meaningful and (relatively) short. A long name with 2 or 3 parts to the namespace belies the general simplicity that I am aiming for.

I hope that someone (besides me) finds this useful.

(Afterthought: It feels good to write in this blog again. I do hope I can continue to do so in 2017…)

Tags: , , , ,

Some Observations on the eBay for iPad 3.4.0 App

December 31st, 2014 | 2 Comments | Posted in Uncategorized

(This is a copy of an “issue” I posted on the eBay Mobile forum on getsatisfaction.com. The original is here. As it happens, some of the more “divisive” posts and comments in the forum are being deleted, so I am making a copy here in case they choose to delete it there.)


(TL;DR version: the 3.4.0 version of the app makes some very dubious interface/experience decisions, and you are quickly losing your power-users)

Dear Ginger et al,

Since the release of the 3.4.0 version of the eBay iPad app, there has been a tremendous amount of discussion, almost all negative, here on this forum. I would like to present some of my own concerns and observations in a (hopefully) more even tone.

First, a bit of background: I’m not quite sure when I first joined eBay as a user, but it’s been in the neighborhood of 20 years. My eBay ID is “randy.j.ray”, and while I haven’t sold a lot of stuff I do have a pretty good feedback rating. So I’ve been using eBay since long before mobile devices were even a thing. My professional background is in software engineering with an emphasis on web-based applications. I’ve been developing web content and web applications since 1993. I do not (and cannot) claim to be an expert in UI/UX issues (user interface/user experience issues), but I have my fair share of experience in creating interfaces and experiences (as well as my fair share of bad design decisions). So I’m not just a typical dissatisfied user of the app, I am looking at it with a critical, professional eye.

The redesign of the app is a failure.

Your own chief product officer, R.J. Pittman, said in an article on re/code that the goal was to make the app look more like Pinterest and less like Amazon. His actual words were, “I’m happy to openly use the Pinterest model as an example.” I don’t understand this– Pinterest is not really a commerce site, whereas Amazon is one of the leading commerce sites on the web. I’m not saying that you should emulate Amazon, nor am I saying that there are not people using Pinterest for commerce purposes. But it seems counter-intuitive to choose something like Pinterest as a model for an app that is intended to be focused on buying and selling things.

For years, I used eBay through the website interface because that was really all there was, save for some platform-specific applications (usually Windows-based, and I don’t run Windows). During this time, my usage pattern for eBay was generally:

  • I need something
  • A Google search leads me to eBay
  • I use eBay’s searching capabilities to try finding the best price
  • I buy the item

Note that I don’t list “browse generally for anything that might interest me” in that list. This was not very easy to do on the website, so I rarely (if ever) did it.

Then a few years ago I bought an iPad, and one of the apps that I happened across for it was the eBay app. I installed it out of curiosity and fired it up. It was amazing! Suddenly, searching and browsing were significantly easier than they had ever been on the website. The endless scrolling of items in a category or search was (and is) highly addictive. Being able to easily jump to a seller’s other items and have the same scrolling experience led to a lot of impulse purchases. (Hey! This seller consolidates shipping! What else can I buy?) But what really made the app for me, was the ability to place “shortcuts” on the primary screen for various searches and categories that I regularly used/browsed. From the feedback I’ve seen on this forum, this feature was the real “killer feature” for your app, because it seems to be the feature that people are most upset over losing.

Well, that and the introduction of “curated content” into the app’s home-screen. Your impression of content to promote has no relation (that I can reason) to my shopping preferences. There are fashions, jewelry, etc. being offered to me when I have never even browsed any of those categories, let alone spent the kind of money that I have on my preferred categories. When I first ran the new app, I thought it must have lost my login credentials, when I saw all that unrelated stuff being put in front of me. But nothing I did could get rid of it. It wasn’t a bug, it was a feature.

What was worse, was the lack of my saved searches on said home-screen. Not only was I being bombarded with useless (to me) content, the content I did want was no longer easily available. I say “easily”, because as you have pointed out it is still reachable. Only now it takes 3 clicks/taps to reach a saved search, instead of just 1. Huh? At what point did someone decide that 3x the effort to perform a common function was an improvement?

Someone (I forget who) said in a different online article that the app was meant to be released after Christmas, but it had tested “so well” with beta users that it was pushed out early. Who were these beta users? Were they regular users of the 3.3.X line of the eBay app? Or were they people with little to no previous experience with eBay apps, who were judging the app without any past context? I can see how the app might look fine to someone who had not used a previous version, but did you engage any of your “power users” in the beta cycle? Because you are losing these users on a daily basis, and I’m not sure if you’re even aware of it.

Which brings up my last complaint: the responsiveness of your staff who are responsible to replying to users specifically in this forum. There has been a real lack of responsiveness here, and when someone has responded it often comes across as condescending. For example, there have been several cases where someone would say (and I am paraphrasing here), “I don’t like that the saved searches have been moved,” and the response (again, paraphrasing) would be, “Oh, you can’t find saved searches? Well just tap here, then here, then…” The user never said they couldn’t find the feature, they were unhappy with having to tap three times for something that was previously directly available on the home screen. But the responses we’ve been getting from your representatives seem to not really be looking at the issue at hand, but rather assuming that we just don’t know how to operate the app. To be frank, it’s insulting.

I’ll stop at this point, because I feel I’ve addressed all the salient points. It seems to me that you made a wholesale change to the fundamental usage-pattern of your app, but without actually considering the users themselves. And in the process, you have angered and alienated a large number of them. Perhaps not enough of them are leaving for you to notice. Perhaps there have been enough new users, people who received iPads for Christmas and have never seen the previous version of the app for comparison, that you consider the lost users an acceptable rate of churn. I don’t know. For my part, I’ve used iTunes to restore the 3.3.1 version of the app and am once again happily buying more things than I probably should be. But if you should fully deprecate the 3.3.X line (by making a non-portable API change, etc.) and force me into the new model of app, I’ll simply quit spending money on eBay (as so many people have already chosen to do). There is very little that I find on eBay that I cannot find elsewhere; it’s just easier to find on eBay. But if you take away that easiness, then why shouldn’t I just go straight to other storefronts?

Please consider what I have said here, at least think about it. Your users are, or at least have been in the past, fiercely loyal to you. That sort of loyalty should be valued, not squandered.

Randy J. Ray
Sunnyvale, CA

Tags: ,

Atom Editor Invites

March 20th, 2014 | No Comments | Posted in GitHub, Software

I have two invites for the beta of the Atom editor under development by GitHub. I haven’t played with it enough yet to know whether it is spiffy, or if so how spiffy it indeed is.

Two things to know if you are interested in trying it out:

  • It currently is only available for MacOS. Appears to be compatible with Mountain Lion and Mavericks only, as well.
  • You will have to register via your GitHub account, assuming you have one. Then again, if you are considering test-driving an editor like this then you probably have a GH account…

If you are interested, email me at my primary address (rjray <at> blackperl.com). I will send invites to the first two respondents, using the email address you email from. Do not comment on this post for an invite, as the email notification goes to a different account that I do not check as often.

Tags: ,

Some Modules I’m Looking At Currently

March 16th, 2014 | 1 Comment | Posted in CPAN, Perl

Now that I’m writing (a little) more reliably in this blog, I hope to soon revive my “Module Monday” series. At the time I started doing it, it was fairly easy to come up with new modules to review, as I was the owner/operator of a CPAN-oriented Twitter-bot called “@cpan_linked“. But then Twitter switched over to OAuth-only and I haven’t had the time and energy at the same time, in order to update the bot.

But I still frequently look at new and interesting modules, always looking for new tools with which to fill my Perl toolbox. Here are some that I am currently reading up on and/or playing with:


I learned of this module from this blog post, and am both intrigued and frustrated. Intrigued, because I’ve always felt that changes-files needed to be electronically parseable so that the information could be re-purposed to other uses and formats. But also frustrated, because I wrote something similar to this a while back (App::Changelog2x, which is based on an XML format I designed), which as far as I can tell only I have ever used.

Mind you, I’ve known that ChangeLogML has drawbacks from the very start– for one thing, authoring in XML is a HUGE pain in the ass. I’ve only just recently gotten my emacs set-up to handle ChangeLogML very well, and for that I had to first convert the XML Schema to RelaxNG, then that to the compact format, then hand-edit the resulting RNC file to get around things that hadn’t converted correctly (though one or two of them may have been problems in the original XML Schema, to be fair). Even still, with all that, I’m still editing raw XML. I just have syntax highlighting and validation to help me find errors. And a lot of people just don’t like dealing with XML, let alone editing it by hand.

ChangeLogML keeps more information than the CPAN::Changes::Spec specification does, but mine doesn’t have the concept of groupings, something I may look at trying to add if I ever revisit ChangeLogML and try to make it more useable.

One thing I’ll look at either way, is transforming ChangeLogML to a conforming “Changes” file for my modules. That should be a lot easier than convincing people to hand-edit XML on a regular basis.


Being easily distracted and fascinated by blinking lights and colors, this tool just amuses me greatly. That is is also incredibly useful is a great bonus.

It’s a module from Damian Conway, and I learned about it recently when he included it in a talk he gave to the London Perl Mongers group. I couldn’t go (the commute from California to London being prohibitive and all), but Damian was helpful in posting a follow-up message to the mailing list with a list of the modules he had discussed. I had already taken some interest based on the “Regexp” in the name, as I’m not the best wielder of regular expressions in the Perl world, so I often need debugger-ish tools to fall back on. But just to make it even better, in response to a comment by someone else, Damian followed up his list of modules with a YouTube link to a demonstration of the module.

Watch the video, and I’m sure you’ll be as interested in this module as I am. Note that this dates to 2012, and there have been several releases and new features since then.

Test::Trap and Test::Effects

I’m grouping these together as one provides a foundation that the second one builds upon.

Test::Trap, by Eirik Berg Hanssen, is an interesting approach to testing code in terms of blocks. You pass “trap” the block to execute, and it handles the eval mechanics for you, with the results and any error information wrapped in an object (called “$trap” by default). There’s a great deal of flexibility and configurability here, as well.

Test::Effects is another module from Damian, one that builds on Test::Trap with an interface that lets you test all the effects of your code-block in one go. Where the former module would have you pass the block to trap, then choose what to test against using $trap, this module gives you a testing routine called “effects_ok” that looks at everything, or alternately just those things you want looked at. It is also extremely configurable, including some options that may make my long-neglected Test::Without module obsolete.


Lastly (for this listing, at least), is the well-known Mojolicious package. I’m kind of late to the game on this one, as it has been around quite a while. A co-worker of mine has used it some, and convinced me to give it a look for some web-services applications I’ve been tinkering with. What interests me the most about it is the built-in support for WebSockets, something Apache doesn’t have (and doesn’t seem to be planning for any time soon). I’m currently looking at how easily I can use it in different styles of event-loop processing, as the main place I’d like to use it would be an application that has to manage both an HTTP event-loop and a separate client/server socket simultaneously. (At this point, I don’t yet know how the other socket works, or if it will play well with others. So Mojo might end up not being the deciding factor in that after all.)

So, there are five new modules that I’m currently focused on, any of which might end up getting their own post some day. Here’s hoping that one or two might be of interest to other readers.

Tags: ,

Understanding Encoding (Trying To, At Least)

March 3rd, 2014 | 2 Comments | Posted in Perl, XML

Earlier this evening was a presentation at the San Francisco PM group, on “Unicode & Everything”. I wanted to go, but had a conflict so I had to miss it.

Character encoding is an area I’m weak in, and one that I need to be better at. My biggest module, RPC::XML, supposedly supports encodings other than us-ascii but in truth it’s pretty broken. I recently applied a patch that fixes the handling of UTF-8 content, but that’s not what I need. What I need is for it to properly handle content in (theoretically) any encoding. I don’t think that a talk focused on Unicode would have covered that, but I was hoping that I might be able to corner the speaker afterwards to bounce some questions off of him.

What it comes down to, is this: my library creates requests and responses in well-formed XML, complete with an encoding attribute in the declaration line:

<?xml version="1.0" encoding="us-ascii"?>
            <value><string>Some string data</string></value>

What matters here is not the structure of XML (in this case)– it’s the encoding="us-ascii" part, and this line:

<value><string>Some string data</string></value>

See, my library generates the XML around the “Some string data“, but the string data itself comes from whatever the user provides, and the user expects that to be in the encoding he or she specified. And here is where I start to get confused: I know that the boilerplate code is US-ASCII (in the range that makes it passable as UTF-8), but I suspect that I can’t just paste in a string encoded in ShiftJIS and slap on encoding="shiftjis" in the XML declaration. Or can I?

XML-RPC has a very limited vocabulary and set of data-types. The character range, funny-encoded-strings notwithstanding, is just basic ASCII. You have the tags, then strings, integers, doubles, date/time values (ISO8601) and base-64 data. Regardless of encoding, all of these except the strings stay in the ASCII range.

So for those reading this that are more adept as working with encodings than I, how to approach this? Is the magic sauce somewhere in Perl’s Encode module? I really want to get this part of the RPC::XML module working right, so I can move on to the next big hassle, data compression…

(I also need to figure out why my code-highlighting plugin isn’t doing its job…)

(I think I got it now. Something was missing from one of the theme files. Gotta love WordPress/PHP…)

Tags: , ,

Rant: On Website Passwords

February 27th, 2014 | 2 Comments | Posted in Rants, Security

This morning, my email included a now-familiar tune: A site that I occasionally visit may have been penetrated, so the company was informing the registered users that they needed to reset their passwords. Logging in would take you immediately to a page that let you request a link for password reset. Nothing new, and not that big of a deal. Like I said, it’s been happening quite a bit lately.

But then when I entered a new password, the form comes back immediately, with a message in red that is also becoming all-too-familiar: My password contained characters other than letters and digits. That is not allowed. Please choose another password.

Are you freakin’ kidding me? This is 2014, not 1993. Why in the world does your system limit passwords to just the [A-Za-z0-9] character range? What database, what content-management package are you using, that having a “!” or “$” in the password would break the application? This is just ridiculous. And surely you aren’t storing these passwords as plain-text, I hope? I recently read an excellent article on how one should store passwords in databases, using a combination of salting and one-way hashing functions. Unfortunately I’ve lost the link or I’d include it here. But the point is, you aren’t storing the passwords as plaintext, you shouldn’t be comparing them as plaintext, so why should you care what characters are in the passwords? And then, just to put icing on the cake, once I had selected a new (weak) password they emailed it back to me as part of the confirmation message.

They emailed it to me. In the clear. Not encrypted. They are not the only ones who have done that, either.

I use KeyPassX as a password store. I like it because it is cross-platform; I have it on my MacBook, on my Linux desktop machines, and I have apps that are compatible with the key-store that run on my Android phone and my iPad. All of the installations can configure the location of the key-store, so I have them all pointing to the file on my Dropbox account, thus keeping my passwords in sync across all devices. (Yes, Dropbox is not 100% secure, but the encryption on the key-store file is strong so I’m not worried about someone getting it.) When I looked at the list of sites and passwords this morning, as I was updating the entry for this site, I could tell by looking at the passwords which sites do or do not have this silly restriction on password characters. Among the offenders are two telecoms, a health insurance provider and no fewer than three financial institutions. And these are just the ones that I personally use. I would really have expected more from the financial places, because they have so much more on the line.

Look, I write software for a living, even if I don’t specialize in security. Characters are characters… your software doesn’t view “$” any differently than “Z“, unless the software is doing something horribly badly. Unless your stuff is written amazingly poorly, you can do better on the password front. And if it is written that poorly, consider finding a new vendor… you aren’t doing yourselves any favors with what you currently have.

Tags: ,

Dear LazyWeb: Good Key/Value Data Store for Perl?

February 24th, 2014 | 10 Comments | Posted in CPAN, Perl

I’m working on a web-based tool at my day job in which I will be caching a lot of data. Well, not a lot of data in the Facebook sense of “a lot”, but a non-trivial amount nonetheless.

Currently, for the prototype, I’ve been using a Berkeley DB file with DB_File::Lock to lock it for writes. While this is fine for one or two users, to make sure that my AJAX-driven concurrent data requests don’t trample on each other, it won’t even remotely scale. If five people hit the page at once, four of them will have to wait, and at least one will have a really long wait.

My data requirements are actually pretty simple: I’m storing data about directories, using the directory paths as keys. The values I’m storing are large-ish hashes of data frozen with the Storable module. For example, for a given top-level directory /A, with a few dozen directories simply numbered from 01 on up, I will end up with a collection of keys in the DB_File similar to:

/A        (meta-data, including the list of sub-dirs)
/A.tags   (some cross-reference tag data)
/A/01     (dir-specific data)

So, what should I use for this? The Perl process that creates and accesses the data will be running from an Apache/mod_perl environment, so file-permissions will have to work within that constraint. I haven’t looked at many of the cache-related modules on CPAN yet, but I have looked at MongoDB (and at the Perl driver MongoDB). But I feel like it is almost certainly overkill for this application, and I’m not sure how it handles binary data like what I am storing (I’m sure that it does handle it, I’m just not sure how).

Thoughts and/or suggestions?

Tags: ,

Test Suites: To Critic or Not?

February 19th, 2014 | 2 Comments | Posted in CPAN, Perl

Over the holiday weekend, I spent a good amount of time cleaning up my RPC::XML test suites with regards to Perl::Critic. It wasn’t quick work, either: Much of the suite was written when RPC::XML was first under development, which makes a lot of it well over 10 years old. But I had an extra day, and plenty of time to kill. So 25 files and one git commit later, my test are all clean with respect to my chosen Perl::Critic configuration. The exceptions are a couple of Modules::ProhibitExcessMainComplexity complaints, and most of them also chirp about Modules::RequireVersionVar because the policy doesn’t allow for exclusion of files like test suites.

But all through this, I was wondering: Does it really matter if your test suites are as clean as the code itself?

How often does someone else look at your test suites as closely as they look at the modules that those tests support? I’m not setting up a life-lesson here, I’m genuinely curious. I spent some time thinking about it, and I can say that the only times I’ve looked at test suites in other distributions has been when I was trying to figure out how they did something, or how a particular testing module is used. I’ve very rarely looked for examples of how to write my own tests, and I probably should. I’m pretty sure I could learn from some (most?) of them.

See, I look at Perl::Critic as at least partly geared towards the readability of code; using Perl::Critic helps me make my code more consistent in style and structure, which in turn makes it more readable. I introduced it into my day job (along with a web front-end) largely to try and encourage certain consistencies in style, uniformity even, in the name of corralling Perl coders who had no collective style. Yes, there are a lot of other benefits to critic that are there to be had. It has led me directly to bugs, or to code that could potentially fail, on numerous occasions. But for me, myself, style and structure are the order of the day. So, if few people look at your test suites, is there any reason to run them through the critic?

After a day or two of idle background-thought on it, I’ve decided that it is worth it. Even if the only person who looks at your tests is you. Because you do eventually have to look at those tests. And RPC::XML is not even remotely the oldest CPAN module, so there are distributions out there whose tests are much older than my 10+ years. And that’s plenty of time to forget WTF you were thinking when you wrote the test. The least you can do is try to make it readable for the next time you have to look at it.

Tags: , , ,