I have a new plan for 2012 onward. It’s much harder to find topics now and it takes much longer to research and write them. I’ve exhausted all of the advice I have and all of the easy topics. When I think I have a good idea, I now know to search everything else I’ve already written. More than a couple of times I thought I had the next week’s idea, but it was already in the book or on the website.

The search for content has another problem: I don’t want to add Items for anything that’s already been written—not just by me, but by anyone. I don’t want to repeat content unless I have a different take on it and I can illuminate something new.

I’m not going to do weekly big Items anymore. I’ll try one a month, I think. I’ll see how it goes.

There’s still roomer for shorter content, such as the short demonstrations of new features, and ideas that don’t have have 1,000 words in them. There are also many interesting, although esoteric, features that I would probably never recommend for production code.

This doesn’t mean that I’m going to write less, though. If I do less for The Effective Perler, I can do more somewhere else. There another edition of Intermediate Perl that needs attention, the Learning Perl website, or a few other things.

Many people have asked for Items about specific modules, but that’s not really the idea of The Effective Perler. We want to teach people about core Perl and thinking in Perl. Modules are essentially all the same—they give you an interface and you do what the interface tells you to do. For the most part, they are just subroutines or method calls. There’s not much interesting there. You already know how to do that. The much more interesting advice is researching modules, but we put that in the book already.

If you want to read and write lines, you can use the -n switch to wrap a while loop around your tiny program. In this case, those tiny programs just call a normalization function from Unicode::Normalize. Here are the bash aliases:

alias nfc="perl5.14.1 -MUnicode::Normalize -CS -ne 'print NFC(\$_)'"
alias nfd="perl5.14.1 -MUnicode::Normalize -CS -ne 'print NFD(\$_)'"
alias nfkd="perl5.14.1 -MUnicode::Normalize -CS -ne 'print NFKC(\$_)'"

You can run these as if they were programs with those names. Here you convert those ligature characters, fi (U+FB01) and fl (U+FB02), to their compatible, two-character forms fi and fl as it reads standard input:

$ nfkd
Let's find that flying squirrel!
Let's find that flying squirrel!

If you wanted to do it with command line arguments as strings instead of files, it’s a couple small changes. You can add the A flag to the -C switch to interpret the command-line arguments as UTF-8 (unless you want to decode it yourself), and use say to add the newline in the output:

alias nfc="perl5.14.1 -MUnicode::Normalize -CSA -E 'say NFC( qq(@ARGV) )'"
alias nfd="perl5.14.1 -MUnicode::Normalize -CSA -E 'say NFD( qq(@ARGV) )'"
alias nfkd="perl5.14.1 -MUnicode::Normalize -CSA -E 'say NFKC( qq(@ARGV) )'"

The output decomposes the ligatures just as before:

nfkd "Let's find that flying squirrel."
Let's find that flying squirrel.

You can read more about these program features in Item 73. Tell Perl which encoding to use and Item 77. Work with graphemes instead of characters.

Suppose you want to find some dates inside a big string. The problem with dates is that there are some many ways to write them, and even if you can come up with a pattern to get the structure right, can you handle the different locales and languages that use different words to refer to the same day or month?

In Item 42. Don’t reinvent the regex, you saw the Regexp::Common module. It creates the regular expressions that many people often get wrong because they miss some subtle part of the pattern.

Regexp::Common::time‘s date handling is quite amazing though. It’s a plugin, so you need to install it separately. Instead of specifying a regular expression, you can use the -pat option to specify the structure of the date, using a string much like that for strftime, although with some regular expression bits added. From the semi-pattern, it constructs a much more complicated pattern that does the right thing. Since the module gives you a regex object, you can print it to see the pattern:

In this example, you extract the

use Regexp::Common qw(time);

my @lines = `ls -l`;

# May  3  2010
# Jan 17 18:21
$date_re = $RE{time}{strftime}{
	-pat => '%b\s+%_d\s+(?:%Y|%_H:%M)'
	};

print "Pattern is------\n$date_re\n-------\n";

This pattern reflects the national representation for the en_US locale:

Pattern is------
(?=[SAFOJNMD])(?>Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+(?:0[1-9]|[12]\d|3[01]|(?<!\d)[1-9])\s+(?:\d{4}|(?:(?=\d)(?:[01]\d|2[0123]|(?<!\d)\d)):(?:[0-5]\d))
-------

You can change your locale, in this case, to tr_TR for Turkish, to get a different pattern that has the same structure, although I don’t know if the Turks write their dates like this:

Pattern is------
(?=[AOTNKEHM\Å])(?>Oca|\Å\žub|Mar|Nis|May|Haz|Tem|A\Ä\Ÿu|Eyl|Eki|Kas|Ara)\s+(?:0[1-9]|[12]\d|3[01]|(?<!\d)[1-9])\s+(?:\d{4}|(?:(?=\d)(?:[01]\d|2[0123]|(?<!\d)\d)):(?:[0-5]\d))
-------

You can now use this pattern to match dates in text. Here’s a program that takes in a line and puts ^ characters under the parts it thinks are dates:

use Regexp::Common qw(time);

my @lines = `ls -l`;

# May  3  2010
# Jan 17 18:21
$date_re = $RE{time}{strftime}{
	-pat => '%b\s+%_d\s+(?:%Y|%_H:%M)'
	};

while( defined( my $line = <> ) {
	next unless $line =~ /$date_re/;
	my $start = $-[0];
	my $stop  = $+[0];

	my $underline = ( ' ' x $-[0] ) . ( '^' x ($stop - $start) );

	print $line;
	print $underline, "\n\n";
	}

You can test this by piping some output into this program. Here’s an extract of output from the Unix ls command. Notice that the first date has a time instead of a year, but you still find it:

$ ls -l /usr/local/perls/perl-5.10.1/lib/site_perl/5.10.1 | perl date_finder.pl
drwxr-xr-x   4 brian  wheel    136 Dec  9 01:58 Acme
                                   ^^^^^^^^^^^^

-r--r--r--   1 brian  wheel  32517 Jul  6  2007 AppConfig.pm
                                   ^^^^^^^^^^^^

-r--r--r--   1 brian  wheel  54725 Jul 19  2007 Expect.pm
                                   ^^^^^^^^^^^^

-r--r--r--   1 brian  wheel  43735 Jul 19  2007 Expect.pod
                                   ^^^^^^^^^^^^

drwxr-xr-x   3 brian  wheel    102 May 16  2010 ExtUtils
                                   ^^^^^^^^^^^^

drwxr-xr-x   3 brian  wheel    102 Jun 17  2010 local
                                   ^^^^^^^^^^^^

-r--r--r--   1 brian  wheel   9137 Jun 15  2009 lwpcook.pod
                                   ^^^^^^^^^^^^

-r--r--r--   1 brian  wheel  25447 Jun 15  2009 lwptut.pod
                                   ^^^^^^^^^^^^

drwxr-xr-x   4 brian  wheel    136 May 28  2010 namespace
                                   ^^^^^^^^^^^^

-r--r--r--   1 brian  wheel   1931 Sep 22  2009 oose.pm
                                   ^^^^^^^^^^^^

Notice that this would be hard to do with split if you run into filenames that have spaces. You can’t depend on fixed column widths because the file sizes can move things around. It turns out to be pretty annoying.

I had to find some dates inside a big string, and the problem with dates is that there are some many ways to write them, and even if I get the format right, some of the machines might use another locale. My string comes from an ls I run as a remote command, which might show the date in one of two formats. The files changed in the last six months replaces the year with the time:

$ ls -l
total 7400
-rw-r--r--@  1 brian  staff      433 Jun 22  2010 Makefile
-rw-r--r--@  1 brian  staff   107721 Jan 19 09:08 appa.xml
-rw-rw-r--@  1 brian  staff    76873 Jan 19 00:18 appb.xml
-rw-rw-r--   1 brian  staff     1802 Jan 14 21:17 book.xml
-rw-rw-r--   1 brian  staff  2457812 Jul 21  2010 book.xml.pdf
-rw-rw-r--   1 brian  staff     4360 Jul 21  2010 bookinfo.xml
-rw-r--r--@  1 brian  staff    25626 Jan 19 09:07 ch00.xml

Here’s the program I wrote to figure out which parts of that string is the dates, using Regexp::Common (Item 42. Don’t reinvent the regex):

use Regexp::Common qw(time);

my @lines = `ls -l`;

# May  3  2010
# Jan 17 18:21
$date_re = $RE{time}{strftime}{
	-pat => '%b\s+%_d\s+(?:%Y|%_H:%M)'
	};

foreach my $line ( @lines ) {
	next unless $line =~ /$date_re/;
	my $start = $-[0];
	my $stop  = $+[0];

	my $underline = ( ' ' x $-[0] ) . ( '^' x ($stop - $start) );

	print $line;
	print $underline, "\n\n";
	}

That regex is more sophisticated than it looks. I didn’t have to do anything to deal with month names and abbreviations, but the module will figure it out for me based on the locale of the machine on which I run the command. The regex changes depending on the language that I decide to use:

$ LC_ALL=tr_TR perl -MRegexp::Common=time -le 'print $RE{time}{strftime}{-pat=>"%b"}'
(?:(?=[AOTNKEHM\Å])(?>Oca|\Å\žub|Mar|Nis|May|Haz|Tem|A\Ä\Ÿu|Eyl|Eki|Kas|Ara))

$ LC_ALL=en_US.UTF-8 perl -MRegexp::Common=time -le 'print $RE{time}{strftime}{-pat=>"%b"}'
(?:(?=[SAFOJNMD])(?>Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec))

$ LC_ALL=es_ES.UTF-8 perl -MRegexp::Common=time -le 'print $RE{time}{strftime}{-pat=>"%b"}'
(?:(?=[enadmsjfo])(?>ene|feb|mar|abr|may|jun|jul|ago|sep|oct|nov|dic))

Part of my code demonstrated that it found the date part of the string by underlining what it thought the date portion was. That’s all the fooling around with the @- and @+ special variables. Those are the string locations for the start and end positions of the various capture buffers. The numbers in index 0 applies to $&, the index 1 applies to $1, and so on:

-rw-r--r--@  1 brian  staff      433 Jun 22  2010 Makefile
                                     ^^^^^^^^^^^^

-rw-r--r--@  1 brian  staff   107721 Jan 19 09:08 appa.xml
                                     ^^^^^^^^^^^^

-rw-rw-r--@  1 brian  staff    76873 Jan 19 00:18 appb.xml
                                     ^^^^^^^^^^^^

-rw-rw-r--   1 brian  staff     1802 Jan 14 21:17 book.xml
                                     ^^^^^^^^^^^^

-rw-rw-r--   1 brian  staff  2457812 Jul 21  2010 book.xml.pdf
                                     ^^^^^^^^^^^^

-rw-rw-r--   1 brian  staff     4360 Jul 21  2010 bookinfo.xml
                                     ^^^^^^^^^^^^

-rw-r--r--@  1 brian  staff    25626 Jan 19 09:07 ch00.xml
                                     ^^^^^^^^^^^^

This code also has to work on systems with very ancient versions of ls. There are some switches that could have made this code much easier, especially if I can make the date column the epoch time instead do it’s not a combination of whitespace-separated fields itself.

• The -T switch on Mac OS X and FreeBSD displays all dates in the same format, even for the recently changed ones.
• Linux versions might have the --time-style.
• FreeBSD has the -D switch to specify the date format.

I’d much rather use perl, but the equivalent is much uglier even though I can choose my field separator. Perl is ultra-portable and available in most places, but I have to do more work on a one-liner:

$ perl -le 'for(glob(q|*|)){print join qq|\t|, stat(), $_}'

However, this causes headaches later when I need to run this as a remote command and I still have to process the results to turn the data into human-readable output. The ls -l is much nicer without requiring more work than I’d do normally.

And, as a bonus to this bonus, I discovered that Date::Parse is smart enough to deal with a date like Dec 31 12:34. It realizes that it was last December, not the one from the current year. I can feed both formats into that module and still have the dates sort correctly.