Memory-map files instead of slurping them

The conventional wisdom for slurping a file into a Perl program is to actually load the file into a program. We showed some of these in Item 53: Consider different ways of reading from a stream.

There are several idioms for doing it, from doing it yourself:

my $text = do { local( @ARGV, $/ ) = $file; <> };

or using an optimized module such as File::Slurp.

use File::Slurp qw(read_file);

my $text = read_file( $file );

Given a large file, say, something that is 2 GB, you end up with a memory footprint that is at least the file size. This program to load a 2 GB file took 11 seconds to load the file on my Mac Pro. The memory footprint rose to 2.25 GB and stayed there even after $text went out of scope:

use strict;
use warnings;

print "I am $$\n";

use File::Slurp;

my $start = time;
my $text = read_file( $ARGV[0] );
my $loadtime = time - $start;
print "Loaded file in $loadtime seconds\n";

my $count = () = $text =~ /abc/;

print "Found $count occurances\n";

print "Press enter to continue...";


The problem is in the concept that you have to somehow capture and retain control of the data to make use of it.

To solve this, you should avoid the painful part. That is, don’t load the file at all. That I/O is really slow! You can memory-map, or mmap, the file. The name comes from the system call that makes it possible.

Instead of loading the file, you use mmap to make a connection between your address space and the file on the disk. You don’t have to worry about how this happens, but basically you use part of a disk file as if it was actually in memory. The advantage is that you don’t have the I/O overhead, so there is no load time, and since you don’t have to make space to hold the file in memory, you don’t pay a memory footprint.

This program use File::Map, you “load” the file instantly and it’s actual memory footprint was under 3 MB (three orders of magnitude less!):

use strict;
use warnings;

use File::Map qw(map_file);

print "I am $$\n";

my $start = time;
map_file my $map, $ARGV[0];
my $loadtime = time - $start;
print "Loaded file in $loadtime seconds\n";

my $count = () = $map =~ /abc/;

print "Found $count occurances\n";


The $map acts just like a normal Perl string, and you don’t have to worry about any of the mmap details. When the variable goes out of scope, the map is broken and your program doesn’t suffer from a large chunk of unused memory.

In Tim Bray’s Wide Finder contest to find the fatest way to process log files with “wider” rather than “faster” processors, the winning solution was a Perl implementation using mmap (although using the older Sys-Mmap). Perl had nothing special in that regard because most of the top solutions used mmap to avoid the I/O penalty.

The mmap is especially handy when you have to do this with several files at the same time (or even sequentially if Perl needs to find a chunk of contiguous memory). Since you don’t have the data in real memory, you can mmap as many files as you like and work with them simultaneously.

Also, since the data actually live on the disk, different programs running at the same time can share the data, including seeing the changes each program makes (although you have to work out the normal concurrency issues yourself). That is, mmap is a way to share memory.

The File::Map module can do much more too. It allows you to lock filehandles, and you can also synchronize access from threads in the same process.

If you don’t actually need the data in your program, don’t ever load it: mmap it instead.

What’s the difference between a list and an array?

I recently updated perlfaq4‘s answer to “What’s the difference between a list and an array?”. The difference between data and variables is often lost of the person who starts their programming career in a high level language.

We hit this subject pretty hard in the first chapter of Effective Perl Programming, 2nd Edition in at least three Items:

  • Item 9: Know the difference between lists and arrays.
  • Item 10: Don’t assign undef when you want an empty array.
  • Item 12: Understand context and how it affects operations.

Here’s the current answer in perlfaq4:

A list is a fixed collection of scalars. An array is a variable that holds a variable collection of scalars. An array can supply its collection for list operations, so list operations also work on arrays:

# slices
( 'dog', 'cat', 'bird' )[2,3];

# iteration
foreach ( qw( dog cat bird ) ) { ... }
foreach ( @animals ) { ... }

my @three = grep { length == 3 } qw( dog cat bird );
my @three = grep { length == 3 } @animals;

# supply an argument list
wash_animals( qw( dog cat bird ) );
wash_animals( @animals );

Array operations, which change the scalars, reaaranges them, or adds or subtracts some scalars, only work on arrays. These can’t work on a list, which is fixed. Array operations include shift, unshift, push, pop, and splice.

An array can also change its length:

$#animals = 1;  # truncate to two elements
$#animals = 10000; # pre-extend to 10,001 elements

You can change an array element, but you can’t change a list element:

$animals[0] = 'Rottweiler';
qw( dog cat bird )[0] = 'Rottweiler'; # syntax error!

foreach ( @animals ) {
	s/^d/fr/;  # works fine

foreach ( qw( dog cat bird ) ) {
	s/^d/fr/;  # Error! Modification of read only value!

However, if the list element is itself a variable, it appears that you can change a list element. However, the list element is the variable, not the data. You’re not changing the list element, but something the list element refers to. The list element itself doesn’t change: it’s still the same variable.

You also have to be careful about context. You can assign an array to a scalar to get the number of elements in the array. This only works for arrays, though:

my $count = @animals;  # only works with arrays

If you try to do the same thing with what you think is a list, you get a quite different result. Although it looks like you have a list on the righthand side, Perl actually sees a bunch of scalars separated by a comma:

my $scalar = ( 'dog', 'cat', 'bird' );  # $scalar gets bird

Since you’re assigning to a scalar, the righthand side is in scalar context. The comma operator (yes, it’s an operator!) in scalar context evaluates its lefthand side, throws away the result, and evaluates it’s righthand side and returns the result. In effect, that list-lookalike assigns to $scalar it’s rightmost value. Many people mess this up becuase they choose a list-lookalike whose last element is also the count they expect:

my $scalar = ( 1, 2, 3 );  # $scalar gets 3, accidentally

Get a review copy of Effective Perl Programming

Addison-Wesley is ramping up its promotion of Effective Perl Programming, 2nd Edition. They’ll send out several free copies to people who like to review books and tell their friends about it. If you’re interested in getting one of those review copies, send me a note at . If I don’t already know you, introduce yourself and send me some links to your previous book reviews, your blog, and so on.

Effective Perl Programming at Frozen Perl 2010

On Friday, February 5, 2010 at Frozen Perl, I’m teaching a new master class based on our book, Effective Perl Programming, 2nd Edition. The master class is a low-cost, condensed class format given at a public Perl event. The cost of this one-day class is $100, and there are a limited number of $25 seats for full-time students.

In the one-day class for intermediate Perl programmers, I’ll cover selected topics from the book, including:

  • Working with Unicode in Perl
  • Tricks with filehandles
  • New regex features in Perl 5.10 and later
  • Playing with pack
  • Using closures to make things simpler
  • and other topics as time allows

Although the book hasn’t been published yet, it is available for pre-order on Amazon, and attendees to the class can get a sneak peek at the working manuscript as well as a soft copy of the course slides.

Effective Perl Programming, the blog

Effective Perl Programming, 2nd Edition, front cover. Josh McAdams and I recently gave Addison Wesley our final manuscript for Effective Perl Programming, 2nd Edition, an update to Joseph Hall’s excellent first edition published in 1998. Not only have we updated the existing material up to Perl 5.10, but we’ve also quite a bit of new material for the evolution of Perl and its practices in the last 12 years.

We’re not stopping there, though. We had plenty more that we wanted to write and couldn’t fit in the book. Sometimes we had more to say on an item in the book and sometimes we wanted to include a topic that didn’t have the space for it. We will trickle out that extra material here. Additionally, we’d like to keep up a conversation with the readers of the latest edition to refine and expand the material that is already there.

Perl 5.8 features

Perl 5.8, mostly a release that improved much of the unseen things in Perl, did have some notable, user-visible features.

  • Restricted hashes with Hash::Util (5.8.0)
  • Use to affect all programs (5.8.7)
  • PerlIO is now the default (5.8.0)