Recipe 7.7 Caching Open Output Filehandles

7.7.1 Problem

You need more output files open simultaneously than your system allows.

7.7.2 Solution

Use the standard FileCache module:

use FileCache;
cacheout ($path);         # each time you use a filehandle
print $path "output";

7.7.3 Discussion

FileCache's cacheout function lets you work with more output files than your operating system lets you have open at any one time. If you use it to open an existing file that FileCache is seeing for the first time, the file is truncated to length zero, no questions asked. However, in its opening and closing of files in the background, cacheout tracks files it has opened before and does not overwrite them, but appends to them instead. This does not create directories for you, so if you give it /usr/local/dates/merino.ewe to open but the directory /usr/local/dates doesn't exist, cacheout will die.

The cacheout function checks the value of the C-level constant NOFILE from the standard system include file sys/param.h to determine how many concurrently open files are allowed on your system. This value can be incorrect on some systems and even missing on a few (for instance, on those where the maximum number of open file descriptors is a process resource limit that can be set with the limit or ulimit commands). If cacheout can't get a value for NOFILE, set $FileCache::cacheout_maxopen to be four less than the correct value, or choose a reasonable number by trial and error.

Example 7-1 splits an xferlog file (created by most FTP servers nowadays) into separate files, each named after the authenticated user. Fields in xferlog files are space-separated, with the fourth field from the last holding the authenticated username.

Example 7-1. splitwulog

  #!/usr/bin/perl
  # splitwulog - split wuftpd log by authenticated user
  use FileCache;
  $outdir = "/var/log/ftp/by-user";
  while (<>) {
      unless (defined ($user = (split)[-4])) {
         warn "Invalid line: $.\n";
         next;
      }
      $path = "$outdir/$user";
      cacheout $path;
      print $path $_;
  
  
  
  
  }

7.7.4 See Also

Documentation for the standard FileCache module (also in Chapter 32 of Programming Perl); the open function in perlfunc(1) and in Chapter 29 of Programming Perl