NodeJS Low Level File System Operation

Home / NodeJS Low Level File System Operation

NodeJS Low Level File System Operation

December 9, 2015 | Article | 1 Comment

Node has a nice streaming API for dealing with files in an abstract way, as if they were network streams. But sometimes, we might need to go down a level and deal with the filesystem itself. Node has facilitate this by providing a low-level file system operation, using module fs.

Get File Metainfo

A metainfo is information about information of a file or directory. In POSIX API, we use stat() and fstat() function to do this. Node, which is inspired by POSIX, has taken this approach too. The stat() and fstat() has been encapsulated to fs module.

var fs = require('fs');

fs.stat('file.txt', function(err, stats) {
    if (err) { console.log (err.message); return }
    console.log(stats);
});

Here we need a fs module and do stat(). A callback is set with two arguments, first is for get the error message if there is an error occurred, and second is the stat information if the function succeeded.

If succeeded, the callback funtion might print something like this (result taken on Cygwin64 on Windows 8):

{ dev: 0,
  mode: 33206,
  nlink: 1,
  uid: 0,
  gid: 0,
  rdev: 0,
  ino: 0,
  size: 2592,
  atime: Thu Sep 12 2013 19:25:05 GMT+0700 (SE Asia Standard Time),
  mtime: Thu Sep 12 2013 19:27:57 GMT+0700 (SE Asia Standard Time),
  ctime: Thu Sep 12 2013 19:25:05 GMT+0700 (SE Asia Standard Time) }

stats is Stats instance, an object, which we cal call some methods of it.

stats.isFile();
stats.isDirectory();
stats.isBlockDevice();
stats.isCharacterDevice();
stats.isSymbolicLink();
stats.isFIFO();
stats.isSocket();

If we have a plain file descriptor, we can use fs.fstat(fileDescriptor, callback) instead.

If using low-level filesystem API in node, we will get file descriptors as a way to represent files. These file descriptors are plain integer numbers given by kernel that represent a file in Node process. Much like C POSIX APIs.

Open and Close a File

Opening a file is a simple matter by using fs.open()

var fs = require('fs');
fs.open('path/to/file', 'r', function(err, fd) {
    // got file descriptor (fd)
});

It’s like C function, if you are familiar with.

The first argument to fs.open is the file path. The second argument is the flags, which indicate the mode with which the file is to be open. The valid flags can be ‘r’, ‘r+’, ‘w’, ‘w+’, ‘a’, or ‘a+’.

  • r = open text file for reading. The stream is positioned at the beginning of the file.
  • r+ = open for reading and writing. The stream is positioned at the beginning of the file.
  • w = truncate file to zero length or create text file for writing. The stream is positioned at the beginning of the file.
  • w+ = open for reading and writing. The file is created if it does not exist, otherwise it is truncated. The stream is positioned at the beginning of the file.
  • a = open for writing. The file is created if it does not exist. The stream is positioned at the end of the file. Subsequent writes to the file will always end up at the current end of file.
  • a+ = open for reading and writing. The file is created if it does not exist. The stream is positioned at the end of the file. Subsequent writes to the file will always end up at the current end of file.

On the callback function, we get the file descriptor or fd as second argument. It is a handler to read and write the file which is opened by fs.open() function.

After operation, it is recommended to close the opened file using fs.close(fd).

Read From a File

Once it’s open, we can read from a file but make sure we have set the mode to allow us read it.

var fs = require('fs');
fs.open('file.txt', 'r', function(err, fd) {
   if (err) { throw err; }
   var readBuffer = new Buffer(1024),
      bufferOffset = 0,
      bufferLength = readBuffer.length,
      filePosition = 100;

   fs.read(fd, readBuffer, bufferOffset, bufferLength, filePosition,
      function(err, readBytes) {
         if (err) { throw err; }
         console.log('just read ' + readBytes + ' bytes');
         if (readBytes > 0) {
            console.log(readBuffers.slice(0, readBytes));
         }
   });
});

Here we open the file, and when it’s opened we are asking to read a chunk of 1024 bytes from it, starting at position 100 (so basically, we read data from bytes 100 to 1124).

A callback is called when one of the following three happens:

  • there is an error
  • something has been read
  • nothing could be read

If there is an error, the first argument of callback – err – will be set. Otherwise, it is null.

The second argument of callback – readBytes – is the number of bytes read into the buffer. If the read bytes is zero, the file has reached the end.

Write Into a File

Once file is open, we can write into a file but make sure we have set the mode to allow us read it.

var fs = require('fs');
fs.open('file.txt', 'a', function(err, fd) {
   if (err) { throw err; }
   var writeBuffer = new Buffer('Writing this string'),
      bufferOffset = 0,
      bufferLength = writeBuffer.length,
      filePosition = null;

   fs.write(fd, writeBuffer, bufferOffset, bufferLength, filePosition,
      function(err, written) {
         if (err) { throw err; }
         console.log('wrote ' + written + ' bytes');
   });
});

Here we open the file with append-mode (‘a’), and we are writing into it, starting at position 0. We pass in the buffer with data we want to written, an offset inside the buffer where we want to start writing from, the length of what we want to write, the file position and a callback.

In this case we are passing in a file position of null, which is to say that we writes at the current file position. As noted before, we open the file using append-mode, so the file cursor is positioned at the end of the file.

Case on Appending

If you are using these low-level file-system functions to append into a file, and concurrent writes will be happening, opening it in append-mode will not be enough to ensure there will be no overlap. Instead, you should keep track of the last written position before you write, doing something like this:

var fs = require('fs');

var startAppender = function(fd, startPos) {
   var pos = startPos;
   return {
      append: function(buffer, callback) {
         var oldPos = pos;
         pos += buffer.length;
         fs.write(fd, buffer, 0, buffer.length, oldPos, callback);
      }
   };
}

Here we declare a function stored on a variable named “startAppender”. This function starts the appender state (position and file descriptor) and then returns an object with an append function.

To use the Appender:

fs.open('file.txt', 'w', function(err, fd) {
   if (err) { throw err; }
   var appender = startAppender(fd, 0);
   appender.append(new Buffer('append this!'), function(err) {
      console.log('appended');
   });
});

And here we are using the appender to safely append into a file.

This function can then be invoked to append, and this appender will keep track of the last position, and increments it according to the buffer length that was passed in.

Actually, there is a problem: fs.write() may not write all the data we asked it to, so we need to modify it a bit.

var fs = require('fs');

var startAppender = function(fd, startPos) {
    var pos = startPos;
    return {
        append: function(buffer, callback) {
            var written = 0;
            var oldPos = pos;
            pos += buffer.length;
            (function tryWriting() {
                if (written < buffer.length) {
                    fs.write(fd, buffer, written, buffer.length - written,
                             oldPos + written, 
                        function(err, bytesWritten) {
                            if (err) { callback(err); return; }
                            written += bytesWritten;
                            tryWriting();
                        }
                    );
                } else {
                   // we have finished
                   callback(null);
                }
            })();
        }
    }
};

Here we use a function named “tryWriting” that will try to write, call fs.write, calculate how many bytes have already been written and call itself if needed. When it detects it has finished (written == buffer.length) it calls callback to notify the caller, ending the loop.

Also, the appending client is opening the file with mode “w”, which truncates the file, and it’s telling appender to start appending on position 0. This will overwrite the file if it has content. So, a wizer version of the appender client would be:

fs.open('file.txt', 'a', function(err, fd) {
   if (err) { throw err; }
   fs.fstat(fd, function(err, stats) {
      if (err) { throw err; }
      console.log(stats);
      var appender = startAppender(fd, stats.size);
      appender.append(new Buffer('append this!'), function(err) {
         console.log('appended');
      });
   })
});

, ,

About Author

about author

xathrya

A man who is obsessed to low level technology.

1 Comment
  1. NodeJS UNIX Sockets - Xathrya.ID

    […] it’s a file-system file descriptor, we can use the Node low-level “fs” module API to read or write […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Social media & sharing icons powered by UltimatelySocial