Streams: big files in pieces

readFile() loads the whole file into memory at once. For a server log of several gigabytes that will not do: memory runs out. A stream reads or writes data in small pieces — 64 KB each — and processes them as they arrive. This way you can handle a file of any size.

A write stream

createWriteStream() from the node:fs module opens a file for writing in pieces. Let us create a blog access log of ten thousand lines:

import { createWriteStream } from 'node:fs';
 
const log = createWriteStream('access.log');
 
for (let i = 1; i <= 10000; i++) {
  const page = i % 3 === 0 ? '/posts/1' : '/';
  log.write(`2025-01-15 GET ${page}\n`);
}
 
log.end(() => console.log('Log written'));

write() adds a piece of data, end() closes the file and calls the function once everything is written. An access.log file appears next to the program.

A read stream

createReadStream() delivers a file in pieces. Streams support for await, so the pieces are iterated with an ordinary loop:

import { createReadStream } from 'node:fs';
import { writeFile } from 'node:fs/promises';
 
await writeFile('access.log', 'GET /\n'.repeat(50000));
 
let bytes = 0;
let chunks = 0;
for await (const chunk of createReadStream('access.log')) {
  bytes += chunk.length;
  chunks++;
}
 
console.log(`Read ${bytes} bytes in ${chunks} chunks`);

It prints "Read 300000 bytes in 5 chunks": each chunk is a Buffer of up to 64 KB.

Reading line by line

Chunks cut the file at random places, in the middle of a line. To get whole lines, pass the stream to the node:readline module:

import { createReadStream } from 'node:fs';
import { writeFile } from 'node:fs/promises';
import readline from 'node:readline';
 
await writeFile('access.log', 'GET /\nGET /posts/1\nGET /\nGET /posts/1\nGET /posts/1\n');
 
const lines = readline.createInterface({ input: createReadStream('access.log') });
 
let views = 0;
for await (const line of lines) {
  if (line.includes('/posts/1')) views++;
}
 
console.log(`Post 1 was opened ${views} times`); // 3 times

Only one line is in memory at any moment, so the code works the same for five lines and for five million.

pipeline: from stream to stream

pipeline() from node:stream/promises connects streams in a chain: data flows from the first to the last. This is how a file is compressed into a .gz archive with the node:zlib module:

import { createReadStream, createWriteStream } from 'node:fs';
import { stat, writeFile } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { createGzip } from 'node:zlib';
 
await writeFile('access.log', 'GET /posts/1\n'.repeat(20000));
 
await pipeline(createReadStream('access.log'), createGzip(), createWriteStream('access.log.gz'));
 
const before = (await stat('access.log')).size;
const after = (await stat('access.log.gz')).size;
console.log(after < before); // true

pipeline() closes the files itself and passes an error from any link to await.

Common mistake

Connecting streams with the old .pipe() method is risky: if the read stream fails (for example, the file is missing), the write stream stays open, and the error may crash the program. Use pipeline() and wrap it in try...catch:

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
 
try {
  await pipeline(createReadStream('missing.log'), createWriteStream('copy.log'));
} catch (error) {
  console.log(error.code); // ENOENT
}

Thanks for reading!