fs: pass err to callback if buffer is too big #3485

evanlucas · 2015-10-22T12:02:18Z

In fs.readFile, if an encoding is specified and toString fails, do not
throw an error. Instead, pass the error to the callback.

Fixes: #2767

evanlucas · 2015-10-22T12:23:50Z

Went ahead and started the CI to make sure the test passes on all platforms.

https://ci.nodejs.org/job/node-test-pull-request/565/

jasnell · 2015-10-22T22:30:32Z

lib/fs.js

+  if (context.encoding) {
+    const result = tryToString(buffer, context.encoding);
+    if (result.error)
+      return callback(result.error, buffer);


In the error case, I'd much rather the second argument be undefined as opposed to it being a different type from the success case in line 405. Perhaps something like, callback(err,undefined,buffer), and in the success case, callback(null, result.buffer, buffer), so that the second and third arguments are always consistent?

Ugh... nevermind on that, I see you're just copying the existing behavior on this.

Yea, that is why I did that. And honestly, I feel like this is something that should go into v4.x if possible. We could make it better for v5?

However, since you're returning in either case, why not simply pass the callback to the tryToString method...?

if (context.encoding) return tryToString(buffer, context.encoding, callback); //... function tryToString(buf, encoding, callback) { try { return callback(null, buf.toString(encoding)); } catch (err) { return callback(err, buf); } }

Yea, I did it that way first, and then changed it to this way. Not sure why now... :]

Will update

Given that it removes a throw, it's at least a semver-minor or major, which means it wouldn't make it in to v4.x

ah, true. That really sucks...

evanlucas · 2015-10-22T22:47:39Z

ok, updated

jasnell · 2015-10-22T22:52:15Z

I'm going to tag this semver-major for now given the change in the throw. @nodejs/tsc if any of you feel it could be a minor, feel free to change it :-) ... otherwise, LGTM so long as CI is green

trevnorris · 2015-10-22T22:57:49Z

test/parallel/test-fs-readfile-tostring-fail.js

+
+const size = 269 * 1024 * 1024;
+
+const a = new Array(1000000).join('a').toString();


much less expensive to do const a = Buffer(1000000).fill('a').toString();

good call. knocked about 1/2 a second off of it locally

what about const a = 'a'.repeat(1000000) ?

The operation itself is faster, but operating on that string is slower.

'a'.repeat(1000000): 4.5 us/op Buffer(1000000).fill('a').toString(): 180.8 us/op

Now turn those back into a Buffer to be written to disk:

Buffer('a'.repeat(1000000)): 7060.1 us/op Buffer(''+Buffer(1000000).fill('a')): 721.8 us/op

Though there may be some bias from running these in a loop. So overall test time would be the best measurement.

evanlucas · 2015-10-22T23:37:10Z

@trevnorris this isn't ideal, but works and runs in < 0.1 seconds:

'use strict';

const common = require('../common');
const assert = require('assert');
const fs = require('fs');
const path = require('path');

common.refreshTmpDir();

const file = path.join(common.tmpDir, 'toobig.txt');
const stream = fs.createWriteStream(file, {
  flags: 'a'
});

const a = new Buffer(100).fill('a').toString();

for (var i = 0; i < 20000; i++) {
  stream.write(a);
}

stream.end();
stream.on('finish', common.mustCall(function() {
  const orig = Buffer.prototype.toString;
  Buffer.prototype.toString = function() {
    throw new Error('toString failed');
  }
  // make sure that the toString does not throw an error
  fs.readFile(file, 'utf8', common.mustCall(function(err, buf) {
    Buffer.prototype.toString = orig;
    assert.strictEqual('toString failed', err.message);
  }));
}));

process.on('exit', function() {
  fs.unlinkSync(file);
});

trevnorris · 2015-10-23T17:40:40Z

lib/fs.js

+  try {
+    buf = buf.toString(encoding);
+  }
+  catch (err) {


style? i thought the linter would catch this. catch should be on the same line as }.

current version of the eslint doesn't catch it but it will after the update.

trevnorris · 2015-10-23T18:04:50Z

@evanlucas Need to make sure that the amount written to disk is greater than process.binding('buffer').kStringMaxLength.

Oh, why are you toString()'ing the data being written to disk? You can simply pass the buffer directly.

evanlucas · 2015-10-23T19:43:52Z

Ok, updated. PTAL @trevnorris

new CI: https://ci.nodejs.org/job/node-test-pull-request/585/

trevnorris · 2015-10-23T20:01:48Z

test/parallel/test-fs-readfile-tostring-fail.js

+
+// To make sure we don't leave a very large file
+// on test machines in the event this test fails.
+process.on('uncaughtException', function(err) {


may want to add process.on('SIGINT' to capture any ^C. would be the same as here, except run process.exit() at the end. optional, non-blocker.

trevnorris · 2015-10-23T20:02:15Z

one optional suggestion, but LGTM if CI is happy.

evanlucas · 2015-10-23T20:18:16Z

Ok, changes made. CI again: https://ci.nodejs.org/job/node-test-pull-request/586/

evanlucas · 2015-10-23T20:49:47Z

yay, we at least got a yellow build instead of red.

trevnorris · 2015-10-23T21:07:14Z

Awesome. Let's land it.

evanlucas · 2015-10-23T21:07:55Z

k, landing shortly

In fs.readFile, if an encoding is specified and toString fails, do not throw an error. Instead, pass the error to the callback. Fixes: nodejs#2767 PR-URL: nodejs#3485 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

evanlucas · 2015-10-23T21:11:42Z

Landed in b620790. Thanks!

In fs.readFile, if an encoding is specified and toString fails, do not throw an error. Instead, pass the error to the callback. Fixes: #2767 PR-URL: #3485 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Notable changes: * buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859. * console: (Breaking) Values reported by console.time() now have 3 decimals of accuracy added (Michaël Zasso) nodejs#3166. * fs: - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file descriptor as their first argument (Johannes Wüller) nodejs#3163. - (Breaking) In fs.readFile(), if an encoding is specified and the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) nodejs#3485. - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding, callback) form), if the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) nodejs#3503. * http: - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342. - (Breaking) When parsing HTTP, don't add duplicates of the following headers: Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition to the following headers which already block duplicates: Content-Type, Content-Length, User-Agent, Referer, Host, Authorization, Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location, Max-Forwards (James M Snell) nodejs#3090. - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a function or a TypeError is thrown (James M Snell) nodejs#3090. - (Breaking) HTTP methods and header names must now conform to the RFC 2616 "token" rule, a list of allowed characters that excludes control characters and a number of separator characters. Specifically, methods and header names must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown (James M Snell) nodejs#2526. * node: - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078. - (Breaking) Removed require.paths and require.registerExtension(), both had been previously set to throw Error when accessed (Sakthipriyan Vairamani) nodejs#2922. * npm: Upgraded to version 3.3.6 from 2.14.7, see https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a major version bump for npm and it has seen a significant amount of change. Please see the original npm v3.0.0 release notes for a list of major changes (Rebecca Turner) nodejs#3310. * src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary due to the V8 upgrade. Native add-ons will need to be recompiled (Rod Vagg) nodejs#3400. * timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes a long-standing known issue where unrefed timers would perviously hold beforeExit open (Fedor Indutny) nodejs#3407. * tls: - Added ALPN Support (Shigeki Ohtsu) nodejs#2564. - TLS options can now be passed in an object to createSecurePair() (Коренберг Марк) nodejs#2441. - (Breaking) The default minimum DH key size for tls.connect() is now 1024 bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS option can be used to override the default. (Shigeki Ohtsu) nodejs#1831. * util: - (Breaking) util.p() was deprecated for years, and has now been removed (Wyatt Preul) nodejs#3432. - (Breaking) util.inherits() can now work with ES6 classes. This is considered a breaking change because of potential subtle side-effects caused by a change from directly reassigning the prototype of the constructor using `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })` to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)` (Michaël Zasso) nodejs#3455. * v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351. - Implements the spread operator, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator for further information. - Implements new.target, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target for further information. * zlib: Decompression now throws on truncated input (e.g. unexpected end of file) (Yuval Brik) nodejs#2595. PR-URL: nodejs#3466

Notable changes: * buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time (Sakthipriyan Vairamani) #2859. * console: (Breaking) Values reported by console.time() now have 3 decimals of accuracy added (Michaël Zasso) #3166. * fs: - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file descriptor as their first argument (Johannes Wüller) #3163. - (Breaking) In fs.readFile(), if an encoding is specified and the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) #3485. - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding, callback) form), if the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) #3503. * http: - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342. - (Breaking) When parsing HTTP, don't add duplicates of the following headers: Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition to the following headers which already block duplicates: Content-Type, Content-Length, User-Agent, Referer, Host, Authorization, Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location, Max-Forwards (James M Snell) #3090. - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a function or a TypeError is thrown (James M Snell) #3090. - (Breaking) HTTP methods and header names must now conform to the RFC 2616 "token" rule, a list of allowed characters that excludes control characters and a number of separator characters. Specifically, methods and header names must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown (James M Snell) #2526. * node: - (Breaking) Deprecated the _linklist module (Rich Trott) #3078. - (Breaking) Removed require.paths and require.registerExtension(), both had been previously set to throw Error when accessed (Sakthipriyan Vairamani) #2922. * npm: Upgraded to version 3.3.6 from 2.14.7, see https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a major version bump for npm and it has seen a significant amount of change. Please see the original npm v3.0.0 release notes for a list of major changes (Rebecca Turner) #3310. * src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary due to the V8 upgrade. Native add-ons will need to be recompiled (Rod Vagg) #3400. * timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes a long-standing known issue where unrefed timers would perviously hold beforeExit open (Fedor Indutny) #3407. * tls: - Added ALPN Support (Shigeki Ohtsu) #2564. - TLS options can now be passed in an object to createSecurePair() (Коренберг Марк) #2441. - (Breaking) The default minimum DH key size for tls.connect() is now 1024 bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS option can be used to override the default. (Shigeki Ohtsu) #1831. * util: - (Breaking) util.p() was deprecated for years, and has now been removed (Wyatt Preul) #3432. - (Breaking) util.inherits() can now work with ES6 classes. This is considered a breaking change because of potential subtle side-effects caused by a change from directly reassigning the prototype of the constructor using `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })` to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)` (Michaël Zasso) #3455. * v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351. - Implements the spread operator, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator for further information. - Implements new.target, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target for further information. * zlib: Decompression now throws on truncated input (e.g. unexpected end of file) (Yuval Brik) #2595. PR-URL: #3466

mscdex added the fs Issues and PRs related to the fs subsystem / file system. label Oct 22, 2015

jasnell reviewed Oct 22, 2015
View reviewed changes

evanlucas force-pushed the fixreadfileerr branch from ad523b5 to 3574b97 Compare October 22, 2015 22:43

jasnell added the semver-major PRs that contain breaking changes and should be released in the next major version. label Oct 22, 2015

trevnorris reviewed Oct 22, 2015
View reviewed changes

evanlucas mentioned this pull request Oct 23, 2015

v5.0.0 Planning & Checklist #3397

Closed

13 tasks

trevnorris reviewed Oct 23, 2015
View reviewed changes

evanlucas force-pushed the fixreadfileerr branch from 3574b97 to 2047282 Compare October 23, 2015 19:42

trevnorris reviewed Oct 23, 2015
View reviewed changes

evanlucas force-pushed the fixreadfileerr branch from 2047282 to ac227cb Compare October 23, 2015 20:15

evanlucas force-pushed the fixreadfileerr branch from ac227cb to b620790 Compare October 23, 2015 21:11

evanlucas closed this Oct 23, 2015

evanlucas deleted the fixreadfileerr branch October 23, 2015 21:11

evanlucas merged commit b620790 into nodejs:master Oct 23, 2015

evanlucas mentioned this pull request Oct 23, 2015

fs: don't throw in read if buffer too big #3503

Merged

rvagg mentioned this pull request Oct 27, 2015

Release proposal: v5.0.0 #3466

Merged

zbinlin mentioned this pull request Nov 10, 2015

fs: fix readFile will pass undefined to callback but not null #3740

Closed

Trott mentioned this pull request Mar 4, 2019

Investigate flaky test-fs-readfile-tostring-fail #16601

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: pass err to callback if buffer is too big #3485

fs: pass err to callback if buffer is too big #3485

evanlucas commented Oct 22, 2015

evanlucas commented Oct 22, 2015

jasnell Oct 22, 2015

jasnell Oct 22, 2015

evanlucas Oct 22, 2015

jasnell Oct 22, 2015

evanlucas Oct 22, 2015

jasnell Oct 22, 2015

evanlucas Oct 22, 2015

evanlucas commented Oct 22, 2015

jasnell commented Oct 22, 2015

trevnorris Oct 22, 2015

evanlucas Oct 22, 2015

targos Oct 23, 2015

trevnorris Oct 23, 2015

evanlucas commented Oct 22, 2015

trevnorris Oct 23, 2015

targos Oct 23, 2015

trevnorris commented Oct 23, 2015

evanlucas commented Oct 23, 2015

trevnorris Oct 23, 2015

trevnorris commented Oct 23, 2015

evanlucas commented Oct 23, 2015

evanlucas commented Oct 23, 2015

trevnorris commented Oct 23, 2015

evanlucas commented Oct 23, 2015

evanlucas commented Oct 23, 2015


		const size = 269 * 1024 * 1024;

		const a = new Array(1000000).join('a').toString();

fs: pass err to callback if buffer is too big #3485

fs: pass err to callback if buffer is too big #3485

Conversation

evanlucas commented Oct 22, 2015

evanlucas commented Oct 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlucas commented Oct 22, 2015

jasnell commented Oct 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlucas commented Oct 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevnorris commented Oct 23, 2015

evanlucas commented Oct 23, 2015

Choose a reason for hiding this comment

trevnorris commented Oct 23, 2015

evanlucas commented Oct 23, 2015

evanlucas commented Oct 23, 2015

trevnorris commented Oct 23, 2015

evanlucas commented Oct 23, 2015

evanlucas commented Oct 23, 2015