Add default validation based on attribute type #3472

fixe · 2015-04-06T13:23:30Z

We were thinking about adding a validation error to prevent database errors from being thrown since we already know the type before performing the query.

This is immensely useful with PostgreSQL to stop malformed queries from reaching the database.

@janmeier is this something that you guys are willing to merge? If so I can add some test cases.

mickhansen · 2015-04-06T13:25:18Z

Definitely! Schema inferred validations is something we've talked about previously that we'd like to do.
We just need to make sure they are skipped if allowNull: true and val === null

mickhansen · 2015-04-06T13:26:10Z

However like with the allowNull schema validator we might get some complaints that the error messages aren't customizable.

mickhansen · 2015-04-06T13:27:43Z

lib/sql-string.js

@@ -61,6 +61,10 @@ SqlString.escape = function(val, stringifyObjects, timeZone, dialect, field) {
    return val + '';
  }

+  if (field && field.type && field.type.validate) {
+    field.type.validate(val);


Not sure if this should be part of the validator codebase instead?

Yeah, I feel this would fit better in built in validators

What is the best place to put this without losing the type validation when performing queries?

@fixe You mean all non instance-create/update queries? Not sure, instance-validator wouldn't be enough certainly.

I'm talking about every type of query that can be generated by Sequelize. Since we already know the data type of the columns that we are querying we can prevent an error from being thrown by the database.

ping @mickhansen

@fixe I'm still giving it some thought, i like the idea of catching all types before they hit the database but escape seems like an odd place to have it (but yes it's likely the only place being called by all types of values before going into sql).

@janmeier ?

Perhaps in query-generator.escape?

Not much better, but at least the qg is our own code, whereas sql-string is adapted from https://www.npmjs.com/package/sql-string so we might want to keep that as free of external modifications as possible

My original take at this was exactly in that method but it's being overridden in postgres dialect. Suggestions?

Hmm, no, not really sure I can come up with better suggestions then :)

BridgeAR · 2015-04-07T18:49:04Z

Nice +1

BridgeAR · 2015-04-22T13:36:31Z

@fixe Are you still going to fix the existing issues? I really like your proposal and if you do not have the time to finish it, I'd volunteer, even though I'd prefer you finishing the PR.

fixe · 2015-04-22T21:00:42Z

@BridgeAR: still unsure what's missing.

@mickhansen if I fix the outstanding issues will you guys merge this?

mickhansen · 2015-04-23T07:54:01Z

@fixe I need to confer with @janmeier, but yeah i think that it's extremely likely.
I like this as general protection but i also think it would be nice to apply to the schema validations so that it happens during the .validate() process aswell.

fixe · 2015-04-23T23:19:55Z

Let me know :-)

janmeier · 2015-04-24T07:41:23Z

I think we should move the validate call to escape in the query generator. It would introduce some duplication since we need ti both in abstract and in PG, but I think it would still be nicer than in sql string.

And as mick said, I think type validation should also be called in the validation step https://github.com/sequelize/sequelize/blob/master/lib/instance-validator.js#L189

So: move validate to escape, add validate to instance-validator, and add some tests that calling .validate throws the appropriate errors.

If you can come up with an awesome way to let users specify custom messages for the validations that'd be awesome, but if not thats fine and we can defer that to later :)

BridgeAR · 2015-05-19T20:36:20Z

Ping @fixe: Do you think you can finish this this week? I need this soon, since I had some issues today because of the missing validation.

fixe · 2015-05-19T20:42:58Z

@BridgeAR impossible -- drowned this week, sorry.

BridgeAR · 2015-05-19T20:43:47Z

Ok thx

BridgeAR · 2015-06-01T02:24:58Z

This fixes #1431

fixe · 2015-06-08T22:22:50Z

@janmeier: I have updated the PR by moving the code from sql string to the query generator. I need feedback in some validators (returning true for now).

janmeier · 2015-06-10T07:37:18Z

lib/data-types.js

@@ -376,6 +440,9 @@ var HSTORE = function() {
 util.inherits(HSTORE, ABSTRACT);

 HSTORE.prototype.key = HSTORE.key = 'HSTORE';
+HSTORE.prototype.validate = function(value) {
+  return true;


_.isPlainObject and maybe check that it is not more than one level deep

How do you suggest to check for more than one level deep here?

janmeier · 2015-06-21T10:10:02Z

Ping @fixe

fixe · 2015-06-21T16:38:24Z

@janmeier implemented your suggestions and added some tests. Take a look.

janmeier · 2015-06-22T15:21:20Z

test/unit/sql/data-types.test.js

+        test('should throw an error if `value` is invalid', function() {
+          var type = DataTypes.STRING();
+
+          try {


expect(function () { type.validate(12345); }).to.throw(Sequelize.ValidationError, '12345 is not a valid string');

Same applies for the other tests

janmeier · 2015-06-22T15:31:55Z

Hmm, for some reason this is failing miserably for postgres - Most of the errors seem to come from .values being empty in enums...

Do you need help on this @fixe or can you debug yourself?

fixe · 2015-06-22T23:31:55Z

@janmeier the thing is that enums are being created in a way that the data type don't have access to values:

owner_type: { type: DataTypes.ENUM, values: ['user', 'org'], defaultValue: 'user', unique: 'combiIndex' }

In order for the validation to work the following must be done:

owner_type: { type: DataTypes.ENUM({ values: ['user', 'org'] }), defaultValue: 'user', unique: 'combiIndex' }

WDYT?

fixe · 2015-07-14T22:17:51Z

@janmeier PR updated, can you guys port this to other dialects?

janmeier · 2015-07-15T07:09:59Z

@fixe I can sure give it a try :)

janmeier · 2015-07-17T20:13:09Z

lib/query-interface.js

@@ -406,7 +406,7 @@ QueryInterface.prototype.renameColumn = function(tableName, attrNameBefore, attr

    _options[attrNameAfter] = {
      attribute: attrNameAfter,
-      type: data.type,
+      type: DataTypes[data.type],


@fixe This was breaking a mysql test so I removed it. Tests seem to pass fine without this change

…tion Add default validation based on attribute type

fixe · 2015-07-18T00:21:26Z

👍

janmeier · 2015-07-19T12:26:40Z

Good work @fixe !

wilsoncook · 2015-09-15T02:47:26Z

lib/data-types.js

@@ -317,6 +368,13 @@ BOOLEAN.prototype.key = BOOLEAN.key = 'BOOLEAN';
 BOOLEAN.prototype.toSql = function() {
  return 'TINYINT(1)';
 };
+BOOLEAN.prototype.validate = function(value) {
+  if (!_.isBoolean(value)) {


I vote for using Validator.isBoolean(), _.isBoolean() is too strict even the value is number 1 or 0

franciscotfmc · 2015-10-02T18:01:25Z

Hello guys,

Maybe I'm missing something here, but suppose we have one model with an integer attribute, with values from 1000 to 2000 in the database. My need is to search for all the rows that begin with '10'. In order for this to work, I would need to disable the type validation for queries. Am I wrong?

Thanks in advance!

dannielhugo · 2015-10-02T18:03:21Z

+1 for @franciscotfmc comment

talitacampos · 2015-10-02T18:05:54Z

+1 for @franciscotfmc comment

ceciliadeveza · 2015-10-02T18:06:07Z

+1 for @franciscotfmc comment

shbmira · 2015-10-02T18:08:02Z

+1 for @franciscotfmc comment

mickhansen · 2015-10-02T21:19:28Z

@franciscocardoso gte 1000 lt 1100?
In any case, see the current issue about the problems with type validators.

franciscocardoso · 2015-10-03T09:07:00Z

Hi,

@mickhansen, I believe you were trying to ping @franciscotfmc.

mickhansen · 2015-10-03T09:21:21Z

@franciscocardoso I was, sorry about that, the github auto complete is not entirely context aware apparently :)

franciscotfmc · 2015-10-05T11:29:38Z

@mickhansen the case is that the query I'm about to execute comes from an input in the browser. The user can type pretty much anything and I need to search in a bunch of fields, including the integer type, checking if its value starts with '10'. The user can even type something like '10 abcd'.

mickhansen · 2015-10-05T11:42:36Z

@franciscocardoso So you just throw whatever input and try to LIKE match it against all fields?

franciscotfmc · 2015-10-05T11:59:02Z

Basically that, @mickhansen. This is one of those screens with only one search input and a list of results. I don't think that using multiple inputs would be very friendly in my case, so that I could use gte and lt as you said. Picture the facebook search as an example. There you can type a profile name or a phone number. If something contains my typed query, it'll show up.

franciscocardoso · 2015-10-05T12:35:43Z

Wrong handle again @mickhansen.

mickhansen · 2015-10-05T12:57:57Z

@franciscocardoso Oh god you must hate me by now.

franciscocardoso · 2015-10-05T13:42:07Z

That's ok @mickhansen , no worries.

mickhansen reviewed Apr 6, 2015
View reviewed changes

fixe force-pushed the feature/add-default-validation branch from 885bb8d to a83ecd2 Compare April 15, 2015 23:26

fixe force-pushed the feature/add-default-validation branch from a83ecd2 to a3b11be Compare April 28, 2015 10:39

BridgeAR mentioned this pull request May 31, 2015

Empty objects in WHERE clause result in error #3341

Closed

fixe force-pushed the feature/add-default-validation branch from a3b11be to c62bafa Compare June 8, 2015 22:20

janmeier reviewed Jun 10, 2015
View reviewed changes

fixe force-pushed the feature/add-default-validation branch from c62bafa to 73c0b70 Compare June 21, 2015 16:37

janmeier reviewed Jun 22, 2015
View reviewed changes

fixe force-pushed the feature/add-default-validation branch from 73c0b70 to 49d4a7c Compare June 22, 2015 23:29

Add default validation based on attribute type

a243a82

fixe force-pushed the feature/add-default-validation branch from b969f47 to a243a82 Compare July 14, 2015 22:15

janmeier assigned janmeier Jul 15, 2015

janmeier reviewed Jul 17, 2015
View reviewed changes

janmeier merged commit a243a82 into sequelize:master Jul 17, 2015

janmeier added a commit that referenced this pull request Jul 17, 2015

Merge pull request #3472 from seegno-forks/feature/add-default-valida…

7073e03

…tion Add default validation based on attribute type

fixe deleted the feature/add-default-validation branch July 18, 2015 00:03

janmeier mentioned this pull request Aug 25, 2015

i18n on validation and database errors #4361

Open

wilsoncook reviewed Sep 15, 2015
View reviewed changes

janmeier mentioned this pull request Oct 21, 2015

DatabaseError instead of ValidationError when inserting invalid enum value #4704

Closed

sushantdhiman mentioned this pull request Dec 3, 2016

Add default validation based on attribute type #1431

Closed

Add default validation based on attribute type #3472

Add default validation based on attribute type #3472

Conversation

fixe commented Apr 6, 2015

mickhansen commented Apr 6, 2015

mickhansen commented Apr 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BridgeAR commented Apr 7, 2015

BridgeAR commented Apr 22, 2015

fixe commented Apr 22, 2015

mickhansen commented Apr 23, 2015

fixe commented Apr 23, 2015

janmeier commented Apr 24, 2015

BridgeAR commented May 19, 2015

fixe commented May 19, 2015

BridgeAR commented May 19, 2015

BridgeAR commented Jun 1, 2015

fixe commented Jun 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janmeier commented Jun 21, 2015

fixe commented Jun 21, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janmeier commented Jun 22, 2015

fixe commented Jun 22, 2015

fixe commented Jul 14, 2015

janmeier commented Jul 15, 2015

Choose a reason for hiding this comment

fixe commented Jul 18, 2015

janmeier commented Jul 19, 2015

Choose a reason for hiding this comment

franciscotfmc commented Oct 2, 2015

dannielhugo commented Oct 2, 2015

talitacampos commented Oct 2, 2015

ceciliadeveza commented Oct 2, 2015

shbmira commented Oct 2, 2015

mickhansen commented Oct 2, 2015

franciscocardoso commented Oct 3, 2015

mickhansen commented Oct 3, 2015

franciscotfmc commented Oct 5, 2015

mickhansen commented Oct 5, 2015

franciscotfmc commented Oct 5, 2015

franciscocardoso commented Oct 5, 2015

mickhansen commented Oct 5, 2015

franciscocardoso commented Oct 5, 2015