One of the things that I have had many feature requests for (and implemented in both JSON fields I've built) is the ability to customise encoding and decoding.

I've done this in the most recent one by allowing __init__ kwargs on the field itself for encode_kwargs and decode_kwargs, but I wonder if there might be better ways of doing this.

The related things I have been playing with related to this are sane defaults for these. My gut feeling is that we should use DjangoJSONEncoder as the default encoder (encode_kwargs={'cls':DjangoJSONEncoder}), and perhaps using decimal.Decimal as the default parse_float for decode_kwargs.

I'm not sure it will be possible to do this per field though: http://initd.org/psycopg/docs/extras.html#json-adaptation

It seems that overriding the dumping function might be possible, but the loading function is global.

Aha. As of psycopg2 2.5.4, JSONB is automatically registered (and converted by psycopg2) into dict/list structures.

Is this something we want to override, and allow the flexibility of parsing per-field? I don't see that it will be a performance hit, as it will be basically the same code, but just running in the context of the django field, rather than the psycogp2 layer.

It will also mean that the JSON field could be used (without the lookup functionality) for other database backends.

Hmm. It seems that the use_decimal flag for json.encode is simplejson-only.

I'm not sure if there is any nice way to handle this.

At present, I'd just let psycopg2 do it's thing. The use of DjangoJSONEncoder is an interesting point, sensible serialisation of datetimes is super useful.

We can handle the dumping on a field by field basis quite easily as we wrap the objects in a psycopg2 json object which can take a dumping function, but it's more difficult to customise the loading on a field by field basis as this is registered in psycopg2 against the field type, not the column specifically. I'm not familiar enough with the internals of psycopg2 to know how this works. It may be worth investigating the performance difference. If it's insignificant we're ok.

I looked a bit more into implementing something with DjangoJSONEncoder, then I realised there's a clue in it's name - it doesn't do decoding at all. When loading back into models from a dumped json format fixture this is fine because the model creation is happy to understand the string versions it it's to_python method, but this won't be sufficient for loading back into a json based field. It is important that saving and loading the model field value should always give the original result back.

I think there is a possible solution to override the encoders on a field by field basis, but I am concerned about the performance implications. Django+postgres is almost completely free of "converter functions", which are a known performance hit (due to a second loop through all the rows after psycopg2). That said, loading large json blobs for a large number of rows will have a performance hit no matter how you look at it, and I think that using .defer() won't make any difference if we allow the conversion to happen at the psycopg2 level.

Nonetheless, I feel this whole issue can be delayed for future consideration. We should get a basic functioning version in and then we can add improvements like this later.

I've been thinking about this too (whilst playing with CompositeField stuff, which is only related in the sense that it deals with conversion of data).

It's always going to be problematic converting data back into whatever it was, as things like datetime are indistinguishable from a string that happens to be an ISO8601 string, for instance.

Perhaps a better solution is to allow passing in a "converter" as needed, that will be applied in the new from_db_value() method. That is, don't even try to inject ourselves into the psycopg2 machinery, but just let django handle it.

I think there's still value in allowing an override of the encoder kwargs, as that can be done on a per-field basis, as it's still done by django.

Given the difficulty of decoding, I'd be inclined to suggest that someone who wants to do it can subclass JSONField to add their own from_db_value() method which can further decode parts of the JSON it expects to be strings in a particular format. For custom encoding they would currently also need to override get_prep_value to wrap the object in psycopg2.extras.JSON with a different encoder passed.

If this becomes a common request we could easily add a utility to deal with it, personally right now I'm not completely sure it's a Good Idea ™️

Yes, that was my other approach. Perhaps a documented example of how to do that, too.

Docs added now, should be ready for review. I'll squash together after integrating feedback from a detailed review.

Could you add a mention the 1.9 release notes (contrib.postgres section)? I only had minor edits to the docs: http://dpaste.com/21RWTQ3

When I submit an invalid value in the admin, the value that's displayed on the subsequent form rendering is escaped with backslashes: "{\"breed\": \"collie\"". Is it a bug?

I also updated Jenkins to PG 9.4.

What is the status of this PR? Has it been abandoned? #24604 says it needs improvement.

Any to allow this field to work with non Postgres DBs where it uses a text field when the the DB doesn't support JSONB? This would be along the lines of the UUID that uses the native type where available.

Needs some tweaks, have been busy sorry. Will definitely get done by end of DjangoConEU.

As for non-PG dbs, all you'd get is the ability to store a JSON string and load it again, no querying operations would be practically fast to be useful, and would require extensive SQL scripting in each database. Consequently, you may as well do this yourself as a TextField (and multiple third party solutions do this).

Even though the querying operations would be slow / non-existent, having the ability to quickly run unit tests using SQLite on my local dev box is still helpful.

@mjtamlyn No worries! Was just discussing an internal implementation of the same with a coworker and came across this PR.

Merged in 33ea472