Switch from pickled blobs to JSON data

[dsblank]

This PR converts the database interface to use JSON data rather than the pickled blobs used since the early days.

1. Uses a new abstraction in the database: db.serializer
   a. abstracts data column name
   b. contains serialize/unserialize functions
2. Updates database format to 21
3. The conversion from 20 to 21 reads pickled blobs, and writes JSON data.
   a. It does this by switching between serializers
4. New databases do not contain pickled blobs
5. Converted databases contain both fields

[Nick-Hall]

If we are moving from BLOBs to JSON then we should really use the new format. See PR #800.

The new format uses the to_json and from_json methods in the serialize module to build the json from the underlying classes. It comes with get_schema class methods which provide a JSON Schema that allow the validation that we already use in our unit tests.

The main benefit of the new format is that it is easier maintain and debug. Instead of lists we use dictionaries. So, for example, we refer to the field "parent_family_list" instead of field number 9.

Upgrades are no problem. We just read and write the raw data.

When I have more time I'll update you on discussion whilst you have been away.

[dsblank]

Oh, that sounds like a great idea! I'll take a look at the JSON format and switch to that. Should work even better with the SQL JSON_EXTRACT().

[Nick-Hall]

There are a few places where the new format is used, so we will get some bonus performance improvements.

Feel free to make changes to my existing code if you see a benefit.

You may also want to have a quick look at how we serialize GrampsType. Enough information is stored so that we can recreate the object, but I don't think that I chose to store all fields.

[dsblank]

Making some progress. Turns out, the serialized format had leaked into many other places, probably for speed. Probably good candidates for business logic.

[dsblank]

I added a to_dict() and from_dict() based on the to_json() and from_json(). I didn't know about the object hooks. Brilliant! That saves so much code.

[dsblank]

@Nick-Hall , I will probably need your assistance regarding the complete save/load of the to_json and from_json functions. I looked at your PR but as it touches 590 files, there is a lot there.

In this PR, I can now upgrade a database, and load the people views (except for name functions which I have to figure out).

[Nick-Hall]

@dsblank I have rebased PR #800 on the gramps51 branch. Only 25 files were actually changed.

You can also see the changes suggested by @prculley resulting from his testing and performance benchmarks.

[dsblank]

Thanks @Nick-Hall, that was very useful. I think that I will cherry pick some of the changes (like attribute name changes, elimination of private attributes).

You'll see that I did many of the same changes you made. But, one thing I found is that if we want to allow upgrades from previous versions, then we need to be able to read in blob_data, and write out json_data. I think my version has that covered.

I'll continue to make progress.

[Nick-Hall]

@dsblank Why are you removing the properties? The validation in the setters will no longer be called.

[dsblank]

@Nick-Hall , I thought that was what @prculley did for optimization, and I thought was needed. I can put those back :)

[Nick-Hall]

Perhaps we could consider a solution similar to that provided by the pickle __getstate__ and __setstate__ methods.

A get_state method in a base class could return a dictionary of public attributes by default. This could be overridden to add properties if required.

Aset_state method could write the values back. In the case of properties we could just set the corresponding private variable rather than calling the setter. The list to tuple conversion could also be done in this method.

I expect that only a handful of classes would need to override the default methods.

[dsblank]

CC: @Nick-Hall

[dsblank]

@Nick-Hall, you have any estimate on possible review on this PR (and the #1794 filter fixes)?

I have some available time coming up, and would like to start work on checking the addons for this next version (6.0 or 5.3).

[Nick-Hall]

@dsblank I'll make time this weekend, but may be able to start sooner.

[dsblank]

@Nick-Hall, if you'd like to meet over Google Meet or Zoom so that I can walk you (and others) through proposed changes, I'd be glad to.

[Nick-Hall]

This PR is looking good now.

I agree with you that the remaining serialize/unserialize code in the upgrade path should be left to another PR. Changes to the upgrades always require extra testing.

Would it be useful to log database upgrades? Perhaps a version table that could store the dates of database creation and any upgrades. I'm not suggesting adding to this PR though.

I'll convert my changes to add a create timestamp field to the primary objects so that they work with the new raw JSON format. Upgrades to the schema should be easier after this PR is merged. I'm not sure if we'll want to include it in the next release. We can discuss this later.

[dsblank]

Sounds good!

[dsblank]

Shall we merge this PR then?

[Nick-Hall]

I'm just about to do some final checks, followed by a rebase and merge now. It was getting late last night.

[stevenyoungs]

Good from my point of view

[dsblank]

Thank you all for the reviews and comments!

[Nick-Hall]

@dsblank Please don't merge PRs until I have done a final review. I was about to merge this, but noticed that the new gen.db.conversion_tools package was not listed in the setup.py and the two files it contains are not in the POTFILES.skip file.

In your merge a "Co-authored-by: stevenyoungs [email protected]" credit seems to have been lost. Was this intentional?

Otherwise, I appreciate that you squashed the commits and rebased to maintain a linear history according to our committing policies.

[dsblank]

Oh, sorry... how do you want to fix? I did not mean to lose Steve's credit.

[Nick-Hall]

I've created PR #1823 with the changes I was going to include.

Unfortunately, we can't go back and add the credit to the commit now. We can add a copyright line in a file if it hasn't already been done. I'll make sure that I mention Steve in the release announcement.