Parent ID field not set when deserializing

Hi,

I am trying to "abuse" serialization and deserialization to achieve database export and import functionality. This concerns exporting entities with three levels of sub-entities (embedded/proxies)
I have two questions:
  1. Initially, the embedded entities were not included due to lazy-loading.  I worked around this by reading something from each proxy to a dummy value.  Not very elegant as I now get several "value assigned to ... never used" warnings.  Is there a procedure to force loading of all lazy-loading proxies?
  2. Using TDataSnapJsonDeserializer, I am reading the JSON into a TEntity.  However, when using ObjMgr.Replicate(MyEntity), I get an exception that the Parent ID of the embedded entity fails the NOT NULL Constraint.  Clearly, the Parent ID value (GUID) is in the JSON.  Any thought what causes this and how to solve it?  I am at a loss (being a newbie to JSON)....
Thanks and regards,
Mark

Hello Mark,

1. You don't need to assign the value to any variable, this will remove the hint.
2. I don't know what to suggest in this case, I would have to have more details to guess what's going on. However, I'd suggest you use the serializer used by XData, not the TDataSnap one which is a legacy one. Here is how you could use it:



Serializer := TXDataJsonServerSerializer.Create(TXDataAureliusModel.Default);
try
  Json := Serializer.Write(MyEntity);
  // or
  Serializer.Write(MyEntity, ContentStream);
finally
  Serializer.Free;
end;

Wagner,

I tried to change my code to the following (replaced irrelevant code with ...):

I have the following TEntity structure:

TSchematic
 - TSettings (1-1)

 - TFormationList (1-M) -> TFormation
 - TBarrierList (1-M) -> TBarrier
 - TSectionList (1-M) -> TSection, each TSection has the following proxies:

   - TExternalList (1-M) -> TExternal
   - TInternalList (1-M) -> TInternal
   - TAncillaryList (1-M) -> TAncillary 

I changed the code to use the TXDataJsonServerSerializer:
 
var
  LFileName, LStrJson: string;
  LCurID: TGUID;
  LSchematic: TSchematic;
  LSerializer: TXDataJsonServerSerializer;
  LJsonFile: TextFile;
  ii, jj: Integer;
begin
  ...
  if sdExport.Execute then
  begin
    LFileName := sdExport.FileName;

    LCurID := adsSchematicsView.Current<TSchematicView>.Id;  // DataSet displaying a list obtained from a View (Model 'Views').
    LSchematic := omDatabase.ObjManager.Find<TSchematic>(LCurID); // Retrieve full TSchematic entity (Model 'Database').

    if assigned(LSchematic) then
    begin

      // Dummy reads to force loading proxies
      ii := LSchematic.Settings.Casing_Line; // Integer property of TSettings
      ii := LSchematic.FormationList.Count;
      ii := LSchematic.BarrierList.Count;
      for ii := 0 to LSchematic.SectionList.Count - 1 do
      begin
        jj := LSchematic.SectionList[ii].ExternalList.Count;
        jj := LSchematic.SectionList[ii].InternalList.Count;
        jj := LSchematic.SectionList[ii].AncillaryList.Count;
      end;

      LSerializer := TXDataJsonServerSerializer.Create(TXDataAureliusModel.Default);
      LStrJson := LSerializer.Write(LSchematic);
      try
        AssignFile(LJsonFile, LFileName);
        Rewrite(LJsonFile);
        WriteLn(LJsonFile, LStrJson);
        CloseFile(LJsonFile);
        messageSkin('Schematic successfully exported.', mtInformation, [mbOK], 0);
      finally
        LSerializer.Free;
      end;
    end
    else
      raise Exception.Create('Unable to obtain Schematic for export!');
  end;

Using this code, I get an exception: "Could not find JSON converter for type "Proxy<uSD_WellSchematicEntitites.TSettings>""
TSettings is the first proxy in the main entity class I am trying to export (1-1 relation).

Previously, I used:

  LSerializer := TDataSnapJsonSerializer.Create;
  LJsonValue := LSerializer.ToJson(LSchematic);

Writing LJSonValue.ToString to the TextFile worked fine (albeit that the resulting text file was not nicely structure; it was just a single string).

On the import side, I have tried to use the TXDataJsonClientDeserializer but I get nowhere either.  The constructor requires a IJsonSolver as parameter, but I cannot find any documentation on this.  I tried a nil value, but this gets me nowhere either:

var
  LFileName, LStrJson, LStrTemp: string;
  LJsonFile: TextFile;
  LJsonValue: TJsonValue;
  LDeserializer: TXDataJsonClientDeserializer;
  LSchematic: TSchematic;
  LSchematicView: TSchematicView;
begin
  ...
  if odImport.Execute then
  begin
    LFileName := odImport.FileName;

    // Read JSON file
    LStrJson := '';
    AssignFile(LJsonFile, LFileName);
    Reset(LJsonFile);
    while not eof(LJsonFile) do
    begin
      ReadLn(LJsonFile, LStrTemp);
      LStrJson := LStrJson + LStrTemp;
    end;
    CloseFile(LJsonFile);

    LJsonValue := TJSONObject.ParseJSONValue(LStrJson);
    LDeserializer := TXDataJsonClientDeserializer.Create(TXDataAureliusModel.Default, nil); // nil acceptable????
    try
      LSchematic := LDeserializer.Read<TSchematic>(LJsonValue.ToJson);
      omDatabase.ObjManager.SaveOrUpdate(LSchematic);

      // Add imported schematic to view data...
      LSchematicView := omView.ObjManager.Find<TSchematicView>(LSchematic.Id);
      FSchematicList.Add(LSchematicView);
      adsSchematicsView.Refresh;
    finally
      // LSchematic.Free; // Should not be required.  After SaveOrUpdate, ObjectManager owns the entity...
      LDeserializer.Free;
      LJsonValue.Free;
    end;
  end;

With this code I get an exception: "Property "$type" does not refer to a know property in type "uSD_WellSchematicEntitites.TSchematic".

Using the DataSnapDeserializer I have the following:

    LJsonValue := TJSONObject.ParseJSONValue(LStrJson);
    LDeserializer := TDataSnapJsonDeserializer.Create;
    try
      LSchematic := LDeserializer.FromJson<TSchematic>(LJsonValue); // Transient object, must be destroyed!
      omDatabase.ObjManager.SaveOrUpdate(LSchematic);

      // Add imported schematic to view data...
      LSchematicView := omView.ObjManager.Find<TSchematicView>(LSchematic.Id);
      FSchematicList.Add(LSchematicView);
      adsSchematicsView.Refresh;

      showMessage('Import succeeded');
    finally
      // LSchematic.Free;
      LDeserializer.Free;
      LJsonValue.Free;
      Screen.Cursor := crDefault;
    end;
  end;

This code "works" (showMessage executed) without an exception occuring but the entity (not even the top-level TSchematic) is imported.  As a result, the View data shows a blank record (understandable, as failure to retrieve LSchematicView (nil) is adding a nil object to FSchematicList).

In essence I am trying to write a TEntity to a file (JSon format) and later read it (JSon format) to re-create a TEntity to save (or update) in the database.
Surely this is not that complex?  Hopefully you can tell me where I am going wrong.

PS: If you give any example code, can you please include the uses clause?  This will save a lot of time searching for the right units. ;-)

Regards,
Mark
The object tree you are trying to serialize are entities that come from different XData models? The origin of the error is that the serializer is trying to serialize the entity as a regular Delphi object, not Aurelius entity, probably because it can't find the object in the model.
Make sure you create the serialize with the proper model that holds all the entities you want to serialize/deserialize.
Maybe the problem with the DataSnap serializer has the same origin, but I can't be sure.

Hi Wagner,


The trick for export was to change/add the following;

      LSerializer := TXDataJsonServerSerializer.Create(TXDataAureliusModel.Get('Database'));
      LSerializer.SetExpandLevel(3);

This appears to export OK (indeed working with two models ('Database' and 'Views').  I just tried the SetExpandLevel and then all sub-entities (proxies) are included.

However, I am still struggling with the import. I now have:

var
  LFileName, LStrJSon, LStrTemp: string;
  LJsonFile: TextFile;
  LDeserializer: TXDataJsonServerDeserializer;
  LSchematic: TSchematic;
  LSchematicView: TSchematicView;
...

if odImport.Execute then
  begin
    LFileName := odImport.FileName;

    // Read JSON file
    LStrJSon := '';
    AssignFile(LJsonFile, LFileName);
    Reset(LJsonFile);
    while not eof(LJsonFile) do
    begin
      ReadLn(LJsonFile, LStrTemp);
      LStrJSon := LStrJSon + LStrTemp;
    end;
    CloseFile(LJsonFile);

    LDeserializer := TXDataJsonServerDeserializer.Create(TXDataAureliusModel.Get('Database'), nil);
    try
      LSchematic := LDeserializer.Read<TSchematic>(LStrJSon);
      omDatabase.ObjManager.SaveOrUpdate(LSchematic);

      // Add imported schematic to view data...
      LSchematicView := omView.ObjManager.Find<TSchematicView>(LSchematic.Id);
      if assigned(LSchematicView) then
      begin
        FSchematicList.Add(LSchematicView);
        adsSchematicsView.Refresh;
        showMessage('Import succeeded');
      end
      else
        showMessage('Import did not succeed');
    finally
      LDeserializer.Free;
    end;
  end;

A few comments;
  • I tried both TXDataJsonServerDeserializer and TXDataJsonClientDeserializer but it made no difference? What is actually the difference between those two?
  • The program reaches 'Import did not succeed' without any exception, but nothing is saved in the database.  As such LSchematicView is not found either....
  • I am now using nil for the IJSonReferenceSolver parameter in TXDataJsonServerDeserializer.Create. I wonder if this is acceptable but I cannot find any documentation.  If not, how to correctly set the parameter?
Thanks,
Mark

Forgot to mention; I am attempting to create/read the JSon from the same application / database.

Essentially it's meant to be an export/import file for data exchange between databases...
Just some predefined values for properties. Actually the only difference right now references are solved (values of type "@xdata.ref": "Customers(1)")
The server will try to create a proxy for those values, while the client will try to load it right away.
Those values are loaded exactly by the solver.

Hard to tell with such relatively complex code without debugging. But posting here the JSON value you are trying to deserialize and the involved mapped classes might help.

Yes, you can use those. As said, the solver is only used to solve values of type "@xdata.ref" or "@xdata.proxy", which you don't have because you have expanded everything you need.

Hi Wagner,


I have run into another strange problem as well (trying to copy/clone entities), so if you allow me, I will write a small sample application (stripped of all 3rd party components) to address all these issues.  I'll send it to you directly (email), including a small SQLite DB with some sample data.

It would be nice if you could review that and let me know where I am (or what is) going wrong.  Please allow me a few days to build the sample application.

Regards,
Mark

Ok, great.

OK, I thought I had it solved after our direct email, but now I run into another problem and I just cannot seem to find the cause. Unfortunately, there's no really clear documentation on the (de)serializer classes....

The simple example (Parent-Child-Grandchild entities) I sent you is exported like this (irrelevant code omitted:

  LParent := adsParent.Current<TParent>;
  if assigned(LParent) then
  begin
    // "Touch" all child/grandchild list to ensure loaded (lazy)
    for ii := 0 to LParent.ChildList.Count - 1 do
      jj := LParent.ChildList[ii].GrandChildList.Count;

    LSerializer := TXDataJsonServerSerializer.Create(TXDataAureliusModel.Default);
   try
    LSerializer.SetExpandLevel(3);
    LStrJSon := LSerializer.Write(LParent);
    // Write LStrJson to (text) file
   finally
      LSerializer.Free;
   end;

This produces the following Json (note yellow highlight):
image

Now doing the same for my real-life application (complex 3-level) model;

    LSchematic := omDatabase.ObjManager.Find<TSchematic>(LCurID);
    if assigned(LSchematic) then
    begin
      // "Touch" all child/grandchild entities/collections to ensure loaded (lazy)
      ii := LSchematic.Settings.Casing_Line;
      ii := LSchematic.Settings.BreakList.Count;
      ii := LSchematic.FormationList.Count;
      ii := LSchematic.BarrierList.Count;
      for ii := 0 to LSchematic.SectionList.Count - 1 do
      begin
        jj := LSchematic.SectionList[ii].ExternalList.Count;
        jj := LSchematic.SectionList[ii].InternalList.Count;
        jj := LSchematic.SectionList[ii].AncillaryList.Count;
      end;

      LSerializer := TXDataJsonServerSerializer.Create(TXDataAureliusModel.Get('Database'));
      try
        LSerializer.SetExpandLevel(3);
        LStrJSon := LSerializer.Write(LSchematic);
        // Write LStrJson to file....
      finally
        LSerializer.Free;
      end;

This produces the following JSon:

The only difference in the code I can see is that the serializer for the simple model uses TXDataAureliusModel.Default whereas the more complex uses TXDataAureliusModel.Get('Database') (yet the in the Json all is still XData.Default...).

Using the same TXDataJsonServerDeserializer and trying to read the Json (import into application/database) the simple model works fine but in the more complex model this fails as I get 'NOT NULL contstraint failed' for all child entities (as to be expected I suppose, given that there's not Parent ID value in the Json parts).

How to force writing the Parent ID values (TGUID)?? Or is it something else that I am doing wrong?

Regards,
Mark

Another difference is that in second case the reference to the parent seems to be at a deeper level. There is a depth level limit for the serialization. Also, it seems you are not touching the specific Settings property so it might not have been loaded, thus it's serialized as a reference.

OK, final attempt to solve this:

Initially, I was using two models (Database and View). As I no longer use a View (database), I decided to remove all Model attributes (i.e. everything now under the Default Model). As a result, all code is now exactly the same (pseudo-code):

  Touch all entities to ensure loaded
  Create Serializer ->  TXDataJsonServerSerializer.Create(**TXDataAureliusModel.Default**);
  try
    Set Serializer ExpandLevel to 3;
    Serializer write top-level Entitity to Json string
    Write Json string to file
  finally
    Free Serializer
  end;

I also tried the TXDataJsonClientSerializer but this made no difference either.

As you can see in below image of the Data Model, there are more Entities declared in the Data Model, but it also has 3 levels (just like the simple example that does work):

image

I just cannot understand why the same code - using the same serializer class - outputs the parent reference as a reference to the parent entity in the Json (e.g. $ref: 1) whereas the other outputs the parent reference as a proxy (e.g. Schematic@xdata.proxy: "...").

In other words; how can the same code (albeit on different entities) produce different results?
Is there a property/setting I am missing?

Things that can influence:

  • the depth level
  • the fact the proxy is loaded or not
  • if the object is a list of not
  • if the object is saved in the database (in the manager) or not
  • if the object is nil or not
  • if the list is empty or not