Serious performance problem due to lazy loading fields

BIZ TMS Aurelius
1

I noticed the following problem:
I load about 15000 records from a mysql table that has has 71 fields per record.
Loading this table with an TFDQuery and copying to TFDMemTable takes about a 0.3sec in a slow computer.
Loading this table with Taurelius in a list (no lazy loading fields) and inserting the entities from list to a TFDmemTable with RTTI takes 2sec
Loading this table with Taurelius in a list (with 3 lazy loading fields) and inserting the entities from list to a TFDmemTable with RTTI takes 214 seconds....
The code with Aurelius is the same, I only add/remove the TColumnProp.Lazy in the class definition for 3 Tblob entries.

The following takes 9 seconds:
FDMemTable1.CopyDataSet(AureliusDataset1, [ coStructure, coRestart, coAppend]);

Should I remove Lazyloading? Should I have double classes for the same table (with eager and lazy loading) ? Is it possible to disable Lazyloading if needed?

My code follows:

var n,i,m:integer; r:TFDDatSRow; items:tlist<tapitems>; 
    ctx:TRttiContext; objType:TRttiType; fls:array of TRttiProperty; vvv:Tvalue;  

items:=manager.Find<tapitems>.List; 
AureliusDataset1.CreateSelfField:=false;
AureliusDataset1.SetSourceList(items,false);
AureliusDataset1.Open;
FDMemTable1.Close;
FDMemTable1.FieldDefs.Clear;
FDMemTable1.FieldDefs.Assign(AureliusDataset1.FieldDefs);
FDMemTable1.CreateDataSet;
m:=FDMemTable1.FieldDefs.Count; 
setlength(fls,m);
ctx:=TRttiContext.Create; 
objType:=ctx.GetType(tapitems); 
try
for n:=0 to m-1 do 
fls[n]:=objtype.GetProperty(FDMemTable1.FieldDefs.Items[n].Name);
for n:=0 to items.Count-1 do begin  
FDMemTable1.Append;
for i:=0 to m-1 do begin
vvv:=fls[i].GetValue(items[n]);
if vvv.Kind=tkRecord then 
FDMemTable1.Fields[i].AsString:=vvv.AsType<Tblob>.AsString
else 
FDMemTable1.Fields[i].AssignValue(vvv.AsVarRec);  
end;
FDMemTable1.post;
end;
finally 
objType.Free; 
ctx.Free; 
end;

With copydataset:

items:=manager.Find<tapitems>.List; 
AureliusDataset1.CreateSelfField:=false;
AureliusDataset1.SetSourceList(items,false);
AureliusDataset1.Open;
FDMemTable1.Close;
FDMemTable1.CopyDataSet(AureliusDataset1, [ coStructure, coRestart, coAppend]);
2

Yes, you should disable lazy loading, as it doesn't make sense in this scenario.
You don't have to modify your mapping, you can set any association as eager loading in a specific query, using CreateAlias method and passing TFetchMode.Eager as the loading mode for that specific association:

items:=manager.Find<tapitems>
  .CreateAlias('MyAssociation', 'm', TFetchMode.Eager)
  .List;
1 Like
3

Thank you,
Reading the documentation, I understood that it was only for associations, not for columns too.
And it does not work in my tests

1 Like
4

This topic was automatically closed 60 minutes after the last reply. New replies are no longer allowed.

5
6

Yes, it's only for associations, but you mentioned that it was lazy-loaded associations that were causing the issue. Do you have a sample project to reproduce the issue?

7

Drop in form an aureliusmanager, aureliusdataset, aureliusconnect, TFDconnection, Tfdmemtable
Use a table with the class tapitems that has at least one tblob property and run the above code.
To be honest, after yesterday's finding that Tblob and Tarray<byte> have different results in Xdata server, I did not managed to test it with Tarray<byte>

8

That's new information. Indeed, for blobs, it's lazy-loaded and there is currently no way to force eager loading of blobs.

9

Probably, I am a bit idiot, but I found Tblob as a usefull class for transfering data especially to Tstreams.
I have not understood so clearly from the documentation that Tblob is ALWAYS lazy-loading
So, the createalias in Tarray<byte> works with Eager? What other blob like type of data can be used with eager loading?
I noticed also today that "array of variant" does not work as parameter in service implementation in Xdata but Tarray<variant> works

10

The lazy-loading mechanism of a TBlob is defined in the model. When you add the [Column] attribute to your blob field/property, you can choose to add TColumnProp.Lazy flag to the column properties, or leave it without it. This will define if the blob will be loaded using lazy-loading or not:

  // Eager-loading blob
  [Column('BLOB1', [])]
  FBlob1: TBlob;

  // Lazy-loading blob
  [Column('BLOB2', [TColumnProp.Lazy])]
  FBlob2: TBlob;

What I meant is that once the blob is defined as lazy-loading in the mapping, you can't change it from the criteria. So you can map your blob as not loading lazily, and it will behave like that.

CreateAlias is only for associations, no other types.

11

Sorry, you have right, I should not use lazy to have them loaded. In Xdata, they are always lazy.
I will check for Tarray how they behave