I've testet fetching 19000 records with 147 projections.
I'm using Aurelius-Dataset with PageSize 2000.
Aurelius is connected to a postgres using RemoteDB (and firedac on Server-Side).
This took up to 230 seconds, which I think is not quite fast.
Taking a look on how you are creating CriteriaResultList, I've just changed
this:
function TDriverResultSetAdapter<T>.GetFieldValue(FieldName: string): Variant;
var
hField: TField;
begin
hField := FDataset.FieldByName(FieldName);
if hField.IsNull then
Result := Variants.Null
else
Result := hField.Value;
// old
// Result := GetFieldValue(GetFieldIndex(FieldName));
end;
By fetching Field out of Dataset-Fields once it "only" took 215 seconds anymore.
implementing this by our own method, dealing it like that:
Dataset.First;
while not eof do
begin
props := TObjectList<TCriteriaResultProp>.Create(True);
try
for i := Low(fieldNames) to High(fieldNames) do
begin
if userAliases[i] <> '' then
propName := userAliases[i]
else
propName := fieldNames[i];
fieldValue := aDataset.FindField(fieldNames[i]).AsVariant; // shortened, cause it's faster
ConvertFieldValue(fieldValue, fieldTypes[i]); // shortened, cause it's faster props.Add(TCriteriaResultProp.Create(propName, fieldTypes[i], fieldValue));
end;
except
props.Free;
raise;
end;
Result.Add(TCriteriaResult.Create(props));
aDataset.Next;
end;
this own implementation only tooks 147 seconds.
All times are taken for completion of our overall data export process.
Some different times, I took from this process:
all with 147 projections
your code:
fetching 750 records in 1125 ms
build criteria out of fetched data 6487 ms
my own method:
fetching 1000 records in 1703 ms
build criteria out of fetched data 431 ms
My method is using an TXDataset executing SQL fetched from criteria-execution.