When I setted the PageMode := False and wrapped beginupdate and endupdate like the following code, then the issue has been resolved.
Thank you.
Grid.BeginUpdate;
try
Grid.PageMode := False;
Grid.SaveWithHTML := False;
Grid.SaveToCSV(LFileName, SaveDialog.FilterIndex = 1);
Grid.PageMode := True;
finally
Grid.EndUpdate;
end;