We have further investigated this here and the sortorder property is based on the type of node that is added (folder/file) Not on the name itself. The directory treeview is using FindFirst..FindNext statements that return folders/files in a specific order based on the system. Windows might return them in alphabetic order while Linux system returns them in a non-alphabetic order. It really depends on the settings of the machine itself.
It's unclear exactly how the TOpenDialog functions under Linux for sorting purposes. We still need to allocate time for further investigation. The previous months have been very time consuming as we are working on the TMS WEB Core. Please note that this issue might possibly be a shortcoming of the FMXLinux framework, have you already been in touch with the creators of FMXLinux for a possible solution?
Unfortunately there is little interest in FMXLinux and therefore we do not actively support it. The latest FMXLinux has been downloaded and tested and breaks the TMS FMX UI Pack at compile time, so unfortunately there will be no Linux support. We were unaware of sorting issue under macOS, we'll investigate this as soon as possible.