About the first question, stack overflow exceptions are normally fatal, like out of memory exceptions. In most cases you can't recover from them because the stack might have become corrupt. In .NET they don't even let you catch stack overflow exceptions: https://msdn.microsoft.com/en-us/library/system.stackoverflowexception.aspx)
Now, this being said, AdvGridExcelIO shouldn't throw an stack overflow in password protected files (but it can't open them either). I made a small test here:
And it seems to work fine (even if not opening the file). It might be related to an specific file: If you try the same sample app with your file, does it throw an stack overflow? If it does, can you send me the file to adrian@tmssoftware.com so I can study it further?
About the second question, sadly AdvGridExcelIO is based in FlexCel 3, which doesn't have support for opening encrypted xls files at all. You might be able to open the file with FlexCel 6 (and the tms grid filters instead of AdvGridExcelIO: http://www.tmssoftware.com/site/advgridfilters.asp ) but it depends in the file.
A small note: FlexCel 6 supports all 3 types of encryption in xlsx files, and also 3 different types of encryption in xls files. Now, the latest Excel versions default to a "plugin" encryption system where the file can specify any encryption algorithm registered in your machine. We don't support that because it would mean either reading the encryption algorithms registered in your machine (and we try to avoid this kind of stuff), or supporting every possible encryption algorithm (which isn't feasible).
So depending on the Excel version used to create the encrypted xls file, FlexCel 6 will be able to open it or not. FlexCel 6 should open all types of encrypted xlsx files.