We tested this now on Windows 11 starting from the TAdvSmoothRotaryMenu demo with the settings of the TAdvSmoothRotaryMenuDialog adapted to what you have in your DFM but could not see the border effect you see.
Perhaps this is due to a specific Windows 11 setting you did?
Do you have another machine where you can test this?