Yep, thats a new feature in macOS Sierra.
In short, three things are needed: BII app, Macintosh ROM, bootable disk image.
The GUI is nice to have, but not obligatory. (I´m using it too, usually.)
Make all settings in the GUI and click OK at the error message.
Below is an example of my settings, a BasiliskII folder resides in Downloads.
RAM and key codes may be set as you wish.
Next double click BsiliskII.app. If your settings are correct, the emulator will boot.
The DL you pointed to does not provide ROM and boot disk.
Feel free to try with this disk image http://macintoshgarden.org/apps/macos-753-emulators
and the ROM from RedundantRobot. Look for the file Performa.ROM.
Names and file locations do not matter much, as long as all paths for BII are specified correctly.
Apple fortified security settings in 10.12. You may have to allow installation from any source in system settings.
Edit: After running the BasiliskII.app, the "Start" button in the BII GUI is working.