I wrote a doc on 68K A-trap
https://github.com/cebix/macemu/wiki/ROM-Analysis-in-Basilisk-II-Emulation#mac-rom-in-real-macintosh-hardware. In a nutshell, it just invent 68K instruction in software manner. But I'm not sure what your system extension tries to do.
For break point, I sent a PR (
https://github.com/cebix/macemu/commit/ ... 8d664ec6e3) that allow setting an emulation break point even starting in loading ROM. So it should fit your purpose to figure out your system extension at the start up.
Please note that the debugger of target platform compiled code (in your case VS debugger or GDB in Linux) is not useful at all for debugging 68K application in emulation. You must know 68K assembly and use BII emulator break point, instead.
In any case, it is difficult to debug Mac application written in 68K assembly. Good luck.