Setting RAM size more than 512 MB for SheepShaver may indeed cause problems. It is also unnecessary. For many users 64MB will be sufficient, 128MB is fine, 256MB is plenty, and 512MB is probably more than you will need.
SheepShaver does things in memory on the host that it shouldn't and newer host OSes may be more critical. The latest OSX/macOS versions fill up all available memory and do not release memory when you quit applications. Memory is only released when needed. However, 512MB for SheepShaver works with Sierra on my machine. If you want to use 512MB, you can try to restart your host machine to clear memory and try again.