Sparsebundle support, and new builds of SS & B2

[mabinogian]

Until recently, I hadn't realized that support for sparsebundle disk images seems to have been included in the recommended SheepShaver builds on Mac hosts for quite a while now. Since the setup manual doesn't mention them (which could seem like an implicit recommendation against them), I'm curious if they have "cons" that outweigh their "pros."

As I understand it, a clear advantage wIth Time Machine is that backups of sparsebundles can often be far more efficient than those of regular disk images, particularly for boot images. The only negative I've noticed so far is that some programs see the underlying folder structure of the bundle, rather than the "virtual file" that the Finder shows. (This is similar to what happens with Applications.) So maybe such programs could risk corrupting that structure.

Are there other trade-offs to consider -- and are sparsebundles generally advisable?

[vasi]

I may be biased, but I continue to prefer sparsebundles on macOS and Linux hosts (I don't use Windows). There are a couple of potential downsides:

• Depending on what tools you like, it can be harder to create sparsebundles. On Linux, I basically have to use my "rhfs" tool, there's no other reasonable way to do it.
• Many other emulators and tools don't support sparsebundles. So for disks that I share between BasiliskII and Mini vMac, I'm stuck using flat img files.

[adespoton]

I tend to use sparsebundles with SS and B2, but I've noticed that in recent years, it doesn't appear to make much of a difference because:
1. backups are treating the sparsebundle as a single file, so any change to a slice triggers backing up of the entire bundle
2. sometime in recent OS versions, any update to a sparsebundle has resulted in the bundle reporting its max file size. However, with APFS, it still ignores the slack space on disk. I did some experiments and it appears that a regular DMG is treated the same way -- APFS discards its slack space, reports the full file size, and reports the used space as "space on disk".

I've got a method I use for mounting sparse bundles on other emulators, which I no longer use since I began encountering the above issues -- it may still be useful on older versions of OS X:

I write a script that attaches the sparsebundle image without mounting it on the host, and then launches the emulator while referencing the attached image path as the "disk" to launch. This works for Mini vMac and QEMU. I used to use /dev/rdisk to do this, but ran into issues where the lack of buffering caused data loss, so switched to the slower /dev/disk instead.

My script stays running while the emulator is running, and when it detects the emulator process has terminated, it detaches the disk image.

Anyway, as I said, I don't use this anymore because it appears the OS isn't expecting sparse bundles and so behaves in unexpected ways, without providing any measurable benefit if they're used.

[mabinogian]

adespoton wrote:
1. backups are treating the sparsebundle as a single file, so any change to a slice triggers backing up of the entire bundle

Oof. Does Sonoma's Time Machine backup the entire sparsebundle when only a slice changes? Not sure how to tell.

[adespoton]

adespoton wrote:
1. backups are treating the sparsebundle as a single file, so any change to a slice triggers backing up of the entire bundle

Oof. Does Sonoma's Time Machine backup the entire sparsebundle when only a slice changes? Not sure how to tell.

Yeah; my backups are using Time Machine. No idea why it's being so idiotic about the backups when APFS differencing seems to work just fine.

The easy way to tell for me is that I've got sparsebundles that are multiple gigabytes in size; if I do my backups, modify a single byte in that sparsebundle and then do another backup, multiple gigabytes get backed up again, instead of 1MB (my current slice size).

[Kakaze]

Sorry for the thread necromancy but I can't seem to find any solutions through search and this seems to be the best place to ask...

I'm trying to do a new install of 9.0.4 in Sheepshaver 2.5 and I want to do it with a sparsebundle. No matter what size or format I make the bundle the Mac OS installer says it's locked when it tries to format it.

I am on Tahoe beta if that makes a difference. Other than that I can't figure out why it's not working.

Help is appreciated.

[vasi]

Sorry, I don't have any Macs that can run Tahoe. Are you able to reproduce the problem on any other operating systems?

[Kakaze]

So I tried it on my other machine running Sequoia and it worked fine. I noticed when I was creating it that the options were slightly different: Sequoia offers sparse bundle disk image, sparse disk image, read/write disk image, and DVD/CD master. Tahoe's options were: Sparse Bundle Image (UDSB), RAW Image, Apple Sparse Image (ASIF), and DVD/CD master.

On Sequoia sparse bundle disk image is what worked. I moved the file over to the Tahoe machine and it worked as well. On Tahoe I tried both types of image; the Sparse Bundle Image (UDSB), and the Apple Sparse Image (ASIF) and the former stated it was locked and the latter was just ignored by Sheepshaver.

I also noticed, interestingly enough, that Tahoe named them disk.sparsebundle while Sequoia named them disk.dmg.sparsebundle. I tried to rename one of the Tahoe ones to be like Sequoa and that didn't make a difference. When I Show Package Contents on each sparse bundle, the Tahoe one has an extra folder called mapped. Deleting this folder did not make a difference. So it looks like something in Tahoe has changed the way sparse bundles are created but I don't know enough to say what.

[vasi]

Ah, I think it's just a new format that Tahoe is using by default. You could instead try in the terminal, since that tends to be a more stable interface: "hdiutil create -type SPARSEBUNDLE -size 20g mydisk.sparsebundle" . Hopefully what that creates is compatible with SheepShaper.

[Kakaze]

That worked perfectly! Thanks!

[adespoton]

Apple Sparse Images have the APFS partition baked into the image format IIUC, so these aren't going to work in an emulator of anything below High Sierra. And of course, image handling would have to be added to any emulator using them.

You can, however, create such an image and mount it prior to use, unmounting it when the emulator quits. Then you can write a sparse bundle onto it and get the best of both worlds.

I actually used to do this sort of thing for emulators that didn't support sparse bundles (mount the bundle, point to the logical device as the data source, and go), but somewhere along the way this behaviour broke for most macOS/emulator combinations, so I don't do it anymore.

[emendelson]

Details of the new format here:

https://www.computerworld.com/article/4 ... -asif.html

[adespoton]

Details of the new format here:

https://www.computerworld.com/article/4 ... -asif.html

I'm not sure that article is completely correct; the Eclectic Light article it quotes goes into much more detail. But I'm still a bit fuzzy about what's supported, both as far as host filesystem and guest partitions. Has anyone looked at the API docs or played around with the format in Tahoe to see what its limitations are? Is the image format portable to other host OSes? I'm thinking not, as I believe it is tied to APFS on the host system (which may mean you can dump whatever partitions you want inside it).

[vasi]

I'm honestly kinda annoyed they gave it the same name as the original sparsebundle format! ASIF is an interesting format, but has nothing to do with the format we support in SheepShaver. And honestly supporting something like qcow2 would be much easier than this under-documented, proprietary thing.

Maybe we should add a note to the top of the thread that ASIF is not the same thing as the sparsebundle format we support?

[adespoton]

I'm honestly kinda annoyed they gave it the same name as the original sparsebundle format! ASIF is an interesting format, but has nothing to do with the format we support in SheepShaver. And honestly supporting something like qcow2 would be much easier than this under-documented, proprietary thing.

Maybe we should add a note to the top of the thread that ASIF is not the same thing as the sparsebundle format we support?

Might be worthwhile. I'm also wondering if it would be worth it to add qcow2 support to BII/SheepShaver/MAME/DingusPPC....

Probably not, as qcow2, while being a generic improvement, still has some significant performance issues in specific conditions, much like traditional sparsebundles.

[vasi]

Might be worthwhile. I'm also wondering if it would be worth it to add qcow2 support to BII/SheepShaver/MAME/DingusPPC....

I looked into this when I build sparsebundle support. I don't really mind qcow2 performance, it wouldn't likely be the limiting factor in an emulator. But qcow2 is significantly more complex than our other virtual disk formats, so I thought it'd only be worthwhile if we could reuse a library, instead of implementing it ourselves. There was some talk about splitting out a "libqblock" to abstract out all of qemu's block-device support, but that never happened. I also experimented with writing it in Rust, but integrating that into the macemu build would be a real pain. There's some other libraries out there, but they also tend to be read-only, or otherwise immature, or in some other language that isn't C-compatible.

If you find any good libraries though, let me know! I'd be happy to add support if it looks reasonable, it'd be nice to be able to share disks with qemu.

[Kakaze]

I'm honestly kinda annoyed they gave it the same name as the original sparsebundle format! ASIF is an interesting format, but has nothing to do with the format we support in SheepShaver. And honestly supporting something like qcow2 would be much easier than this under-documented, proprietary thing.

Maybe we should add a note to the top of the thread that ASIF is not the same thing as the sparsebundle format we support?

They do separate the two types of sparse bundles in the disk utility. One is called Sparse Bundle Image and the other, the ASIF, is called Apple Sparse Image. It looks like they modified the way regular sparse bundles are created in Tahoe that makes them incompatible with Sheepshaver.