Monday, 9 January 2017

Sharing a Dropbox folder using Bootcamp and exFAT

My main laptop for the last few years has been a 15” Retina MacBook Pro. As I’m primarily a .NET developer, I use Bootcamp to run Windows 10, Visual Studio and SQL Server, but I also boot into macOS almost daily for things like audio recording and video editing using Logic Pro and Adobe Premiere. I’m also a huge fan of Dropbox – I use Evernote for text documents (notes, talk ideas and lyrics), and Dropbox for just about everything else.

My MacBook Pro has a 500Gb SSD, which sounds like a lot, but once you’ve got two operating systems it’s surprising how quickly you start running out of space – and one of my biggest frustrations was having two separate Dropbox folders. I have about 40Gb of stuff in Dropbox, which means a 40Gb Dropbox folder on my NTFS Windows partition, and another 4Gb Dropbox folder on the ExtFS macOS partition. And they store exactly the same filesso much so that Dropbox actually became my default sync mechanism for getting files from Windows into macOS and vice versa.

Being a huge nerd, I thought it would be fun to kick off 2017 by completely repaving my laptop, and I wondered if there was some way to store my Dropbox folder, and other big things like my music library, on a shared partition that both operating systems could use. It took a bit of tweaking but I think I’ve managed to get it working – here’s how. The secret sauce that made it all work is a relatively new file system called exFAT. The killer feature of exFAT here is that both Windows 10 and macOS can read and write it natively, but it doesn’t have any of the file size or volume size restrictions of older interoperable formats like FAT32.

Caveat: this is all completely unsupported. If you try any of this and you lose all your files, I’m not going to help you – and neither are Apple, Microsoft or Dropbox. If you’re happy using batch scripts to toggle file attributes and manually partitioning your OS drive, then this lot might help – but messing around with disk partitions and unsupported filesystems is a good way to end up with an unbootable laptop and a lot of bad mojo. So before we go any further,  back up everything. After all, Dropbox is a file sync tool – if some bizarre combination of macOS and exFAT results in it removing a load of files, the last thing I want is for Dropbox to conveniently remove the same files from all my other devices. I know it’s easy enough to revert to an older version of anything that’s in Dropbox, but I figured having a copy of it all on an external HD probably couldn’t hurt.

Next, work out how you want to partition your disk. In my case, I wanted to go for a ~150Gb partition for each operating system, and a 200Gb shared partition that both of them could see. Your mileage may vary, and – as I did – you might find you can’t get exactly what you wanted, but I managed to get pretty close. Now do a completely clean reinstall of macOS. Give it the entire drive and let it do its thing. Once you’re done, use the Boot Camp Assistant to set up and install Windows. When Boot Camp Assistant asks how you’d like to partition your drive, make the Windows partition equivalent to the total size of your desired Windows partition plus your shared partition – so in this case, I temporarily had a 150Gb macOS partition and a 350Gb Windows partition.

Screen Shot 2017-01-07 at 13.06.39

Once you’ve partitioned the disk, install Windows as usual. You should end up with a regular Boot Camp system, that you can boot into macOS or Windows by holding down the alt key during system startup or by using the Boot Camp options in Windows. Now here’s the fun part. Once you’ve installed Windows, but before you install any other apps or files, you’re going to use the Windows disk utility to shrink that Windows partition down to 150Gb and then create a shared exFAT partition in the remaining space. Right-click the BOOTCAMP (C:) partition, click Shrink Volume...

windows_disk_iutility

There’s some limit of the NTFS filesystem that means “you cannot shrink a volume beyond the point where any unmovable files are located”. Now I don’t know how much variance there is between Windows installations, but I couldn’t shrink my C: drive any smaller than 163Gb, despite the fact that the fresh Windows install only took up about 20Gb. For what I needed, this was close enough – if not, I’d have been looking for something that would let me defrag and reorganize the NTFS partition. Or reading up on how an SSD gets fragmented in the first place – I thought they were just big boxes of non-volatile solid-state memory?

Anyway. Next step was to format the new partition as exFAT, called it Shared, and then reboot into macOS and make sure it was visible. I poked around a bit, created some text documents and other files, rebooted back into Windows… After an hour or so this all seemed to be working quite nicely. Next step was to see how Dropbox coped with it. Rather than throw it at my 40Gb Dropbox account, I set up a new Dropbox account, installed Dropbox on both macOS and Windows and signed into to both of them using the new account.

This is where things got a bit weird. Here’s my best hypothesis as to what’s happening.

  1. macOS is storing some additional metadata for every file. On a regular ExtFS filesystem, the metadata is stored somewhere in the filesystem itself. On an exFAT partition, it’s stored in a tiny hidden file (apparently known as an AppleDouble) next to the original. Alongside foo.txt there’s now a 4k file called ._foo.txt. I don’t think this is a Dropbox thing, it just appears to be caused by running macOS on a non-ExtFS drive.
  2. The Dropbox agent on macOS sees these extra files, thinks that they’re important, and uploads them to Dropbox.
  3. The dot-underscore ._ files are then synced across all your other Dropbox devices – I can see them in the web interface as well as on my Windows partition.

If I wasn’t using Dropbox, the ._ files would still be there, but macOS marks them as hidden and Windows appears to respect this – doing a dir /a:h in one of my exFAT folders will show them but by default they don’t appear in Windows Explorer. I figured this wasn’t a big deal – I could live with a bunch of hidden 4K files – so I removed the temporary Dropbox account and installed the real one. This may have been a bit overly optimistic. Dropbox happily synced 40Gb of files from Windows, but when I rebooted into macOS, it created the ._ files for all of them… and then when I rebooted into Windows again, it started trying to sync all the hidden ._ files. After twelve hours of this, I concluded that something wasn’t working quite as I’d hoped, so I modified the plan slightly.

I realized that I don’t actually need the Dropbox agent running on both OSes – just having the shared Dropbox folder is enough – so I uninstalled the Dropbox agent from macOS, and things suddenly got a lot more straightforward. The dot-underscore files are still there. On the exFAT shared partition they’re hidden by default, but Dropbox helpfully syncs all the ._ files into all my other Windows systems and so I’ve created  a batch script in my Dropbox folder that finds any of these and sets them to hidden so they don’t show up in Windows Explorer. It’s a one-liner:

for /r %1 in (._*) do attrib +h "%1"

The last thing I wanted to try before doing it for real was to see how VMWare Fusion coped with the shared partition. As well as dual-booting between macOS and Windows, I’ll also occasionally use VMWare to run the Windows partition as a VM inside macOS, which works pretty well. Here’s what I discovered:

  • As soon as you boot the Windows VM, both the BOOTCAMP partion and the Shared partition disappear from macOS.
  • The BOOTCAMP partition is mounted in VMWare as drive C: (obviously), and the Shared partition appears in the VM as drive D:
  • The Dropbox agent will immediately pick up and sync any files created from macOS
  • When I shut down the VM, the BOOTCAMP and Shared partitions both show up in macOS again and everything’s back to normal.

So, here’s the final setup I ended up with. Windows 10 is running the Dropbox agent, and syncing files with my Dropbox cloud storage as intended. The Dropbox folder is on the shared exFAT partition, so macOS can read and write my local Dropbox folder, but macOS doesn’t sync anything. If I add files to Dropbox when running macOS, when I reboot into Windows again, the sync agent will pick up these new files and sync them to the cloud. Finally, if I’m in macOS and find I need a file that’s been added to Dropbox from another device and hasn’t synced yet, I can either reboot into Windows and sync it, fire up the VMWare machine for a few moments, or just download it via the web interface.

I've been running it for a couple of weeks, including a week of wrangling PowerPoint slides and shared files at NDC London, and it's working really quite nicely. And it's really rather nice to have nearly 200Gb of free disk space after installing both operating systems, all my apps and utilities, and 40Gb worth of Dropbox files.

No comments: