It's apparently not possible to simply remove symbolic links to directories using Powershell. Instead, Powershell recursively removes the children. For example:
cmd /c mklink /d MyWindows C:\Windows
If you try to remove MyWindows using Powershell, it prompts to delete children, and if you confirm (or if -rec was supplied), it attempts to remove everything in the C:\Windows directory. I don't suppose I have to explain why this is dangerous and ugly.
cmd.exe behaves much better, and just removes the link, which is also congruent with the expected behavior on Unix.
I encounter this issue frequently when developing with an application that creates symbolic links to my Python installation (called a virtualenv). When I remove the virtualenv using Powershell, Powershell also removes libraries in the system Python installation.
I thought I had logged this bug before, but apparently I only asked about it from the community.