How to free up and automatically manage disk space for WSL
freecodecamp.org65 points by twilight-code 4 days ago
65 points by twilight-code 4 days ago
I think Optimize-VHD will do the same thing as messing with diskpart.
It's a shame you can't just turn on live TRIM support.
This kind of disk image is a bunch of multi-megabyte blocks of data, plus a list of where each block goes on the virtual disk. Implementations can support TRIM by deleting the block that's zeroed and moving the block at the end of the file into that spot. VirtualBox can do this, shrinking the file when the guest OS TRIMs, but Hyper-V can't.
It seems like just using a sparse file would make things simpler. I guess there would be potential inefficiency when you try to move the VM to another drive, but for just storing and using the VM in place it doesn't seem like this has to be hard: a TRIM inside the VM gets translated into a hole punch in the disk image file, which gets translated into a TRIM sent to the underlying storage.
rsync in Linux can move sparse files as efficiently as possible, but I don't know if there's an equivalent in Windows.
It'd be very strange if there is not.
Sparse files would be useful but I'm pretty sure sparse files on NTFS always reserve their full size. Eating 1TB of free space for my ext4.vhdx is a non-starter.
https://gist.github.com/Q726kbXuN/c80028aa1714032546d3b442c1...
Nope, they don't eat up space. One of the fun side effects of sparse files on NTFS is calling WriteFile on the sparse file can lead to an ERROR_DISK_QUOTA_EXCEEDED error, which applications tend to not expect.
Based on https://learn.microsoft.com/en-us/windows/win32/fileio/spars... and subsequent links, it sounds like sparse files count against disk quotas as if they weren't sparse, but otherwise sparse files actually work on NTFS.
If sparse files didn't save on actual space allocations, it would simply be a lie to call them sparse files. Granted, that's totally something Microsoft would do, and has done for other features they lack, but that doesn't seem to be the case here.
and the vhdx format already allows for this too. You can mark a block as "present, but zeroed out" (contents of never accessed blocks is undefined). These types of blocks don't have an actual block of data in the file, but still have well defined semantics.
Actually great intro for step-by-step powershell scripting.
I thought I didn't know about "Throw-And-Exit" command, but no, my powershell doesn't have that and google also doesn't know a thing, so wonder what's up with that.
And instead of manual confirmations, one can write script with ShouldProcess support to have support for builtin -Confirm/-WhatIf/-Force parameters. And it would actually be a great use case for that scheduled task to run with -Force parameter instead of changing code to strip out confirmations. https://learn.microsoft.com/en-us/powershell/scripting/learn...
Throw-And-Exit is a custom function in the script @ https://github.com/brooks-code/WSL-VHDX-Compact/blob/main/ws...
Well in that case, a regular throw can be used. Don't see a need for custom function. From docs:
> The throw keyword causes a terminating error. You can use the throw keyword to stop the processing of a command, function, or script.
I recently used https://github.com/okibcn/wslcompact to shrink a couple of WSL VHDX files, and it worked great.
Looking at the source, it seems to use a `wsl --export` option rather than `diskpart`.
Exporting and re-importing the WSL distro is simpler and usually preferred. The tutorial and script are based on this conversation: https://superuser.com/questions/1606213/how-do-i-get-back-un... which mentions some issues with the re-import that includes the need to reset the default username.
You also may need to run fstrim inside WSL to make all the free space actually compactible
Someone should really make a utility to make this easy if Microsoft won't do it.
There are two options mentioned out there: - https://github.com/brooks-code/WSL-VHDX-Compact - https://github.com/okibcn/wslcompact and probably (much) more on GitHub :)
By the way, is WSL1, which doesn't use vhds and so doesn't suffer from such issues (but has slowed io), effectively abandoned by MS?
I'm done with WSL. It just makes life too hard, dual disk is a problem, two network endpoints two firewalls, two different hypervisors you can choose from where some things expect one some the other.
I used to think a windows laptop would be better for hardware management, maybe it is but I just gave up and installed Linux. My life is so much easier.
WSL is really convenient for a lot of use cases though. Much better than dual booting on a machine used primarily for gaming.
I’m also doing WSL-centric development and I’m not even really gaming on that machine, I just appreciate the comforts that come with competent multi monitor support and reliable wake from sleep.
My biggest frustration with WSL at present is that 1Password won’t properly support proxying their ssh agent into it. (Though in googling it again just now, I see a suggestion to try something called npiperelay, so I’ll spend some time seeing again if I can get that working)
I can confirm npiperelay does the job just fine.
If you happen to use Nix, I have a quick and easy package at github:Cu3PO42/gleaming-glacier/next#wsl-ssh-agent. Simply install that and 'source $(which wsl-ssh-agent)' in your bashrc and you should be good to go. (There's also a HM module in the Flake.)
You can actually run the windows ssh.exe binary directly from WSL, no extra setup needed. Give it a try.
I’m aware that 1P themselves recommend this but I don’t like it; a lot of my usage is other things like git or docker or ansible calling ssh and I don’t want to deal with those things having to also be steered to an alternative ssh implementation.
I used to use the `npiperelay` method but have since switched to this and it works nicely.
No such thing needez, just alias ssh to ssh.exe and ssh-add to ssh-add.exe and it works out of the box.
aliasing the ssh.exe generally works, but not for all applications. I had issues with this approach while running ansible playbooks ("getsockname failed: Not a socket").
Wish there was a good option in corporate environments where I can’t just install Linux.
Ask for a Mac ? At this point enough manager were sold on it that most places have to support MacOS, and its popular enough with devs to also have to support dev environments on it.
Like even if you prefer Linux, Mac is a huge step up over windows and WSL if those were your only options.
Also latest Mac machines aren't even that overpriced as they were before their in-house CPUs, for the performance you get really there is only one PC chip out there right now that can be compared and that chip is available in like 2 laptops.
If you want to run Linux how is macOS a step up? It's a non starter.
Linux is Linux Windows offers Wsl1 and Wsl2 integration
The closest thing macOS has in any official capacity is their containers project, running each container in a VM for "security" at the expense of poor resource allocation and no compatibility with table stakes tooling like compose.
At work I asked for a remote server and my pc is just a terminal to use ssh & xwindows.
I also considered this, but I was not able to solve the security problems. That remote server must (or not?) hold all my SSH keys, AWS creds, and other stuff, to even be useful. Secops guys would kill me for this. Until then, WSL is good enough.
I know "secops" guys tend to not really understand what's going on, but you can sell it as just a different computer. You could compromise and install whatever handbrake^w security product they like. Bonus points for that computer being separate from the one where you usually browse the interwebs and such.
However, in practice, at least for the kind of compiling I do, AWS EC2 VMs tend not to be faster than my cheap HP Elitebook. Maybe if you can leverage a hefty number of cores, the situation is different. For regular email pushing, the laptop is good enough.
Our secops guys broke most of our laptops so the engineering teams can't use them for development. They told them to use a Hyper-V VM. So they do. With a Hyper-V vswitch that talks directly to the ethernet adapter rather than the VPN connection. So effectively their policy leads to all those SSH keys, AWS credentials and other stuff to be stored on a virtual machine which is connected directly to the public internet and bypasses all DLP and security controls.
The more I work with secops people the more I fail to trust or respect them.
I have something like that at work for testing Windows things, since my work machine is a Mac.
It's pretty miserable in terms of latency and window management. If you can get this to work well with NX or X forwarding, that's cool, but it's a second class offering when that's your only option.
Hm, are you using native Windows RDP? That one has pretty low latency even in the default configuration (assuming a capable client).
Yes, I'm using Microsoft's RDP client (which on Mac is now called "The Windows App") to connect to an Azure Virtual Desktop.
My home internet connection pretty low latency (wired) and pretty high throughput (gigabit).
It's not absolute torture using the AVD, but it's definitely clunky.
For me using Microsoft's RDP client to a Windows machine a few states over is rather frustrating; using a slower machine locally was much better.
Windows RDP used to be great, but now they figured they'd put animations and transparencies all over the place, so it's become very frustrating to use. Even over a local Gigabit network.
You can turn off effects to restore amazing performance (I know on the Mac version it’s the “optimize for retina displays” option).
I'm talking about effects on the server side. Launching edge for the first time is a major pain, for example.
Edge has a ton of visual noise by default. For me, it's pretty disorienting.
I watched a colleague use VSCode earlier today and was kinda shocked at the amount of pop-ups, and also how they'd cover the active pane, including what the developer was typing.
My attention system isn't built for modern Microsoft apps, I think.
i'm the opposite of you, i'm so done with Linux on my laptop. I can't count how many time my machine cooks itself inside my bag when using Linux. Windows also has the same problem, but much less frequently.
Now if i want linux, WSL2, and if i want linux graphic, Vmware or something similar
Well, this just means PCs are all over the place and it's hard to know what you get beforehand.
My experience is the exact opposite. When I let my laptop running windows sleep overnight, in the morning it's warm. Under Linux, it's room-temperature. I rarely run Windows on it, so there's no checking for mail or whatever in the background, I only have 2-3 games, Prime Video and Lightroom installed. It's also kept up to date, so no updates to install.
But, at least it now wakes back up. Up until a few months ago, it would randomly freeze up or die and reboot and get stuck at the Linux ZFS decryption prompt (I dual boot, and Linux is first). This is a 2020 model laptop, which didn't work under a fresh Windows install out of the box. The webcam wasn't working for a good 3 years (apparently some USB chip wasn't detected). The only issue I have under Linux is that the mic mute LED sometimes turns off even though the mic is muted. It's a standard-fare HP Enterprise, no dual GPU or anything fancy.
I have seen that with both Linux and Windows. And there are bugs like this https://bugzilla.mozilla.org/show_bug.cgi?id=1808039 that prevents sleep on Windows, but somehow does not exist on Chrome, or when you use Linux/Mac. I have determined that if battery is really important for a certain occasion, I just manually power the laptop off. Thankfully, these days laptops are fast enough to start make it tolerable.
Of course, I can't understand why Microsoft or Linux distribution contributors along with software vendors never manage to truly "fix" such important usability issues.
Yeah been there. My company issue laptop cooked itself to death once in my bag when I was on the train to a customer.
I was pleasantly surprised when I bought a Mac (M1 MBP) a few years ago. It doesn't have all the idiot behaviours that Windows and Linux seems to have. It's so boring. It just works. If the Linux/Windows folk can nail that quality down it'd be nice but I have my doubts it'll ever happen (after trying for about 30 years!)
Microsoft has contributed to Mesa for DirectX tunneling drivers. WSL2 supports GPU HW acceleration out of the box.
I'm desperately doing it a different way. I managed to beat off our corporate IT overlords and kept hold of windows 10 for a long time. I'm using that, virtualbox with debian in it and PuTTY. Far more reliable than WSL. I will switch to windows 11 at the last moment and spend a week doing exactly the same there.
They won't let me have a Linux box because I'm supposed to be a windows desktop dev (now about 10% of my workload) and they don't like Linux because none of the invasive security and VPN crap they add works on it.
I use a Mac at home. Mostly proper Unix at your fingertips. Feels much less dirty than either solution.
i bite the bullet, and moved to msys2. it requires a little more porting, but leads to better code in the long run IMHO