Microsoft Office migration from Source Depot to Git
danielsada.tech364 points by dshacker 6 days ago
364 points by dshacker 6 days ago
Always nice to read a new retelling of this old story.
TFA throws some shade at how "a single get of the office repo took some hours" then elides the fact that such an operation was practically impossible to do on git at all without creating a new file system (VFS). Perforce let users check out just the parts of a repo that they needed, so I assume most SD users did that instead of getting every app in the Office suite every time. VFS basically closes that gap on git ("VFS for Git only downloads objects as they are needed").
Perforce/SD were great for the time and for the centralised VCS use case, but the world has moved on I guess.
Some companies have developed their own technology like VFS for use with Perforce, so you can check out the entire suite of applications but only pull the files when you try to access them in a specific way. This is a lot more important in game development where massive source binary assets are stored along side text files.
It uses the same technology that's built into Windows that the remote drive programs (probably) use.
Personally I kind of still want some sort of server based VCS which can store your entire companies set of source without needing to keep the entire history locally when you check out something. But unfortunately git is still good enough to use on an ad-hoc basis between machines for me that I don't feel the need to set up a central server and CI/CD pipeline yet.
Also being able to stash, stage hunks, and interactively rebase commits are features that I like and work well with the way I work.
Doesn’t SVN let you check out and commit any folder or file at any depth of a project you choose? Maybe not the checkouts and commit, but that log history for a single subtree is something I miss from the SVN tooling.
You can indeed. The problem with this strategy is that now you need to maintain the list of directories that needs to be checked out to build each project. And unless this is automated somehow, the documentation will gradually diverge from reality.
Can you not achieve the log history on a subtree with `git log my/subfolder/`? Tools like TortoiseGit let you right click on a folder and view the log of changes to it.
Yes it can, but the point is that in a git repo you store the entire history locally, so whenever you clone a repo, you clone its history on at least one branch.
So when you have a repo that's hundreds of GB in size, the entire history can be massive.
My firm still uses perforce and I can't say anyone likes it at this point. You can almost see the light leaves the eyes of new hires when you tell them we don't use git like the rest of the world.
Yeah it's an issue for new devs for sure. TFA even makes the point, "A lot of people felt refreshed by having better transferable skills to the industry. Our onboarding times were slashed by half".
Interesting to hear it was so much of a problem in terms of onboarding time. Maybe Source Depot was particularly weird, and/or MS were using it in a way that made things particularly complicated? Perforce has never felt especially difficult to use to me, and programmers never seem to have any difficulty with it. Artists and designers seem to pick it up quite quickly too. (By and large, in contrast to programmers, they are less in the habit of putting up with the git style of shit.)
> Interesting to hear it was so much of a problem in terms of onboarding time. Maybe Source Depot was particularly weird, and/or MS were using it in a way that made things particularly complicated?
It was not. It was literally a fork of perforce with executable renamed to sd.exe from p4. Command line was pretty much identical.
> git style of shit
It's a fairly typical Unix tool, but: worse! It's got a rather cryptic and unhelpful command line UI that's often fiddly to use, rather requires you to understand its internals in order to work with it, and has lots of perplexing failure modes. There are GUIs for it, which do help, but people moan at you if you use them, and/because they're often opinionated and oriented towards some specific workflow or other. And they don't really stop it being fiddly to use, you still need to bear in mind how it works internally to use it, and the perplexing failure modes remain.
It's a generalisation, but, by and large, artists and designers don't enjoy using these sorts of tools. Also, they are more likely to be working with unmergeable files, something that git isn't really designed to work with.
(Programmers often don't like these sorts of tools either, but - again, a generalisation - they're harder to avoid if you're a programmer, so the average one is typically a bit more practised at putting up with this crap.)
In 2025, source code version control is incredibly complex. Hell, it was already complex 15 years ago, but the tools were probably worse! Have you ever seen the full manual for GCC/Clang? It is nuts. FFmpeg is legendary hard to use from the command line (no hate on Fabrice Bellard -- you still the man). Same for ImageMagick. I don't fault Git for it. If you want fine-grained control over every single possible action... well, you will wind up with a command line tool with an insane number of options.
Can't say anything about perforce as I've never used it, but I'd give my left nut to get Google's Piper instead of git at work :)
Piper's syntax is Perforce syntax.
I moved to Google from Microsoft and back when employee orientation involved going to Mountain View and going into labs to learn the basics, it was amusing to see fresh college hires confused at not-git while I sat down and said "It's Source Depot, I know this!"[1]
I cannot believe that new hires would be upset by the choice of version control software. They joined a new company after so many hoops and it's on them for having an open mind towards processes and tools in the new company.
I feel like I’ve got an open mind towards processes and tools; the problem with a company using anything other than Git at this point is that unless they have a good explanation for it, it’s not going to be an indicator that the company compared the relative merits of VCS systems and chose something other than Git - it’s going to be an indicator that the company doesn’t have the bandwidth or political will to modernize legacy processes.
Yeah but as a new hire, one doesn't yet know whether there is a good explanation for using a non-git tool. It takes time to figure that out.
A legacy tool might be bad, or it might be very good but just unpopular. A company that devotes political will to modernize for the sake of modernizing is the kind of craziness we get in the JS ecosystem.
I mean if I'm a potential new hire, I'm just going to ask why they're not using Git, and give them the opportunity to give me a good explanation before I start there.
I'm with you on the JS ecosystem - I work on crufty .NET desktop apps - but you couldn't pay me to work on my crufty .NET apps using SVN.
maybe they're on bazel?
I almost cried of happiness when we moved to git from SVN on my first job after being there for 6 months
They might not be upset on the first few weeks but after a month or so they will be familiar with the pain.
> cried of happiness when we moved to git from SVN
I recall moving from RCS and CVS and thinking: "Oh, this is a real improvement." (Forgive me; I was young.)
Then, I moved from CVS to SVN and thought: "This is revolutionary. I can rename and merge." (Again, please withhold throwing your tomatoes until the end of the story!)
Later, projects grew large enough, that SVN became horrible because of how I/O worked on my PCs/networks (file by file, long before 1G network to developer PCs and horribly slow spinning disks).
The upgrade to Git was epic. Sure, the commands are weird and complex. More than 10 years later, I still need to lookup a command once a month, but it beats the pants off anything before. Hat tip to all the people (Hamano-san, et al) who have contributed to Git in the last 20 years. My life as a developer is so much better for it.
I never used CVS but I remember our internal joke about SVN is that SVN is like hotel California, you can branch but you can never leave [by merging the branch]
> I cannot believe that new hires would be upset by the choice of version control software.
I can, if the version control software is just not up to standards.
I absolutely didn’t mind using mercurial/hg, even though I literally haven’t touched it until that point and knew nothing about it, because it is actually pretty good. I like it more than git now.
Git is a decent option that most people would be familiar with, cannot be upset about it either.
On another hand, Source Depot sucked badly, it felt like I had to fight against it the entire time. I wasn’t upset because it was unfamiliar to me. In fact, the more familiar I got with it, the more I disliked it.
You missed the point. The point is, a new hire being asked to use a new version control software likely doesn't know whether the tool is more like hg or more like source depot. It takes times to figure out that. So they need to reserve judgement.
Not what I got out of that comment, but I fully agree with your take on new hires not being able to tell whether an unfamiliar version control piece of software is good or bad until they get at least somewhat familiar with it.
The problem is that you come to a prestigious place like Microsoft and end up using horrible outdated software.
Credit where credit is due at my time at Excel we did improve things a lot (migration from Script# to TypeScript, migration from SourceDepot to git, shorter dev loop and better tooling etc) and a large chunk of development time was spent on developer tooling/happiness.
But it does suck to have to go to one of the old places and use sourcedepot and `osubmit` the "make a change" tool and then go over 16 popups in the "happy path" to submit your patch for review (also done in a weird windows gui review tool)
Git was quite the improvement :D
A craftsman appreciates good tools.
Is git a good tool then? Not necessarily. Some still think hg is better. Others think newer tools like jj are even better while being git compatible.
Perforce is sufficiently idiosyncratic that it's kinda annoying even when you remember the likes of SVN. Coming to it from Git is a whole world of pain.
If companies don't cater to the whims of the youth, they'd have to hire... old people
But those cost so much more!
But they are Analysts and know corporate speak and are really good at filling their schedules with meetings! They must be so busy doing very meaningful work!
I worked with someone who was surprised the company didn’t use Bitbucket and Discord. They were unhappy about both.
Discord I get, at least from a community or network effect, but Bitbucket? I can’t figure out why anyone but a CTO looking to save a buck would prefer Bitbucket.
We use BitBucket where I work. Due to certain export regulations it's simpler for us to keep as many services as possible on-prem if they're going to contain any of our intellectual property, so BitBucket Server it is. There are other options of course, but all of the cloud solutions were off the table.
But why not GitLab or GitHub’s on prem servers?
I wouldn't know, the choice was made years before I joined the company, and I haven't come across anyone who told me what considerations went in back then.
sorry for the tangent, how you deal with AI?
Why would you expect them to? It's really easy to live without AI.
nothing prevents them to run a gpu locally or on their own infra.
I was asking because I wonder what the enterprises that want to both use AI on their workflows like LLM's and have air-gap owned 100% data and pipelines are doing rn.
Feels like one of the few areas where to compete with big labs to me, might be wrong
External AI are banned, local or otherwise on-prem models are allowed. We're currently experimenting with some kind of llama instance running on one of our servers, but I personally don't use it much.
I cannot imagine many jobs use Discord over Slack/Teams unless they’re gaming related. This was not a gaming related job.
I actually quite like the interface of bitbucket. I think it's better, in a lot of ways, compared to gitlab and github.
What I hate about bitbucket is how stagnated it is.
VFS does not replace Perforce. Most AAA game companies still use Perforce. In particular, they need locks on assets so two people don't edit them at the same time and have an unmergable change and wasted time as one artist has to throw their work away
I'm a bit surprised git doesn't offer a way to checkout only specific parts of the git tree to be honest. It seems like it'd be pretty easy to graft on with an intermediate service that understands object files, etc.
I spent nearly a week of my Microsoft internship in 2016 adding support for Source Depot to the automated code reviewer that I was building (https://austinhenley.com/blog/featurestheywanted.html) despite having no idea what Source Depot was!
Quite a few devs were still using it even then. I wonder if everything has been migrated to git yet.
Naah still a lot of stuff works on sd !! Those sd commands and setting up sd gives me chills !!
I miss CodeFlow everyday. It was such a great tool to use.
CodeFlow lives on and is still held in high regard. It's even made it's way to support the github repos, not just git. https://chromewebstore.google.com/detail/codeflow/aphnoipoco...
Still buried as internal only though.
Having used vss in the 90s myself, it surprised me it wasn't even mentioned.
VSS (Visual SourceSafe) being Microsoft's own source versioning protocol, unlike Source Depot which was licensed from Perforce.
VSS was picked up via the acquisition of One Tree Software in Raleigh. Their product was SourceSafe, and the "Visual" part was added when it was bundled with their other developer tools (Visual C, Visual Basic, etc). Prior to that Microsoft sold a version control product called "Microsoft Delta" which was expensive and awful and wasn't supported on NT.
One of the people who joined Microsoft via the acquisition was Brian Harry, who led the development of Team Foundation Version Control (part of Team Foundation Server - TFS) which used SQL Server for its storage. A huge improvement in manageability and reliability over VSS. I think Brian is retired now - his blog at Microsoft is no longer being updated.
From my time using VSS, I seem to recall a big source of corruption was it's use of network file locking over SMB. If there were a network glitch (common in the day) you'd have to repair your repository. We set up an overnight batch job to run the repair so we could be productive in the mornings.
> ...I seem to recall a big source of corruption was it's use of network file locking over SMB...
Shared database files (of any kind) over SMB... shudder Those were such bad days.
As much as I wish for it, smb shares and access databases live on with the same awful problems they've always had with locking and concurrent users.
Oh, TIL! Thanks for adding that to the story.
Indeed my experiences of vss was also not amazing and certainly got corrupted files too.
I used VSS in the 90s as well, it was a nightmare when working in a team. As I recall, Microsoft themselves did not use VSS internally, at least not for the majority of things.
That’s correct. Before SD, Microsoft orgs (at least Office and Windows; I assume others too) used an internal tool called SLM (“slime”); Raymond Chen has blogged about it, in passing: https://devblogs.microsoft.com/oldnewthing/20180122-00/?p=97...
Yes, I used VSS as a solo developer in the 90s. It was a revelation at the time. I met other VCS systems at grad school (RCS, CVS).
I started a job at MSFT in 2004 and I recall someone explaining that VSS was unsafe and prone to corruption. No idea if that was true, or just lore, but it wasn't an option for work anyway.
The integration with sourcesafe and all of the tools was pretty cool back then. Nothing else really had that level of integration at the time. However, VSS was seriously flakey. It would corrupt randomly for no real reason. Daily backups were always being restored in my workplace. Then they picked PVCS. At least it didnt corrupt itself.
I think VSS was fine if you used it on a local machine. If you put it on a network drive things would just flake out. It also got progressively worse as newer versions came out. Nice GUI, very straight forward to teach someone how to use it (checkout file, change, check in like a book), random corruptions about sums up VSS. That checkin/out model seems simpler for people to grasp. The virtual/branch systems most of the other ones use is kind of a mental block for many until they grok it.
I was mandated to use VSS in a university course in the late 90s -- one course, one project -- and we still managed to corrupt it.
> No idea if that was true
It's an absurd understatement. The only people that seriously used VSS and didn't see any corruption were the people that didn't look at their code history.
I used VSS for a few years back in the late 90's and early 2000's. It was better than nothing - barely - but it was very slow, very network intensive (think MS Access rather than SQL), it had very poor merge primitives (when you checked out a file, nobody else could change it), and yes, it was exceedingly prone to corruption. A couple times we just had to throw away history and start over.
SourceSafe had a great visual merge tool. You could enable multiple checkouts. VSS had tons of real issues but not enabling multiple checkouts was a pain that companies inflicted on themselves. I still miss SourceSafe's merge tool sometimes.
Have you used Visual Studio's git integration? (Note that you could just kick off the merge elsewhere and use VS to manage the conflicts, then commit from back outside. Etc.)
We used to call it Visual Source Unsafe because it was corrupting repos all the time.
As I recall, one problem was you got silent corruption if you ran out of disk space during certain operations, and there were things that took significantly more disk space while in flight than when finished, so you wouldn’t even know.
When I was at Microsoft, Source Depot was the nicer of the two version control systems I had to use. The other, Source Library Manager, was much worse.
iirc, we called it visual source shred
kinda nice to know it wasn't just our experience
My memory is fuzzy on this but I remember VSS trusting the client for its timestamps and everything getting corrupted when someone's clock was out of sync. Which happened regularly because NTP didn't work very well on Windows back in the early 2000s.
I was on the team that migrated Microsoft from XNS to TCP/IP - it was way less involved, but similar lessons learned.
Migrating from MSMAIL -> Exchange, though - that was rough
Is that what inspired the "Exchange: The Most Feared and Loathed Team in Microsoft" license plate frames? I'm probably getting a bit of the wording wrong. It's been nearly 20 years since I saw one.
Probably. A lot of people really loved MSMAIL; not so much Exchange.
I have more long, boring stories about projects there, but that’s for another day
And sometimes they loved MSMAIL for the weirdest reasons...
MSMAIL was designed for Win3.x. Apps didn't have multiple threads. The MSMAIL client app that everyone used would create the email to be sent and store the email file on the system.
An invisible app, the Mail Pump, would check for email to be sent and received during idle time (N.B. Other apps could create/send emails via APIs, so you couldn't have the email processing logic in only the MSMAIL client app).
So the user could hit the Send button and the email would be moved to the Outbox to be sent. The mail pump wouldn't get a chance to process the outgoing email for a few seconds, so during that small window, if the user decided that they had been too quick to reply, they could retract that outgoing email. Career-limited move averted.
Exchange used a client-server architecture for email. Email client would save the email in the outbox and the server would notice the email almost instantly and send it on its way before the user blinked in most cases.
A few users complained that Exchange, in essence, was too fast. They couldn't retract a misguided email reply, even if they had reflexes as quick as the Flash.
I re-wrote MSPAGER for Exchange. Hoo boy what a hack that was! A VB3 app running as a service, essentially. I don't know if you remember romeo and juliet; those were PCs pulled from pc-recycle by a co-worker to serve install images.
My neurons are firing on "MSPAGER" in recognition, but romeo and juliet draw blanks - which is a good thing. Because if you know the infrastructure, but you weren't involved in the implementation, then there's typically a bad reason why you know how the sausage is getting made. :)
Also, I survived Bedlam DL3 (but I didn't get the t-shirt).
see https://techcommunity.microsoft.com/blog/exchange/me-too/610... for the explanation for the Bedlam reference.
if you had a thread in MSMAIL, and cc: MSPAGER, anyone on the thread with a pager would get a page from the thread.
> A few users complained that Exchange, in essence, was too fast.
That is something that's actually pretty common and called "benevolent deception" - it has been discussed on HN some years past, too [1].
I wouldn't call either Mail Pump's slow email processing a "benevolent deception", nor Exchange's quick email processing an attempt to be perceived as a fast email server.
In MSMail/Exchange Client/Outlook, the presence of email in the Outbox folder signifies that an email is to be sent, but that the code for sending the email hasn't processed that particular email.
MSMail being slower than Exchange to send email is a leaky abstraction due to software architecture.
Win3.x doesn't support multithreaded apps, using a cooperative multitasking system. Any app doing real work would prevent the user from accessing the system since no user interface events would be processed.
So the Mail Pump would check to see if the system was idle. There are no public or secret/private Windows APIs (despite all the MS detractors) for code to determine if the system is idle - you, the developer, had to fall back to using heuristics. These heuristics aren't fast - you didn't want to declare that the system was idle only to discover the user was in the middle of an operation. So the Mail Pump had to be patient. That meant the email could sit in the outbox for more than a second.
Exchange Server was a server process running on a separate box. When an email client notified the Exchange Server that an email was to be sent (whether via RPC or SMTP command), Exchange Server didn't have to wait in case it would block the user's interaction with the computer. Exchange Server could process the email almost immediately.
But there was a happy resolution to the conundrum - no, the Exchange Server didn't add a delay.
Some architect/program manager had added a "delay before sending/processing" property to the list of supported properties of an email message. The Exchange/Win95/Capone email client didn't use/set this property. But a developer could write an email extension, allow the user to specify a default delay for each outgoing email and the extension could get notified when an email was sent and set this "delay before sending/processing" property, such that Exchange Server would wait at least the specified delay time before processing the email message.
The user who desired an extended delay before their sent email was processed by the Exchange Server, could install this client extension, and specify a desired delay interval.
Outlook eventually added support for this property a few years later.
I notice that Gmail has added support for a delay to enable the user to undo sent emails.
Ha, maybe my old memory is rusty, but I feel like I recognize this name and you had an old blog with some quotable Raymond Chen -- one bit I remember was something like
"How do you write code so that it compiles differently from the IDE vs the command line?" to which the answer was "If you do this your colleagues will burn you in effigy when they try to debug against the local build and it works fine"
> Authenticity mattered more than production value.
Thanks for sharing this authentic story! As an ex-MSFT in a relatively small product line that only started switching to Git from SourceDepot in 2015, right before I left, I can truly empathize with how incredible a job you guys have done!
Yeah, it was a whole journey. I can't believe it happened. Thanks for your comment.
Thank you! Btw, it reminds me of the book "Showstopper" about the journey of releasing Windows NT; highly recommended!
Thanks for the recommendation! I was just about to reread "Soul Of A New Machine", but will try Showstopper instead, since it sounds to be the same genre.
tangentially, if you like that genre one of my favourite books in it is "where wizards stay up late", about the development of the internet.
I spent a lot of time coaching people out of source depot, it was touch and go there for a while. It was worth it though thank you for Your effort.
I’d like to know when Microsoft internally migrated away from Visual SourceSafe…
They should have recalled it to avoid continued public use…
I doubt most teams ever used it.
I spent a couple years at Microsoft and our team used Source Depot because a lot of people thought that our products were special and even Microsoft's own source control (TFS at the time) wasn't good enough.
I had used TFS at a previous job and didn't like it much, but I really missed it after having to use Source Depot.
I was surprised that TFS was not mentioned in the story (at least not as far as I have read).
It should have existed around the same time and other parts of MS were using it. I think it was released around 2005 but MS probably had it internally earlier.
SLM (aka slime, shared file-system source code control system) was used in most of MS, aka systems & apps.
NT created (well not NT itself, IIRC, there was some an MS-internal developer tools group in charge)/moved to source depot since a shared file-system doesn't scale well to thousands of users. Especially if some file gets locked and you DoS the whole division.
Source depot became the SCCS of choice (outside of Dev Division).
Then git took over, and MS had to scale git to NT-size scale, and upstream many of the changes to git mainline.
Raymond Chen has a blog that mentions much of this - https://devblogs.microsoft.com/oldnewthing/20180122-00/?p=97...
TFS was used heavily by DevDiv, but as far as I know they never got perf to the point where Windows folk were satisfied with it on their monorepo.
It wasn't too bad for a centralized source control system tbh. Felt a lot like SVN reimagined through the prism of Microsoft's infamous NIH syndrome. I'm honestly not sure why anyone would use it over SVN unless you wanted their deep integration with Visual Studio.
After the initial TFS 1.0 hiccups, merging was way, way better than SVN. SVN didn't track anything about merges until 1.6. Even today git's handling of file names has nothing on TFS.
I don't know that they ever used it internally, certainly not for anything major. If they had, they probably wouldn't have sold it as it was...
Can't explain TFS though, that was still garbage internally and externally.
Around 2000? The only project I ever knew that used it was .NET and that was on SD by around then.
I didn't even know Microsoft SourceSafe existed.
We used it. We knew no better. It was different then, you might not hear about alternatives unless you went looking for them. Source Safe was integrated with Visual Studio so was an obvious choice for small teams.
Get this; if you wanted to change a file you had to check it out. It was then locked and no-one else could change it. Files were literally read only on your machine unless you checked them out. The 'one at a time please' approach to Source Control (the other approach being 'lets figure out how to merge this later')
Which is exactly how CVS (and its predecessors RCS and SCCS) worked.
They were file based revision control, not repository based.
SVN added folders like trunk/branches/tags that overlaid the file based versioning by basically creating copies of the files under each folder.
Which is why branch creation/merging was such a complicated process, because if any of the files didn't merge, you had a half merged branch source and a half merged branch destination that you had to roll back.
CVS was called the “concurrent version system” because it did not lock files on checkout. Nor did svn. Perforce does.
Perforce does not lock files on checkout unless you have the file specifically configured to enforce exclusive locking in the file's metadata or depot typemap.
True dat, my mistake. That was its major feature, from memory though it still used the same reversed diff file format?
I am quite sure that you can edit files in an svn repo to your heart’s content regardless of whether anyone else is editing them on their machine at the same time.
Yep, svn has a lock feature but it is opt-in per file (possibly filetype?)
A pretty good tradeoff, because you can set it on complex structured files (e.g. PSDs and the like) to avoid the ballache of getting a conflict in an unmergeable file but it does not block code edition.
And importantly anyone can steal locks by default. So a colleague forgetting to unlock and going on holidays does not require finding a repo admin.
The file lock was a fun feature when a developer forgot to unlock it and went on holidays. Don't forget the black hole feature that made files randomly disappear for no reason. It may have been the worst piece of software I have ever used.
The lock approach is still used in IC design for some of the Cadence/Synopsis data files which are unmergable binaries. Not precisely sure of the details but I've heard it from other parts of the org.
Similar in video game shops - lots of binary files, or even huge (non-editable by human) text ones.
A lot of engineering is the same. You cannot diff and merge CAD files, so you lock them.
I remember a big commercial SCM at the time that had this as an option, when you wanted to make sure you wouldn’t need to merge. Can’t remember what it was called, you could “sync to file system” a bit like dropbox and it required teams of full time admins to build releases and cut branches and stuff . Think it was bought by IBM?
I guess you’re talking about Rational Rose? I had the misfortune of using that at my first industry job (fintech in 2004).
Rose is a UML modeling tool
Oops, it was ClearCase that was Rational’s SCM: https://en.wikipedia.org/wiki/IBM_DevOps_Code_ClearCase
Lucky you. Definitely one of the worst tools I’ve had the displeasure of working with. Made worse by people building on top of it for some insane reason.
I remember when we migrated from Visual Source Safe to TFS at my place of work. I was in charge of the migration and we hit errors and opened a ticket with Microsoft Premier Support. The ticket ended up being assigned to one of creators of Source Safe who replied "What you are seeing is not possible". He did manage to solve it in the end after a lot of head scratching.
It was at least a little better than CVS, but with SVN available at the same time, never understood the mentality of the offices that I worked at using Source Safe instead of SVN.
> It was at least a little better than CVS
Highly debatable.
CVS has a horrendous UI, but didn’t have a tendency to corrupt itself at the drop of a hat and didn’t require locking files to edit them by default (and then require a repository admin to come in and unlock files when a colleague went on holidays with files checked out). Also didn’t require shared write access to an SMB share (one of the reasons it corrupted itself so regularly).
Uhh, CVS definitely regularly corrupted itself. I nearly lost my senior research thesis in undergrad because of it. The only thing that saved me was understanding professors and the fact that I could drive back home to my parents' house and get back to my review in 30 minutes, where I had a good copy of my code I could put on an Iomega Zip disk from my desktop instead of the corrupted copy we couldn't pull from CVS in the CS lab.
Agreed. It had a funny habit of corrupting its own data store also. That's absolutely what you want in a source control system.
It sucked; but honestly, not using anything is even worse than SourceSafe.
> Agreed. It had a funny habit of corrupting its own data store also. That's absolutely what you want in a source control system.
I still ‘member articles calling it a source destruction system. Good times.
> It sucked; but honestly, not using anything is even worse than SourceSafe.
There have always been alternatives. And even when you didn’t use anything, at least you knew what to expect. Files didn’t magically disappear from old tarballs.
It was pretty janky. We used it in the gamedev world in the 90s once the migration to Visual C started.
Not doubting it but I don't understand how a shallow clone of OneNote would be 200GB.
Must have videos or binaries.
I want to thank dev leads who trained this green-behind-the-ears engineer on mysteries of Source Depot. Once I understood it, it was quite illuminating. I am glad we only had a dependency on WinCE and IE, and so the clone only took 20 minutes instead of days. I don't remember your names but I remember your willingness to step up and help and onboard new person so they could start being productive. I pay this attitude forward with new hires here in my team no matter where I go.
funny how most folks remember the git migration as a tech win but honestly the real unlock was devs finally having control over their own flow no more waiting on sync windows, no more asking leads for branch access suddenly everyone could move fast without stepping on each other that shift did more for morale than any productivity dashboard ever could git didn’t just fix tooling, it fixed trust in the dev loop
I feel like we're well into the longtail now. Are there other SCM systems or is it the end of history for source control and git is the one and done solution?
Mercurial still has some life to it (excluding Meta’s fork of it), jj is slowly gaining, fossil exists.
And afaik P4 still does good business, because DVCS in general and git in particular remain pretty poor at dealing with large binary assets so it’s really not great for e.g. large gamedev. Unity actually purchased PlasticSCM a few years back, and has it as part of their cloud offering.
Google uses its own VCS called Piper which they developed when they outgrew P4.
git by itself is often unsuitable for XL codebases. Facebook, Google, and many other companies / projects had to augment git to make it suitable or go with a custom solution.
AOSP with 50M LoC uses a manifest-based, depth=1 tool called repo to glue together a repository of repositories. If you’re thinking “why not just use git submodules?”, it’s because git submodules has a rough UX and would require so much wrangling that a custom tool is more favorable.
Meta uses a custom VCS. They recently released sapling: https://sapling-scm.com/docs/introduction/
In general, the philosophy of distributed VCS being better than centralized is actually quite questionable. I want to know what my coworkers are up to and what they’re working on to avoid merge conflicts. DVCS without constant out-of-VCS synchronization causes more merge hell. Git’s default packfile settings are nightmarish — most checkouts should be depth==1, and they should be dynamic only when that file is accessed locally. Deeper integrations of VCS with build systems and file systems can make things even better. I think there’s still tons of room for innovation in the VCS space. The domain naturally opposes change because people don’t want to break their core workflows.
It's interesting to point out that almost all of Microsoft's "augmentations" to git have been open source and many of them have made it into git upstream already and come "ready to configure" in git today ("conical" sparse checkouts, a lot of steady improvements to sparse checkouts, git commit-graph, subtle and not-so-subtle packfile improvements, reflog improvements, more). A lot of it is opt-in stuff because of backwards compatibility or extra overhead that small/medium-sized repos won't need, but so much of it is there to be used by anyone, not just the big corporations.
I think it is neat that at least one company with mega-repos is trying to lift all boats, not just their own.
Meta and Google both have been using mercurial and they have also been contributing back to upstream mercurial.
git submodules have a bad ux but it's certainly not worse than Android's custom tooling. I understand why they did it but in retrospect that seems like an obvious mistake to me.
Perforce is used in game dev, animation, etc. git is pretty poor at dealing with lots of really large assets
I've heard this about game dev before. My (probably only somewhat correct) understanding is it's more than just source code--are they checking in assets/textures etc? Is perforce more appropriate for this than, say, git lfs?
I'm not sure about the current state of affairs, but I've been told that git-lfs performance was still not on par with Perforce on those kinds of repos a few years ago. Microsoft was investing a lot of effort in making it work for their large repos though so maybe it's different now.
But yeah, it's basically all about having binaries in source control. It's not just game dev, either - hardware folk also like this for their artifacts.
Assets, textures, design documents, tools, binary dependencies, etc…
And yes, p4 just rolls with it, git lfs is a creacky hack.
And often binaries: .exe, .dll, even .pdb files.
Interesting. Seems antithetical to the 'git centered' view of being for source code only (mostly)
I think I read somewhere that game dev teams would also check in the actual compiler binary and things of that nature into version control.
Usually it's considered "bad practice" when you see, like, and entire sysroot of shared libs in a git repository.
I don't even have any feeling one way or another. Even today "vendoring" cpp libraries (typically as source) isn't exactly rare. I'm not even sure if this is always a "bad" thing in other languages. Everyone just seems to have decided that relying on a/the package manager and some sort of external store is the Right Way. In some sense, it's harder to make the case for that.
It's only considered a bad idea because git handles it poorly. You're already putting all your code in version control - why would you not include the compiler binaries and system libraries too? Now everybody that gets the code has the right compiler to build it with as well!
The better organised projects I've worked on have done this, and included all relevant SDKs too, so you can just install roughly the right version of Visual Studio and you're good to go. Doesn't matter if you're not on quite the right point revision or haven't got rough to doing the latest update (or had it forced upon you); the project will still build with the compiler and libraries you got from Perforce, same as for everybody else.
I would say it's considered a bad idea, since git comes from an OpenSource world and its implied understanding, where you don't depend on a specific binary release to reproduce your results, but your results can always be reproduced purely from source - binaries are just considered a product of "the plaintext truth recorded in your repository".
Sure, there are nuances to it as the need for ReproducibleBuilds [1] project demonstrated or people proving this concept of "bootstrapped reproducibility" to the extremes like Guix' full-source bootstrap [2], but I believe the fundamental understanding is the same.
[1] https://reproducible-builds.org/
[2] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-...
> It's only considered a bad idea because git handles it poorly. You're already putting all your code in version control - why would you not include the compiler binaries and system libraries too? Now everybody that gets the code has the right compiler to build it with as well!
No arguments here, it makes perfect sense to me as a practice. It's shortsighted to consider only "libraries" (as source, typically) to be "dependencies"--implicitly you're relying on a compatible compiler version/runtime/interpreter (and whatever those depend on) and etc
What was the nature of this project? Was this something related to game development?
That seems to be the only domain where this approach is used (from what I've heard).
Yes, this was a video game, targeting games consoles - so the executable statically links with pretty much everything, including stuff that would count as part of the OS on a PC. And then, if you can make sure everybody builds with the same libraries then you can be sure that everybody is running the same code. (I imagine this approach is also used for embedded development sometimes, another case where the build artefacts cover a much larger percentage of what gets run on the target than is the case with a PC.)
I will admit that even though I think it might be worth doing (and not doing it because your version control system doesn't make it easy is the worst reason for not doing it!), I'm not sure I'd consider it absolutely mandatory, certainly not for video games anyway. Most projects I've worked on haven't done this, and it's been fine, especially if you have a CI system doing the builds - and now the individual developers' systems just have to be roughly right. But it is nice to have, if you've got the right combination of personpower and institutional will to actually do it. It eliminates a whole category of possible issues straight away.
(I'd like to think that for safety critical stuff like medical or automotive this sort of approach, but done even more carefully, would be par for the course...!)
This is really cool, thanks for the info. I've always wondered about how this was done, especially for games.
> so the executable statically links with pretty much everything, including stuff that would count as part of the OS on a PC.
So this is something like whatever libc implementation was used? What era was this? "modern" ish consoles (ps4/XBox-whatever-from-the-last-decade)?
why is this still the case ?
I've been checking in large (10s to 100s MBs) tarballs into one git repo that I use for managing a website archive for a few years, and it can be made to work but it's very painful.
I think there are three main issues:
1. Since it's a distributed VCS, everyone must have a whole copy of the entire repo. But that means anyone cloning the repo or pulling significant commits is going to end up downloading vast amounts of binaries. If you can directly copy the .git dir to the other machine first instead of using git's normal cloning mechanism then it's not as bad, but you're still fundamentally copying everything:
$ du -sh .git
55G .git
3. The cure for this (git lfs) is worse than the disease. LFS is so bad/strange that I stopped using it and went back to putting the tarballs in git.
This is a problem that occurs across game development to ML datasets.
We built oxen to solve this problem https://github.com/Oxen-AI/Oxen (I work at Oxen.ai)
Source control for large data. Currently our biggest repository is 17 TB. would love for you to try it out. It's open source, so you can self host as well.
Why would someone check binaries in a repo? The only time I came across checked binaries in a repo was because that particular dev could not be bothered to learn nuget / MAVEN. (the dev that approved that PR did not understand that either)
Because it’s way easier if you don’t require every level designer to spend 5 hours recompiling everything before they can get to work in the morning, because it’s way easier to just checkin that weird DLL than provide weird instructions to retrieve it, because onboarding is much simpler if all the tools are in the project, …
And it’s no sweat off p4’s back.
Hmm, I do not get it.... "The binaries are checked in the repo so that that the designer would not spend 5 hours recompiling" vs "the binaries come from a nuget site so that the designed would not spend 5 hours recompiling".
In both cases the designer does not recompile, but in the second case there are no checked in binaries in the repo... I still think nuget / MAVEN would be more appropriate for this task...
Everything is in P4: you checkout the project to work on it, you have everything. You update, you have everything up to date. All the tools are there, so any part of the pipeline can rely on anything that's checked in. You need an older version, you just check that out and off you go. And you have a single repository to maintain.
VCS + Nuget: half the things are in the VCS, you checkout the project and then you have to hunt down a bunch of packages from a separate thing (or five), when you update the repo you have to update the things, hopefully you don't forget any of the ones you use, scripts run on a prayer that you have fetched the right things or they crash, version sync is a crapshoot, hope you're not working on multiple projects at the same time needing different versions of a utility either. Now you need 15 layers of syncing and version management on top of each project to replicate half of what just checking everything into P4 gives you for free.
> VCS + Nuget: half the things are in the VCS, you checkout the project and then you have to hunt down a bunch of packages from a separate thing
Oh, and there's things like x509/proxy/whatever errors when on a corpo machine that has ZScaler or some such, so you have to use internal Artifactory/thing but that doesn't have the version you need or you need permissions to access so.. and etc etc.
I have no idea what environment / team you worked on but nuget is pretty much rock solid. There are no scripts running on a prayer that everything is fetched. Version sync is not a crapshot because nuget versions are updated during merges and with proper merge procedures (PR build + tests) nuget versions are always correct on the main branch.
One does not forget what nugets are used: VS projects do that bookkeeping for you. You update the VS project with the new nugets your task requires; and this bookkeeping will carry on when you merge your PR.
I have seen this model work with no issues in large codebases: VS solutions with upwards of 500,000 lines of code and 20-30 engineers.