There’s been a bunch of stuff on my mind lately, some of it about software engineering and architecture, some about life and mental health and stuff I’m going through, that kind of thing. So I’m spinning up Hugo, sorting out some hosting and CI/CD pipelines and all that so I can get it all out of my head and on the web so you can read it. If you want to keep up to date there’s an RSS feed which I’m hoping will go ping at least a couple of times a week.

In no particular order…

Programming’s Greatest Mistakes

An ever-evolving look at some of the mistakes in software engineering over the years that have led to explosions, bankruptcies, world-wide Windows outages, or in one case just a severe telling-off from my boss :)

From Devoxx Belgium 2024

How JavaScript Happened

A history of programming languages, but limited to the ones that introduced features, concepts or syntax that led to or influenced JavaScript.

From Øredev 2024

The Albatross Project

The opposite of The Phoenix Project, basically. I’ve started doing it as a one-person play now, which is a lot of fun.

From NDC Porto 2024

The Worst Programming Language Ever

People always ask “Is it JavaScript?” but this is so much worse. It takes all the bizarre and inexplicable design decisions from a wide range of languages and munges them all together in one unusable monstrosity. Also leaves time at the end for audience members to share their own ideas (and has an amazing comment section on YouTube).

From NDC Oslo 2021

How Simple Is “As Simple As Possible”

I do also do (mostly) serious talks sometimes. This one is about unnecessary complexity in software architecture and development, from microservices (do you really need them) to front-end JavaScript frameworks (do they really need to be 16 times the size of the original Quake?).

From NDC Porto 2024

My Speaking Journey

I have done a lot of talks over the last 15 years, from local user groups to international conferences. Here’s how I got to where I am.

In the 90s (yes, I’m old) I did a bit of stand-up comedy, from open spots in pubs to the Edinburgh Fringe, eventually winning the Jongleurs/Times Metro New Act of the Year competition in 1999 and going professional for a couple of years. Bits of being funny for a living were fun, but a lot of bits weren’t, and I eventually ended up back in the software engineering world (although I did get to work as a writer for a BBC comedy show and I have the IMDb entry to prove it).

The urge to get up on stage and show off never really went away though, so when I discovered user groups and meetups and conferences I had to get up there and talk about something. My first talk was Functional Alchemy and was about some of the interesting and extremely hacky things you could do with the functional features in C# 3.0. I did it at DDD Southwest (which is still going) in 2011, and then DDD Reading at the Microsoft campus (which is sadly no more afaict). In Reading, a certain Jon Skeet was in the audience and really enjoyed himself; I think he lasted about 15 minutes before joining in and suggesting things. We ended up pairing on a couple of ideas after the talk and I have to admit, I was more than a little starstruck.

Jon had also been tweeting during the talk, so I came out to find a couple of hundred new Twitter followers (remember Twitter?), which was nice. And then he recommended me to a couple of international conferences, Øredev and Code Mash, giving me a welcome boost into the wider speaking circuit. At Øredev I met Greg Young and mentioned having done stand-up, and he invited me to do a talk at Build Stuff but also asked me to MC the attendee party: tell some jokes, do some crowd work…

“What language do you use?”

“Ruby.”

“I’m sorry?”

“RUBY.”

“No, I heard what you said, I’m just sorry.”

…and introduce the musical acts, which chance I leapt at. That was a fun night.

There was a closing keynote that year, I forget the details, but it was a very dry, very academic talk and after three days of the kind of hardcore technical content you get at Build Stuff, it was tough going for the audience. At some point, somebody tweeted at Greg “Next year let Rendle do the closing keynote”; he asked if I would do something funny and I said “hell yes” and spent a year trying to think of something technical and funny. Eventually I came up with The Worst Programming Language Ever, which has been preserved for posterity on InfoQ.

That talk went very well and I started getting invited to do it, and any others I could come up with, at other conferences, and to build a bit of a reputation for them. Fast-forward to today and I’m practically the resident “locknote” speaker for the NDC conferences, and get invited to speak at other events outside of my .NET bubble, which is amazing. I’m incredibly grateful to be in this position, and very aware of the privilege and luck that (along with a lot of hard work) has led me to this point.

Anyway, that’s the story so far. If you’re interested in getting into speaking I’m always happy to share whatever I’ve learned along this journey. You can reach out on LinkedIn or Bluesky, say hi, and ask for advice or help with the speaking itself, writing abstracts, or anything else.

As I was reading Dustin’s post, a possible alternative model occurred to me: something like YouTube Premium’s subscription model.

How does YouTube Premium work?

My understanding, and please correct me on Twitter if I’m wrong, is that when you buy a YouTube premium subscription, your money is divided between the creators of the videos you watch, based on the total amount of time you spent watching their videos. If I spend three hours watching people building computers on Linus Tech Tips and one hour watching Hot Wheels cars being raced on 3Dbotmaker, then Linus gets 75% of my subscription, and 3D gets 25%. Or something like that.

There’s also a Subscribe button which YouTubers are extremely keen for you to click, so I don’t know if that increases the percentage of your money that they get, but I’m going to imagine it might.

Of course, if you don’t have a YouTube Premium subscription you can still watch all those videos but you’ll get shown ads, and YouTube pay a share of that ad revenue to the creators. Unless you have a YouTube-aware ad-blocker, in which case you’ll enjoy the hard work of all those content creators and they won’t get any money. Which is why I have a Premium subscription and I turn off my ad-blocker on YouTube.

How would that work for Package Managers?

First of all, I’m not suggesting that Package Managers should show you ads every time you install a package. I don’t even know how that would work.

But what if, instead, I could get a NuGet Premium subscription? And I pay $9.99 a month for that subscription, and I get a token which I can add to my feed URL, something like https://api.nuget.org/v3/index.json?npt=9au3efoj9as3ofh9. Which I can configure at machine level for NuGet and dotnet, and in my CI pipelines, and so on. And the people who maintain those NuGet packages can opt-in to monetization, and then every month, a percentage of my Premium subscription money goes to each of those creators based on how often their packages are downloaded using that token? (Hopefully Microsoft would not choose to opt-in to monetization so there’d be some money left for everybody else.)

If there’s a package or a creator I particularly want to support, I can “subscribe” to them, which would increase their share of my subscription.

And if you don’t have a Premium subscription, maybe there could be a not-too-obtrusive message every now and then, in the output of dotnet restore or in one of those little message bars Visual Studio sometimes puts at the top of the document well, saying “hey, you could support the creator of X package by subscribing to NuGet Premium for as little as $4.99 a month!”

Nobody would do this

I’d do it. And I think a lot of people might, if it was as easy as setting up a monthly payment and changing a setting in Visual Studio or Rider or dotnet. And you could have Personal and Corporate subscriptions, and maybe have leaderboards to show who’s the most generous.

I don’t know if it would generate enough revenue for people to live off their OSS work, but if it could be sending a few people a few bucks a month, without them having to set up invoicing or PayPal or Ko-Fi donation pages, maybe it would be worth it.

1.0 beta

It was some time in late 2001 when Microsoft released the beta version of their new framework. I’d just returned to programming after a brief stint as a professional stand-up comic, which I tried for a couple of years before deciding it wasn’t for me.

I was working for a company who shall remain nameless, developing and maintaining a CRM system for non-profits that was written in Gupta Team Developer. It was a dying platform, and we needed something to replace it. We were using Visual C++ and VB 6.0 to create ActiveX components, and Classic ASP for our web application interface, so Microsoft’s new hotness was an obvious line of investigation.

For the first beta, there was no Visual Studio, which made life interesting. No IntelliSense, and no real documentation either. The first program I wrote was a console app that reflected over every public type in the .NET Framework assemblies and generated a very basic set of HTML pages, one per type, divided into a directory per namespace. It didn’t extract the XML doc comments, it just dumped out all the methods, properties, constructors, etc for every type. I wrote the app in Vim, and compiled it from the command line with csc. And then I went browsing around my HTML pages, seeing what was there, comparing it with VB 6.0 and MFC.

1.0

By the time .NET 1.0 and Visual Studio .Net were released in February 2002, I’d written a handful of Windows Forms experiments and was convinced that this was the way forward. It was more fully formed out of the box than Java, fairly easy to learn if you were already familiar with the Microsoft ecosystem, and had some impressive features. ADO.NET! Data-binding! A WYSIWYG window designer! And C#! A language that felt like C++ but without having to remember to delete objects when you were finished with them! I started using it to write utilities for internal use, and lobbying to rewrite our product in C#. But that first version was very rough around the edges. The only resizable collection type was ArrayList, and the only dictionary was HashTable, neither of which were strongly typed. If you wanted a resizable list of int, you had to box them to add them to an ArrayList, and cast them back to int when you accessed them. And if you wanted to pass them to a method, that method had to know the ArrayList had ints in it. It was practically a dynamically typed environment. Microsoft had already announced that .NET and C# 2.0 would be getting “generics”, a magical new language feature that would fix this problem, so we decided to wait before starting the big porting project.

2.0

Four years passed between the releases of .NET 1.0 and 2.0, which is mad to think about now that we get a new version every year. It came with Visual Studio 2005, and the new generics feature was ground-breaking at the time. It was similar to C++ templates, but much easier to work with. It was also implemented properly, thanks to the Microsoft Research team who were working on a functional programming language for .NET. Generics in .NET were reified, which meant they were supported by the runtime and not just the compiler. Using reflection, you could create new generic objects and methods at runtime.

I finally got permission to start a project to rewrite our core product in C#. It was, in the best tradition of 1990s line-of-business applications, mostly just a collection of windows with too many textboxes and comboboxes, providing basic CRUD functionality but with an advanced “query-by-example” feature inspired by Informix 4GL. So rather than doing a screen-by-screen rewrite, I was building my own mini-framework to provide the same functionality.

At the time, .NET had a feature that was supposed to make working with complex database schema less of a chore: Strongly Typed DataSets. At the time, writing plain old C# objects was a chore. C# 2.0 did not have automatic properties like this:

1public class Member
2{
3    public int Id { get; set; }
4}

You had to provide your own backing field and implement the get and set manually, like this:

 1public class Member
 2{
 3    private int _id;
 4
 5    public int Id
 6    {
 7        get
 8        {
 9            return _id;
10        }
11        set
12        {
13            _id = value;
14        }
15    }
16}

Multiply that by a database with some 200 tables averaging 30 columns each. And then implementing the code to map from a SqlDataReader to those objects. NHibernate was in early development, but it was scary “Open Source” and we weren’t allowed to use it. So Strongly Typed DataSets, which you could pretty much just generate from your database and start using, were The Way.

Except they were terrible, and performance was horrible, and they were awful to work with, and data-binding to them from Windows Forms was hideous. But I persisted, and made progress, and had the core part of our core product running on .NET some time in 2006.

And then Microsoft released .NET 3.0.

3.0

Just two years after .NET 2.0, in November 2006, Microsoft released 3.0, at the same time as Windows Vista. The ambition for these two releases had been huge; a new file-system that was part database was one of the many features that never made it to the final code. But we did get Windows Presentation Foundation, a brand new desktop application framework that improved on Windows Forms in many, many ways. The data-binding was far better, for a start, and the layout was declared in XAML instead of hundreds of lines of C# generated by the WinForms designer. And applications developed with WPF looked modern.

So of course I decided to rewrite my rewrite in WPF.

At this point I had a junior developer working with me on the rewrite project, but there were three senior developers and three juniors still hard at work on the old Gupta Team Developer codebase, adding features that customers asked for and polishing the UI as best they could. And this junior developer and I were trying to catch up to them and get to a point where we had a new product with the same functionality as the old one. So throwing out thousands of lines of code and starting again with a brand new framework, which we had to learn as we went along, was probably one of the worse decisions I’ve made in my career.

3.5

Less than a year after 3.0, along came 3.5, bringing with it Language Integrated Query, or LINQ. This included LINQ-to-SQL, which allowed you to generate proper C# classes from your database and query against it right in your code. This was much better than Strongly Typed DataSets, so out they went and in came LINQ. More thousands of lines of code ditched and replaced with the new hotness.

By the end of 2008 my attempts to keep up with Microsoft’s new .NET features were seriously damaging my ability to deliver the promised rewrite. .NET just kept getting more and more awesome, and I couldn’t help chasing after every new feature they added. Except WCF. I never liked WCF.

I ended up burning myself out, and the company scrapped my project in favour of a VB.NET application from a company they’d acquired in the meantime. I didn’t want to do VB, so I quit and got a job at .NET Solutions, which sounded like my kind of place.

.NET Solutions

My new employers were a Microsoft Gold Partner and we got to work on some amazing projects. Getting excited about new features was pretty much part of the job, and projects were shorter so if something new landed while you were working on something, you could just finish that and then jump into a new one using the latest version. When I joined, they were just finishing up a .NET Micro Framework project that ran on a tiny circuit board connected to a fridge.

You know how during the World Cup Final there’s a spike in demand for electricity at half time? People think it’s all the kettles being boiled, but it’s not. It’s all the fridges being opened and warming up while people put milk in their teas and coffees, and then kicking in the refridgeration unit to cool down again. It spikes so much we have to buy extra electricity from France to cope with it.

The idea behind this project was that all the fridges would be able to talk to a web API that would say whether or not they could turn on. So when everybody put the milk back and closed the door, the fridges could coordinate and avoid all demanding extra electricity at once.

It was a fun project. The .NET Micro Framework didn’t have an HttpClient built in, so we had to implement our own over the Socket types. And there were inexplicable bugs which turned out to be due to voltage fluctuations causing problems with the custom ASIC on the board or some such thing.

.NET Solutions also had a bonus scheme that rewarded, among other things, blogging and speaking at user groups, so that was when I started doing that stuff. The first talk I did was about the LINQ-related features in .NET 3.5, particularly lambdas and extension methods, which allowed you to do some funky things with C#. For example, catching more than one exception type…

 1public static void Try<TException1, TException2>(
 2    Action action,
 3    Action<Exception> catchAction)
 4{
 5    try
 6    {
 7        action();
 8    }
 9    catch (TException1 ex)
10    {
11        catchAction(ex);
12    }
13    catch (TException2 ex)
14    {
15        catchAction(ex);
16    }
17}

The third time I gave that talk was at the Developer Developer Developer conference at the Microsoft Campus in Reading. I was setting up when one of the other speakers came up to me and said “you know Jon Skeet’s in the audience?” So that was terrifying. But he loved it, and couldn’t help joining in and making suggestions for ways to push things even further, and I ended up pairing with him once the talk was finished, when I also discovered he’d been tweeting about it all the way through. That was the day I passed 1,000 followers on Twitter, thanks to .NET, C# and Jon.

ASP.NET MVC

In 2007, Scott Guthrie announced ASP.NET MVC, a .NET implementation of the Model/View/Controller pattern for building web applications. Up to then, the way to create web apps with .NET had been WebForms, an ambitious framework which tried to let developers pretend they were building a desktop application without having to worry about things like the HTTP protocol or round-trips to the browser. Unfortunately this required some ugly hacks, most notably ViewState, a potentially enormous chunk of data that was sent back and forth between browser and server on every request.

MVC did away with all of this, replacing it with Controllers that could handle HTTP requests and render Views, embracing the HTTP paradigm. It also came with Razor, a new way of combining C# with HTML to render pages from models.

But maybe the most interesting thing about ASP.NET MVC was that it was open source, with all the source code available on Microsoft’s CodePlex site. Before this, Microsoft, under Steve Ballmer, had been the sworn enemy of open source software; ASP.NET MVC was the first sign that this was starting to change.

4.0

.NET 4.0 and C# 4.0 were a weird release. The headline feature was the Dynamic Language Runtime, or DLR, and the dynamic keyword in C#. I think the main reason for it was to improve Office interop, but there were more interesting things you could do with it.

Around the same time that 4.0 came out, I’d been learning Ruby, and using an ORM called DataMapper which used Ruby’s method_missing feature to map dynamic properties and methods to database calls. There was a new base class in .NET 4.0, DynamicObject, that let you do pretty much the same thing, so as an experiment I wrote a simple library that let you have a dynamic object and call properties and methods on it to do database things, like this:

1dynamic db = new Database(connectionString);
2var order = db.Orders.FindById(id);
3Console.WriteLine(order.Description);

I called it Simple.Data, wrote some blogs about it and announced it on Twitter, and to my surprise it became quite popular. It’s had 800K downloads from NuGet, which isn’t bad. But as people asked for more and more features, and I added them, it became quite complicated and difficult to maintain. Eventually what it really needed was a ground-up rewrite, but by then I had a five year old daughter and a baby son and time was a scarce resource, so I wound the project down. Sorry if you liked it.

4.5

The big thing in .NET 4.5 was the Task-based asynchronous programming model. Instead of methods with Begin and End prefixes, we had methods that returned a Task which could trigger a callback when the underlying operation, usually some form of IO, completed. It was easier to use than the Begin/End model, but still resulted in code like this:

1client.GetAsync(uri)
2    .ContinueWith(r => r.ReadBodyAsStringAsync()
3        .ContinueWith(body => ...))

The levels of callback nesting in a chain with a lot of Task-returning methods quickly got completely out of control.

C# 5.0

This was hugely improved with the release of C# 5.0, with the new async/await keywords, which made this kind of asynchronous programming much easier. Now we could just write code like this:

1var response = await client.GetAsync(uri);
2var body = await response.ReadBodyAsStringAsync();
3...

This was much easier to understand and reason about, although there was an enormous amount of bike-shedding about what the actual keywords should be, particularly await, which many people thought sounded too much like Wait and implied that the current thread would be blocked waiting for the result, when what actually happened was that the current thread was returned to the thread pool and another one would be used when the Task completed.

I remember spending a fair amount of time going through code using either Task.ContinueWith or the old Begin/End methods and replacing it with async/await around this time, and also wishing there were an IAsyncEnumerable type.

4.6, 4.7, 4.8

After 4.5, .NET seemed to stagnate for a while. New features were added, but nothing that merited a major version bump. But the C# team were working on something completely new.

C# 6.0

Up to this point, the C# and VB.Net compilers had been written in C++. With C# 6.0, the compiler teams embarked on a multi-year project, codenamed Roslyn, to rewrite the compilers in their own languages. So the C# compiler was rewritten in C#, and the VB.Net compiler was rewritten in VB.Net. This would make it easier to add new features, although the only changes in C# 6.0 itself were some syntactic sugar for string interpolation and the like.

The really revolutionary thing about Roslyn, though, was that it represented a move to a Compiler-as-a-Service. The Roslyn packages were published on NuGet, and you could use them to parse, modify, analyse and generated C# code at runtime. It became possible to write your own code analysers and fixes and publish them as Visual Studio extensions or NuGet packages. More on this later…

Project K

In late 2015, on the ASP.Net Insiders mailing list, which I was lucky enough to be on, Damian Edwards and David Fowler announced a new project they’d been working on: a cut-down, light-weight and, most importantly, cross-platform version of .NET. It would run on Linux and Mac as well as Windows, and was optimised for writing web applications, without all the baggage of the full-fat, Windows-only .NET Framework. No Windows Forms, no WebForms, no WPF or WCF, just the minimum necessary to run ASP.NET MVC applications.

Moreover, Project K would be properly open source, developed in the open on GitHub, and accepting contributions from the community.

Obviously I downloaded and started playing with this right away, and I loved it. OK, so a huge chunk of the Base Class Library was missing, but it was C# and ASP.NET MVC and Web API working on Linux.

At the time I was a working at a place where the new Linux container technology, Docker, was being enthusiastically embraced. Apps were written in Node.js and Python to run in a cluster using an orchestration platform that I don’t remember the name of, but it wasn’t Kubernetes.

Project K meant that we could use .NET in this environment, and I found an actual use for it. I was working on a “web bug” tracking system that would follow users through the site and then off to external links to verify that they purchased something so we could send them a stuffed animal. This required generating a lot of UUIDs, and under heavy load. And it turns out that generating UUIDs is a compute intensive task, something that Node.js is not particularly good at.

So I wrote the simplest possible microservice in Project K, although it might have been called ASP.NET 5 by that point, which had a single endpoint that just returned a Guid as plain text. The Node.js app could call this service asynchronously, which is something that Node is very good at, and suddenly the system scaled. I don’t know if this was the first production use of the new .NET outside of Microsoft, but it must have been pretty close.

.NET Core 1.0

Project K became ASP.NET 5 became .NET Core 1.0. Along the way the promised new project.json project files were scrapped, which upset a lot of people, but the csproj files for Core applications were far simpler than those for .NET and despite being XML are mostly pretty simple to work with and edit manually when the need arises.

I remember watching a talk given by Damian and David at NDC Oslo where they talked about performance in .NET Core. Damian shared an anecdote wherein Kelly Sommers benchmarked the new Kestrel web server that was the engine of ASP.NET Core against her own Haywire engine, written in C. Kestrel was not fast: something like 70,000 requests per second compared to Haywire’s 1 million requests per second. Per Damian, he asked the Kestrel team why it was so slow, to which they responded “you didn’t tell us you wanted it to be fast.”

I don’t know how much of that is apocryphal, but there was a concerted effort to optimise the hell out of Kestrel, with a lot of contributions from Ben Adams, who had a vested interest in the project as his company, Ilyiad Games, make a game that runs on .NET on the server. By the time 1.0 was released, .NET Core was actually becoming competitive on the famous TechEmpower benchmarks.

Annoyingly, when .NET Core 1.0 launched, I was working at a place where new stuff was frowned upon so I had to continue working with .NET 4.something and WebAPI.

.NET Core 2.1

There was a 2.0 release, which was mainly concerned with adding back a lot of missing APIs from the .NET Framework, but the big release was .NET Core 2.1, which was when performance really became a feature of the new framework. 2.1 introduced the new Span<T> and Memory<T> types, which provided a better way of working with arrays or other chunks of memory, including unmanaged memory. This came with C# 7.2, which added a ref struct modifier that reduced copies when passing struct types between methods and is, I’m pretty sure, only really used for Span<T>; I’ve certainly never used it in my own code.

At this time I was working for one of the big investment banks, with a remit to extract functionality from a behemoth of a WPF application into shared services, and we were allowed to use .NET Core 2.1, so I got to play with it. But the biggest thing I actually achieved there was to use some of the .NET Core packages that were backward-compatible with .NET 4.x to add telemetry to the WPF application.

My favourite thing was a custom-written client for InfluxDB. The official client for .NET was quite inefficient: it allocated a lot of Dictionary<string,string> or Dictionary<string,object> objects for every line of metrics, and was too slow for our purposes. So I got to write an alternative client which used large byte arrays and wrote metrics passed through the new System.Diagnostics.DiagnosticSource model directly to Span<byte>s. My only regret was that we weren’t allowed to open source that library. If I can find the time, I might recreate it as an InfluxDB target for the new OpenTelemetry framework.

.NET Core 3.1

Again, there was a 3.0 release, but at this time Microsoft were using the .1 versions of .NET Core as the “LTS” (Long Term Servicing) releases. The big thing with 3.1 was that Windows Forms and WPF were ported over to Core, so you could build desktop Windows apps with it. Sadly this did not extend to making them cross-platform, which would be a huge effort since both WinForms and WPF are fairly thin wrappers over the underlying Windows APIs.

At the time .NET Core 3.1 was released, Scott Hunter announced that the next version would be .NET 5.0. .NET Core was becoming .NET; this would be where Microsoft invested time and energy in moving the platform forwards, and .NET 4.x would go into long term support, getting security updates and continuing to work on new versions of Windows, but no longer being improved or enhanced.

With this announcement came the news that the work of migrating .NET APIs to .NET Core was finished; if something hadn’t been migrated across now, it wasn’t going to be. This included Windows Communication Foundation, the framework that had been added way back in .NET 3.0 for building Service-Oriented Applications, usually using SOAP over HTTP as the protocol. The official Microsoft recommendation was to migrate WCF services to gRPC, the current standard for building distributed systems and microservices using the RPC pattern. gRPC had been added to ASP.NET Core as a first-class citizen, with a brand new implementation that was fast and efficient and worked with the rest of the ASP.NET Core stack.

As someone who had failed a Microsoft certification twice on the WCF test, and had fought against it on a variety of projects, I was delighted at this news, but it made a lot of people very angry. And I can understand why. I started playing with the new gRPC framework and was impressed, and realised that it mapped pretty easily to WCF’s RPC concepts. In fact, it mapped closely enough that I thought it might be possible to use Roslyn to convert WCF code to use gRPC instead.

I spent a week learning the Roslyn SDK and managed to get it doing a very basic conversion of a simple WCF service to gPRC, and could see that it was possible to do more complicated things as well, and Visual ReCode was born. Although I briefly considered making it open source, my wife would have killed me so I partnered with Gibraltar Software to sell it as a commercial product. I’m still working on it today.

Unity

This is not exactly a .NET version, but Unity is a game engine that uses C# as its scripting language, and provides most of the framework classes for building games. I put it here because it was around this time that I learned it. I used it to build an homage to the Club Penguin games with my son; he would draw the “graphics” in felt tip pen, and we’d scan it in and import it into the game, with me writing C# code and teaching him as we went along. I’ve also made a silly little game called Cheese Times Tables and actually released it on the various mobile App Stores. And I could do all of this with the fundamental knowledge of C# and .NET that I’ve picked up over the years writing line-of-business apps and web sites. I think that’s awesome. If I could find the time I’d be working on Extreme Croquet…

5.0 and 6.0

And that brings us to today. I’ve been a .NET developer for 20 years, I’ve built things with WinForms, WebForms, WPF, WCF, MVC, the Micro Framework, Silverlight, Project K, .NET Core and .NET 6.0. For most of that time Microsoft have innovated on the framework and the C# language and compiler platform, and that innovation has accelerated since the Core era began. We now have Blazor, an amazing new web app framework that can run over a WebSocket between a browser and server, or be compiled to WebAssembly to run .NET code right in modern browsers. It’s easier to work with than WebForms ever was, while providing a lot of the same benefits such as simplified event handling and session state. And we’ve very nearly got MAUI, the new desktop and mobile application framework that is the evolution of Xamarin and a replacement for WPF, WinForms and UWP that also runs on Macs and might, one day, support Linux. There’s even a hybrid MAUI+Blazor model that wraps a Blazor Server application in a MAUI app, letting you target desktop, mobile and web with a single codebase.

With every release, .NET gets better performance across ASP.NET Core, Entity Framework Core, and all parts of the framework. C#, now on version 10 (with version 11 being loudly discussed on GitHub and Twitter), continues to evolve, offering ways of writing more expressive code with more optimal and safer runtime behaviour. The first previews of .NET 7.0 and C# 11 are likely to land any day now, and I’m looking forward to seeing what’s new there.

I don’t know if I’ll still be coding professionally in another 20 years; I’m getting on a bit and ideally I’ll be retired by then. But if there’s a celebration of 40 years of .NET in 2042, I hope to be there for the party.

This is pretty straightforward: open a browser page, passing some kind of unique key, then poll an endpoint on the server until it returns the mystic runes you seek. So this needs to be an ASP.NET Core MVC app, and it needs some transient external storage like, say, a Redis instance. And this is an open source side-project and I really don’t want to be paying for hosting for an essential component of it.

A couple of weeks earlier, Khalid tweeted a link to a .NET app running on a .railway.app sub-domain, and I’d been intrigued and checked out what that was. Turns out, Railway is a cloud platform that provides really, really simple hosting and deployment, as well as plug-ins for things like Redis, MongoDB and Postgres. The good bit is that it’s free for up to three projects. So I thought I’d give it a try and last night, on the stream, I went for it.

Creating a Railway project

I used the Railway web UI to create the project and added the Redis plugin to it. This sets some variables like REDISHOST and REDISPASSWORD which are available as environment variables at runtime. I also added my Twitch app details as custom variables here.

Installing the Railway CLI

Railway provides a CLI that lets you set up projects, run them locally and deploy them to the cloud with simple commands like railway run. Unfortunately it doesn’t work very well on Windows, but that’s no problem these days because Windows runs Linux, and it works perfectly there. I initially tried to install it using NPM, but that didn’t work. That’s not Railway’s fault: NPM just didn’t work.

Luckily for me Jake from Railway was in the stream chat and he suggested using the Curl installation, and very helpfully pasted the command into the chat. So I copied and pasted it into my WSL terminal, because running Linux commands from my Twitch chat with sudo is how I roll.

Then I cd’d into the Windows project directory from my WSL shell (cd /mnt/d/Twitch/StreamBadger). Technically there’s a tiny performance overhead from working with Windows directories in WSL, but in this case it made no noticeable difference.

Logging in

First up I had to do railway login which, funnily enough, took me through the exact same process I’m trying to create for StreamBadger: it opened a browser page and, because I was already logged in, just magically authenticated. Now my WSL session was connected to my Railway account.

Linking the project

On the Railway dashboard page from my project there was yet another command I could just copy and paste into my terminal to link the current directory with the Railway project. So I did that, but not with sudo this time because that was not necessary. I linked from the project root directory where the .sln and Dockerfile files were.

Adding a Dockerfile

Railway runs a bunch of different things like Node.js, Python and Ruby, but it will also just run Docker images, which means it can handle pretty much anything. If there’s a Dockerfile in your directory, it will just use it. So I created a standard Dockerfile for my web app:

 1FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build-env
 2WORKDIR /app
 3
 4# Copy csproj and restore as distinct layers
 5COPY ./StreamBadger.sln .
 6COPY ./src/StreamBadger/StreamBadger.csproj ./src/StreamBadger/
 7COPY ./src/StreamBadgerDesktop/StreamBadgerDesktop.csproj ./src/StreamBadgerDesktop/
 8COPY ./src/StreamBadgerLogin/StreamBadgerLogin.csproj ./src/StreamBadgerLogin/
 9RUN dotnet restore
10
11# Copy everything else and build
12COPY . .
13RUN dotnet publish --no-restore -c Release -o out ./src/StreamBadgerLogin
14
15# Build runtime image
16FROM mcr.microsoft.com/dotnet/aspnet:5.0
17WORKDIR /app
18COPY --from=build-env /app/out .
19ENTRYPOINT ["./StreamBadgerLogin"]

I also added a .dockerignore file to ignore bin and obj directories.

**/bin/
**/obj/

This is important if you’re developing on Windows because the obj directory contains a file called project.assets.json, which is like a lock file for Nuget packages and such. If you copy this into the Docker image, which is running Linux, then the dotnet restore build step will explode and kill your cat.

Running the project locally

You can test your code straight away, pointing at the actual resources in the cloud, like my Redis plugin. I just had to tweak the code to read the connection details from the environment variables mentioned earlier. Then it’s just railway run and it builds the Docker image and runs your app on localhost. Slightly weird thing, though: it arbitrarily chooses a port to bind to. In my case it chose 4411. This is also exposed as an environment variable, PORT, which I believe is quite a common practice in the land of script. Jake from Railway pointed this out and I added some code to my app to use this port:

 1    public static IHostBuilder CreateHostBuilder(string[] args) =>
 2        Host.CreateDefaultBuilder(args)
 3            .ConfigureWebHostDefaults(webBuilder =>
 4            {
 5                webBuilder.UseStartup<Startup>();
 6
 7                // For running in Railway
 8                var portVar = Environment.GetEnvironmentVariable("PORT");
 9                if (portVar is {Length: >0} && int.TryParse(portVar, out int port))
10                {
11                    webBuilder.ConfigureKestrel(options =>
12                    {
13                        options.ListenAnyIP(port);
14                    });
15                }
16            });

Then I did railway run again and there was my app, happily running in Docker and connecting to the Railway Redis instance.

Deploying to the cloud

The theory here is that if railway run works on your machine, then it should work in the cloud too. To deploy it, you just use railway up. That’s it. Under the covers it builds your Docker image, pushes it to Railway’s registry, spins up the container and connects it to your sub-domain (in this instance streambadger-production.up.railway.app/). And it just worked. Took a couple of minutes and there it was.

Connecting an actual domain

I’ve bought streambadger.com because of course I have, so I wanted to get that hooked up as well. I do pretty much all my DNS through Cloudflare because they’re awesome and do magic SSL and such, so I just needed to set up a CNAME pointing to the Railway sub-domain and… no wait, infinite redirect loop. I’d missed a helpful bit in the Provider Specific Instructions that tells you to use the Full SSL/TLS encryption mode with Cloudflare. This is because Railway just won’t serve your site over plain old HTTP, but Cloudflare’s Flexible encryption mode only proxies to HTTP.

In summary, then

I’m super impressed with Railway. They’ve obviously worked very hard on the user experience, and it shows. I had Jake in the stream chat to help me out so I wasn’t having to refer to the docs, but they’re really good. The one small hiccup was the Windows issues with the CLI, which isn’t great, but I’d expect any developer in 2021 to be running WSL anyway.

You get three projects with two plugins per project on the Free plan, with a couple of environments (e.g. Staging and Production) as well. The next step up is the $20/month “Early Adopter” plan, which is just unlimited everything: projects, plugins, environments, deploys. And access to something called Teams, which I don’t know what that is.

I’ve got a few things I run online, and I’m pretty sure if I moved them all over to Railway on that $20 plan I’d be saving money over the various cloud platforms I’m using at the moment. It won’t replace Azure for more complex stuff with a wider range of platform services, but for side projects and simple sites or APIs I think Railway looks great.

Disclaimer

This post was not sponsored or requested by Railway. I used it and liked it and just wanted to share.

Twitch

If you want to watch me learn more stuff like this in real-time, stop by my Twitch channel some time. I’m usually on at 7pm UK (6pm UTC) on Tuesdays and Thursdays.

I’ve been working on Visual ReCode for a while now. It’s a plug-in for Visual Studio 2019 that helps developers migrate old .NET 4.x code to new .NET 5.0 or .NET Core 3.1. We just released version 2.0 , with a new feature called Guided Project Upgrades that helps you upgrade your solution one project at a time, switching to the new SDK-style csproj format, and adding modern target frameworks like net5.0 or netstandard2.0. It checks your NuGet packages and upgrades them if necessary, and makes sure you upgrade projects in the right order.

Visual ReCode also provides a way to migrate WCF applications, which are not supported in the new .NET, to gRPC, which is very much supported (in fact, it’s the best-performing gRPC implementation there is). And right now I’m working on providing the same functionality for ASP.NET WebAPI, migrating it to ASP.NET Core MVC, and looking at a WCF ReST to ASP.NET Core MVC feature.

That still leaves a couple of old frameworks on the table, though. ASP.NET MVC is a possible target for future investigation, although there are quite a lot of differences that mean it would be more of a “helper” than an automated migrator. But ASP.NET WebForms is a whole other thing. It’s been around for 20 years now; it’s really, really complicated; it’s completely unlike MVC or Blazor or anything else in the modern .NET stack. There’s really no way to automate the migration of WebForms applications to ASP.NET Core.

And the chances are, if you’ve got a WebForms application in production, it’s a massive behemoth of code that’s built up over years, and rewriting it would be a huge undertaking. Having worked on rewrite projects in the past, I know from experience that they’re a nightmare, and get derailed all the time by the maintenance requirements of the existing code.

So what to do?

The big problem is that there’s no way to gradually upgrade WebForms (or MVC) to ASP.NET Core. You can’t run Core on .NET 4.x, and you can’t run WebForms on Core. But what if you could run two separate applications and make it look as if they’re the same one?

Enter YARP.

YARP stands for Yet Another Reverse Proxy, and it’s a Microsoft project to build a reverse proxy server on top of the blazing-fast Kestrel HTTP server. It’s not a stand-alone application; it’s a package that runs in an ASP.NET Core app. You just reference the Microsoft.ReverseProxy package and add it as an endpoint in your Startup class.

 1public void ConfigureServices(IServiceCollection services)
 2{
 3    services.AddReverseProxy() 
 4        .LoadFromConfig(Configuration.GetSection("ReverseProxy"));
 5}
 6
 7public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
 8{
 9    app.UseRouting();
10
11    app.UseEndpoints(endpoints =>
12    {
13        endpoints.MapReverseProxy();
14    });
15}

Being just part of the ASP.NET Core pipeline means that you can do all sorts of creative things with your reverse proxy by adding more middleware. And of course, one of the pieces of middleware you could add in there is ASP.NET Core MVC.

 1public void ConfigureServices(IServiceCollection services)
 2{
 3    services.AddControllersWithViews();
 4    services.AddReverseProxy() 
 5        .LoadFromConfig(Configuration.GetSection("ReverseProxy"));
 6}
 7
 8public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
 9{
10    app.UseRouting();
11
12    app.UseEndpoints(endpoints =>
13    {
14        endpoints.MapControllerRoute(
15            name: "default",
16            pattern: "{controller=Home}/{action=Index}/{id?}");
17        
18        endpoints.MapReverseProxy();
19    });
20}

Now, let’s say the home page for your WebForms application is ~/Default.aspx. If you add an MVC action with that route to your Core application, the routing will hit that first. Any routes that aren’t hit in your application, though, will be transparently proxied through to the old app. Now you can rewrite individual pages in your site however you want, while continuing to support and maintain the old application, until eventually everything is migrated and the old code can be retired.

The complexity

If only it were just that easy. There are a couple of things in the way of this approach: session state and authentication. WebForms applications in particular tend to be heavy users of server-side session, whereas in ASP.NET Core it’s much less used. And authentication is very different, too.

I’m pretty sure I’ve figured out a way to address these issues, though. And as a new thing for 2021, I wanted to try my hand at streaming. So I’m combining these two things: you can join me on my Twitch stream and watch as I try to gradually upgrade the Wingtip Toys sample application to ASP.NET Core 5.0 using this approach. I’ve done two streams to date, and so far I’ve got the home page to look roughly the same, and implemented basic session sharing between the two applications (that even works with “in-proc” session on the WebForms side), as well as copying and pasting a bunch of Entity Framework 6.0 classes and finding out that they work perfectly in EF Core 5.0.

Once I’ve got everything working in this test application, I’m going to package up the various components as open-source NuGet packages so that everyone can try this approach.

If you want to catch up on the streams, I’ve archived them on YouTube: Part 1 and Part 2. Or if you just want to browse around the code, you can find it on GitHub.

But there are situations where the compiler is unable to work out that you’ve done a null check, meaning you either have to use the “null-forgiving operator” (where you append a bang to the identifier, like item!.Name), disable the null checking, or just ignore the warning.

One of these that I run across a lot is in LINQ chains. For example, I’m just writing some code that enumerates through the MetadataReferences property of a Roslyn Project to find the file paths of references:

1var files = project.MetadataReferences
2    .OfType<PortableExecutableReference>
3    .Select(p => p.FilePath);

In the Roslyn library, PortableExecutableReference.FilePath is declared as string?, because apparently you can have a PortableExecutableReference that isn’t actually a file. So the type of this expression is IEnumerable<string?>.

Now, I can filter out the null file paths with a simple Where call:

1var files = project.MetadataReferences
2    .OfType<PortableExecutableReference>
3    .Select(p => p.FilePath)
4    .Where(f => f is not null);

But the compiler isn’t actually clever enough to work out what that Where does, so the type of the expression is still IEnumerable<string?>. Vexing. There’s actually no easy way to deal with this within a chain of LINQ methods (that I can think of, at least).

So, I’ve added an extension method to my common library of “stuff that’s not in the BCL” that filters out nulls and changes the expression type accordingly.

1public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source) where T : class
2{
3    foreach (var item in source)
4    {
5        if (item is not null) yield return item;
6    }
7}

And I thought that might be useful for other people so I’m sharing it here.

Advertisement

I ran into this while working on Visual ReCode. It’s an extension for Visual Studio 2019 that helps you migrate code from .NET 4.x to .NET Core 3.1 and .NET 5.0, including a WCF to gRPC conversion to upgrade your services to the modern, cross-platform standard for RPC.

nuke-from-orbit

You know how doing dotnet clean (or right-click > Clean solution in Visual Studio) doesn’t actually delete all the bin and obj directories and the stuff in them? And you know how sometimes you really, really have to do that, because something hiding in one of those directories has got itself into such a muddle that the only way to fix it is to nuke the site from orbit? Yeah. That’s why I wrote this.

The tool is published as RendleLabs.NukeFromOrbit, so you can install it by running:

1dotnet tool install -g RendleLabs.NukeFromOrbit

The -g tells dotnet to install the tool globally so you can use it from anywhere.

This should output:

1You can invoke the tool using the following command: nuke-from-orbit
2Tool 'rendlelabs.nukefromorbit' (version '1.0.3') was successfully installed.

Which helpfully tells you that you can now call it as nuke-from-orbit. If you run it with the --help argument, it will helpfully tell you how to use it:

 1NukeFromOrbit:
 2  Dust off and nuke bin and obj directories from orbit. It's the only way to be sure.
 3
 4Usage:
 5  NukeFromOrbit [options] [<workingDirectory>]
 6
 7Arguments:
 8  <workingDirectory>    [default: D:\blog]
 9
10Options:
11  -y, --yes         Don't ask for confirmation, just nuke it. [default: False]
12  -n, --dry-run     List items that will be nuked but don't nuke them. [default: False]
13  --version         Show version information
14  -?, -h, --help    Show help and usage information

That little feature comes from a NuGet package called System.CommandLine, which is pre-release at the moment but dead useful for this kind of thing.

System.CommandLine

This package provides a command line argument parser which makes it easy to handle flags, options and arguments, and even sub-commands if you’re building something big. You just create a Command, assign a Handler to it, and then Invoke it with the args parameter from your Main method.

For nuke-from-orbit the command looks like this:

 1var command = new RootCommand
 2{
 3    new Option<bool>(new[]{"--yes", "-y"}, () => false,
 4        "Don't ask for confirmation, just nuke it."),
 5    new Option<bool>(new[]{"--dry-run", "-n"}, () => false,
 6        "List items that will be nuked but don't nuke them."),
 7    new Argument<string>("workingDirectory", () => Environment.CurrentDirectory)
 8};
 9
10command.Description = "Dust off and nuke bin and obj directories from orbit. It's the only way to be sure.";

It’s using C#’s list initializer syntax so you can add as many Options and Arguments as you need. The Arguments will be positional if there’s more than one, in the order that they’re added in the list. The Options can be anywhere, before or after the Arguments.

For each option, you can specify an array of aliases, like --yes or -y. You can also specify a default value for both Options and Arguments by using a Func<T> that returns the default; if you specify this, the Option or Argument will be optional.

The Description property is output in the Help, just under the name of the command.

To add a handler, we just create a CommandHandler using an Action or Func, like this:

1command.Handler = CommandHandler.Create<bool, bool, string>(
2    async (yes, dryRun, workingDirectory) =>
3    {
4        await Nuke(yes, dryRun, workingDirectory);
5    }
6);

System.CommandLine uses the names of the parameters on the delegate to match to the Option and Argument names; Options with hyphens are converted to camelCase so --dry-run matches dryRun.

Note: at the time of writing System.CommandLine is still in preview, so you’ll need to enable pre-release packages to use it in your project.

Finding bin and obj folders

Have you ever written an application that worked with files or directories and muttered dark curses at whichever API designer decided that the File and Directory classes should be static, so they’re impossible to test against? You are not alone. Rejoice! for there is a NuGet package that fixes this, too: System.IO.Abstractions provides interfaces for all the static System.IO types, along with default implementations, so you can write proper tests without contaminating the file system of whatever machine the tests run on.

1foreach (var directory in _fileSystem.Directory.EnumerateDirectories(currentDirectory))
2{
3    var name = _fileSystem.Path.GetFileName(directory);
4    if (name is null) continue;

It even allows you to mock Path so you can change things like DirectorySeparatorChar and simulate Linux or Windows in your tests.

Here’s an example using NSubstitute to mock the IDirectory interface:

 1        private static IDirectory FakeDirectory(string[] directories, IPath path, string[] files)
 2        {
 3            var directory = Substitute.For<IDirectory>();
 4            
 5            directory.EnumerateDirectories(Arg.Any<string>())
 6                .Returns(c => directories.Where(d => path.GetDirectoryName(d) == c.Arg<string>()).Distinct());
 7            
 8            directory.EnumerateFiles(Arg.Any<string>())
 9                .Returns(c => files.Where(f => path.GetDirectoryName(f) == c.Arg<string>()).Distinct());
10            
11            directory.Exists(Arg.Any<string>()).Returns(true);
12            
13            return directory;
14        }

The other really nice thing about System.IO.Abstractions is that you can add extension methods to the interfaces, like this one to detect whether the file system is case sensitive:

 1public static class FileSystemExtensions
 2{
 3    public static bool IsCaseSensitive(this IFileSystem fileSystem, string directory)
 4    {
 5        if (!fileSystem.Directory.Exists(directory))
 6            throw new InvalidOperationException("Directory does not exist.");
 7        
 8        if (directory.Where(char.IsLetter).Any(char.IsLower))
 9        {
10            if (fileSystem.Directory.Exists(directory.ToUpper())) return false;
11        }
12        else
13        {
14            if (fileSystem.Directory.Exists(directory.ToLower())) return false;
15        }
16
17        return true;
18    }
19}

Not deleting Git-controlled files

There are bad people in the world. People who hate puppies, eat your last Jaffa Cake, and add files in their bin and obj directories to source control. I’m sure they have very good reasons for doing that, but they don’t. Let’s humour them anyway, and check whether and bin or obj files are in Git before deleting them.

From the command line, you can get a complete list of files that are version controlled using the git ls-files command. So we can run that, find any entries with /bin/ or /obj/ in them, and add them to some kind of protected list.

I don’t know if you’ve ever written the code to run a process and capture its output, but it’s more complicated than it should be. You have to remember to redirect standard output, and not use ShellExecute (or is it use ShellExecute? I can never remember), and then handle the events and everything. It’s just annoying.

Enter another great NuGet package: CliWrap. This package wraps all that ProcessInfo and Process.Start business up in a really well-designed fluent API and takes care of remembering the hard stuff for you.

Here’s a call to git ls-files using CliWrap:

 1private async Task<HashSet<string>> ListFileAsync()
 2{
 3    var set = new HashSet<string>(_stringComparer);
 4
 5    var result = await Cli.Wrap("git")
 6        .WithArguments("ls-files")
 7        .WithWorkingDirectory(_workingDirectory)
 8        .WithValidation(CommandResultValidation.None)
 9        .WithStandardOutputPipe(PipeTarget.ToDelegate(line =>
10        {
11            line = _fileSystem.Path.Normalize(line);
12            
13            if (line.Contains(Bin) || line.Contains(Obj))
14            {
15                set.Add(Path.Combine(_workingDirectory, line));
16            }
17        }))
18        .ExecuteAsync();
19
20    if (result.ExitCode != 0)
21    {
22        set.Clear();
23    }
24    return set;
25}

That WithStandardOutputPipe call works with PipeTarget objects: there are implementations built-in for delegates, streams and StringBuilder and you can implement your own, too.

Note the .WithValidation(CommandResultValidation.None) line: the default behaviour for CliWrap is to throw an exception if the command returns a non-zero exit code, and we’re expecting that might happen so we want to avoid an exception. With CommandValidation.None, if git isn’t installed, or the working directory is not a git repo, then result.ExitCode will be non-zero and we can just return an empty HashSet<string>.

Making it a tool

A dotnet tool is just a NuGet package with a bit of extra metadata specified in the project file:

1  <PropertyGroup>
2    <PackAsTool>true</PackAsTool>
3    <ToolCommandName>nuke-from-orbit</ToolCommandName>
4    <PackageId>RendleLabs.NukeFromOrbit</PackageId>
5  </PropertyGroup>

The PackAsTool property sets the tool flag in the NuGet package, and the ToolCommandName is the name used to invoke the command. I’d normally go for something a bit shorter, but I’m generally so annoyed by the time I need this one that I want the satisfaction of typing nuke-from-orbit and smashing the Enter key.

Publishing to NuGet

There are a bunch of ways to publish NuGet packages these days, but I wanted to try GitHub Actions. This involves adding a workflow file to your git repo which is, of course, YAML. This is the workflow for NukeFromOrbit, in a file at .github/workflows/dotnet-core.yml:

 1name: .NET Core
 2
 3on:
 4  push:
 5    branches: [ master ]
 6  pull_request:
 7    branches: [ master ]
 8
 9jobs:
10  build:
11
12    runs-on: windows-latest
13
14    steps:
15    - uses: actions/checkout@v2
16      with:
17        fetch-depth: 0
18    - name: Setup .NET Core
19      uses: actions/setup-dotnet@v1
20      with:
21        dotnet-version: 3.1.301
22    - name: Install dependencies
23      run: dotnet restore
24    - name: Nerdbank.GitVersioning
25      id: nbgv
26      uses: dotnet/nbgv@v0.3.1
27    - name: Build
28      run: dotnet build --configuration Release --no-restore
29    - name: Test
30      run: dotnet test --configuration Release --no-build --verbosity normal test\NukeFromOrbit.Tests
31    - name: Pack
32      run: dotnet pack --configuration Release --no-build -p:PackageVersion=${{ steps.nbgv.outputs.SimpleVersion }} --output . src\NukeFromOrbit\NukeFromOrbit.csproj
33    - name: Push
34      run: dotnet nuget push *.nupkg -s https://api.nuget.org/v3/index.json -k ${{ secrets.NUGET_API_KEY }} --skip-duplicate

This is mostly the default .NET Core workflow template, with a couple of additions.

NerdBank.GitVersioning

This step takes care of assigning version numbers to assemblies, using the “height” of the git history (i.e. the number of commits) as the revision number, and taking the major and minor version numbers from a version.json file in the root of the repo.

1{
2  "$schema": "https://raw.githubusercontent.com/dotnet/Nerdbank.GitVersioning/master/src/NerdBank.GitVersioning/version.schema.json",
3  "version": "1.0",
4  "assemblyVersion": {
5    "precision": "revision"
6  }
7}

The assemblyVersion property tells NBGV to include the revision in the AssemblyVersion value; without this the default is just {version}.0.0. I want it to include the revision because System.CommandLine automatically adds a --version flag that prints the AssemblyVersion value.

The workflow uses an output variable from the NBGV step to set the package version in the dotnet pack step.

NerdBank.GitVersioning is in the GitHub Actions Marketplace, pulled in with the uses: dotnet/nbgv@v0.3.1 line.

NUGET_API_KEY

GitHub Actions provides a Secrets store where you can put API keys, passwords and the like. You can store Secrets at the repo level, or at the organisation level, where you can specify which repos the secrets are available to.

That’s it!

Hopefully this post has shown you how easy it is to publish your own dotnet tool, from dotnet new console to CI/CD. If you have any simple tools lying around then why not share them? And if you do, be sure to drop by github.com/natemcmaster/dotnet-tools and submit a PR to let people know about it.

The source code for the tool, including the workflow stuff, is at github.com/RendleLabs/NukeFromOrbit. Let me know if you like it, or open an issue or submit a PR if it needs fixing.

The problem is, when you use MSBuildWorkspace in a .NET 4.x project, it uses the old, full-fat .NET version of MSBuild. But when you use it in a .NET Core 3.1 project, it uses .NET Core’s version of MSBuild, and they’re not compatible. Specifically, because the MSBuildWorkspace internals fully parse the project file and resolve all the targets and everything, when .NET Core’s MSBuild tries to load the .NET assemblies used in old projects’ targets, it all kind of explodes.

1Msbuild failed when processing the file 'D:\ReCode\Samples\Hotel\Hotel\Hotel.csproj' with message: 
2C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets: (1489, 5): 
3The "AssignProjectConfiguration" task could not be instantiated from 
4"Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
5Method not found: 'Byte[] System.AppDomainSetup.GetConfigurationBytes()'.

Why this is annoying me

This is annoying me because I love .NET Core and I want to use it for everything. Right now, the Visual ReCode engine is all in the same solution as the Visual Studio extension, which means it’s written in C# 7.3 for .NET 4.7.2, and there’s a whole world of useful stuff I don’t have access to: nullable reference types, proper Span support, IAsyncEnumerable, and even a bunch of new APIs that I’d forgotten were new to .NET Core like Path.GetFullPath.

So I’m extracting the engine from the extension, with the plan to run it as an out-of-process server, using (of course) gRPC to talk to the VS extension. Doing this also solves a bunch of other problems like getting out of Visual Studio’s weird Task scheduler, running in a 32-bit process alongside whatever other extensions are installed, and having to spin up a whole new VS instance to do a little debugging.

But my .NET Core 3.1 gRPC app can’t use MSBuildWorkspace to load old-style projects, like, oh, WCF projects, which ReCode kinda needs to work with.

A solution

There is another Workspace implementation: AdhocWorkspace. This is a much more tolerant and forgiving workspace, but it takes a little more effort to use. I already use it extensively in unit tests, where I create solutions, projects and documents in memory from strings. The tests happily pass when running in .NET Core 3.1, so I thought, why don’t I just create an AdhocWorkspace from an actual solution and its projects? All I need to do is parse the .sln file and the .csproj files and build the thing up manually.

It’s a well-known fact that the Visual Studio solution file is weird and hard to parse, but if all you’re doing is looking for projects it’s actually not that bad. A Project entry looks like this:

1Project(""{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}"") = ""Hotel"", ""Hotel\Hotel.csproj"", ""{0AB9EB14-38A1-40FD-B093-B756E9679FE5}""
2EndProject

Se we can just look for lines that start with Project, skip past the = symbol and grab the next two quoted values, which are the name of the project and the path relative to the solution file.

With the path to the csproj file we can open that up and parse it using XDocument. If the project is the old-style csproj, the top-level <Project> element will have an xmlns attribute, which we’ll need to use to find elements.

1var itemGroupName = XName.Get("ItemGroup", xmlns);
2var compileName = XName.Get("Compile", xmlns);

All the C# files (or “Documents” as Roslyn calls them) will be in <Compile> elements

1<ItemGroup>
2  <Compile Include=""HotelService.svc.cs"">
3    <DependentUpon>HotelService.svc</DependentUpon>
4  </Compile>
5  <Compile Include=""IHotelService.cs"" />
6  <Compile Include=""Properties\AssemblyInfo.cs"" />
7</ItemGroup>

We also need to handle references, both to other projects in the solution, and to .NET assemblies which might be in a packages folder, the Global Assembly Cache, or just stored somewhere on disk. Project references are in <ProjectReference> elements, and all the others, including to NuGet packages, are in <Reference> elements, with a <HintPath> element if they’re not in the GAC.

Loading these into an AdhocWorkspace is done using Roslyn’s *Info classes, which are like builders for Solutions, Projects and Documents. For each document, we create a DocumentInfo object:

1var di = DocumentInfo.Create(DocumentId.CreateNewId(projectId), documentSource.Name,
2    filePath: documentSource.Path,
3    loader: new FileTextLoader(documentSource.Path, null));

With a list of all the documents, as well as the Project and Assembly references we got from the project file, we can create the ProjectInfo object:

1return ProjectInfo.Create(projectId, VersionStamp.Default, projectSource.Name, projectSource.Name, LanguageNames.CSharp,
2    projectSource.Path,
3    projectReferences: projectReferences,
4    metadataReferences: metadataReferences,
5    documents: documents);

And with all our ProjectInfo objects we can create a SolutionInfo and add it to our AdhocWorkspace:

1var workspace = new AdhocWorkspace();
2var solutionInfo = SolutionInfo.Create(SolutionId.CreateNewId(),
3                    VersionStamp.Default, solutionFilePath, projectInfos);
4workspace.AddSolution(solutionInfo);

This will turn all those Infos into actual Roslyn projects and documents that we can poke around in using SyntaxNodes and Symbols in our .NET Core 3.1 application.

New “SDK-style” Projects

Some projects in the solution might be the new SDK-style that came out of .NET Core but can be used for .NET 4.x projects as well. These are much simpler than the old-style projects, but actually harder to work with in this instance because all the default references (e.g. mscorlib, System.Runtime, etc) are inferred from the SDK. In this instance you can actually use MSBuildWorkspace to load the project up and then just grab the Document, ProjectReference and MetadataReference info and copy it into the AdhocWorkspace so everything works together.

It’s on NuGet

I’ve wrapped all this up in a NuGet package, RendleLabs.LegacyWorkspaceLoader, and the source code is at github.com/RendleLabs/LegacyWorkspaceLoader.

In its current state, it works for my use case, but I’m sure there are edge cases that it doesn’t catch. If I run into any while using it in Visual ReCode I’ll fix them up and update the package, but if you take it for a spin and run into problems, please create an Issue on the GitHub repo or send a pull request.

One of the big things from C# 8 that I’m missing is IAsyncEnumerable<T> and await foreach. These would be incredibly useful for getting a stream of values from a background thread and updating the UI. But I can’t use them, so instead I turned to System.Threading.Channels, which scratched a similar itch before C# 8 came along.

If you’re not familiar, System.Threading.Channels is a new-ish feature of .NET designed for inter-thread messaging. It’s a really nice abstraction compared to things like BlockingCollection.

Steve Gordon wrote a great Introduction to Channels article that you should read if you want to know more about them.

The fun thing I’ve found is that you can treat a ChannelReader<T> almost exactly like an IAsyncEnumerable<T>, and write useful extension methods over it. For example, while writing unit tests for a method that returns a ChannelReader<Thing>, I wanted to collate all the values it returned into a List<Thing>. The code was so obviously generic that I refactored it into this extension method:

 1internal static class ChannelReaderExtensions
 2{
 3    public static async Task<List<T>> ToListAsync<T>(this ChannelReader<T> channelReader, CancellationToken cancellationToken = default)
 4    {
 5        var list = new List<T>();
 6        while (await channelReader.WaitToReadAsync(cancellationToken))
 7        {
 8            while (channelReader.TryRead(out var item))
 9            {
10                list.Add(item);
11            }
12        }
13
14        return list;
15    }
16}

You could pretty much write any of the IEnumerable<T> extension methods over ChannelReader<T>, although you’d be creating new Channels and piling things on thread pools a lot, so, you know… caveat developer. But anyway, here’s a Where extension method as an example.

 1public static ChannelReader<T> Where<T>(this ChannelReader<T> channelReader,
 2    Func<T, bool> predicate, CancellationToken token)
 3{
 4    var output = Channel.CreateUnbounded<T>(new UnboundedChannelOptions {SingleWriter = true});
 5    var writer = output.Writer;
 6    Task.Run(async () =>
 7    {
 8        while (await channelReader.WaitToReadAsync(token))
 9        {
10            while (channelReader.TryRead(out var item))
11            {
12                if (predicate(item))
13                {
14                    if (!writer.TryWrite(item))
15                    {
16                        await writer.WriteAsync(item, token);
17                    }
18                }
19            }
20        }
21
22        writer.TryComplete();
23    });
24    return output.Reader;
25}