Shipping breaking changes

Breaking changes are always work for your users. Work you are forcing them to do when they upgrade to your new version. They took a dependency on your library or software because it saved them time but now it’s costing them time.

Every breaking change is a reason for them to stop and reconsider their options. If your library is paid-for or the preferred way for paying users to access your services then lost users can come with a real financial cost.

Use breaking changes sparingly.

Good reasons for breaking changes

  • A feature is being removed for business reasons
  • Standards are changing how something works
  • The feature did not work as intended and it’s impossible to fix without a break
  • Service or library dependencies you rely upon are forcing a change
  • A small breaking change now prevents actual significant breaking changes later

Even when presented with these you should think not only about whether you can avoid a break but also take the opportunity to think about what you can do now to avoid similar breaks in the future.

Poor reasons for breaking changes

  • It makes the internals of the library tidier
  • Parameter order, method naming or property naming would be ”clearer”
  • Consistency with other platforms or products
  • Personal subjective interpretations of “better”
  • Compatibility with a different library to attract new users

While many of these are admirable goals in of themselves they are not a reason to break your existing customers.

Managing breaking changes

It goes without saying that intentional breaking changes should only occur in major versions with the exception of security fixes that require users of your library take some action.

Here are some thoughts to ease the pain of breaking changes:

  • List each breaking change in a migration guide with a before and after code fragment
  • Summarize breaking changes in the README with a link to the migration guide for more information
  • Keep breaking change count low even on major releases

Users should ideally also be able to find/replace or follow compiler errors. Consider:

  • Platform-specific mechanisms for dealing with breaking changes. e.g. in C# you can use the [Obsolete] attribute to help guide to the replacement API, Java has the @deprecated annotation.
  • Leaving a stub for the old method in place for one whole major release that calls the new method with the right arguments and produces a log warning pointing to the migration guide.


If a package is drastically different users will need to rewrite code. This is always a bigger deal for them than you expect, because:

  1. It is unscheduled and was not on their radar (no they are not monitoring your GitHub issue discussions)
  2. They use your library in ways you likely don’t expect or anticipate
  3. The more they depend on your product then the more work your rewrite involves

Really consider whether your new API is actually better (ideally before you ship it). One way to do this is to produce a set of example usage code for using the old library vs the new library. Put them side-by-side and open them up to feedback. Is the new API genuinely better? Some indicators it might be are:

  • Less code for users to write
  • Plenty of sensible and safe defaults
  • Less specialized terminology and concepts
  • Easier to test and abstract
  • Existing customers prefer the new syntax and think it’s worth changing

Some indicators that it isn’t really any better is: internal staff prefer it, it aligns better with some other platform or just being different.

Sometimes it’s worth doing because it targets a different crowd or comes at the problem from a simpler direction or abstraction. If so, then seriously consider giving it a new package name and put it out for early access.

Make sure users are led to the right library of the two and if there is a lot of code duplication and users on the “old” library then consider making the next version of the old library a compatibility wrapper around the new library.


Mac OS System 9 on Windows

I’m often digging into old bitmap font and UX design out of curiosity - and someday hope to revive a lot of these fonts in more modern formats using a pipeline similar to that for ZX Origins so we can get all the usable fonts, screenshots etc. out of them.

One limitation I’ve run into is digging into old Macintosh fonts. While James Friend’s PCE.js puts System 6 and System 7 at your fingertips when it comes to later 7.5, 8 or 9 the site doesn’t have you covered as PCE doesn’t support PowerPC emulation (it handles Motorola 68000 and Intel 8086 processors).

This is a shame for me as that’s where the interface started diverging by adding color and some more interesting fonts. Additionally some third-party fonts are distributed in .sit (StuffIt Expander) or only work with later Mac OS versions.

Enter QEMU

Thankfully QEMU has us covered. It’s an open-source emulator that unlike regular virtualization tools is quite capable of emulating completely different CPU architectures from ARM through to MIPS, PowerPC, RISC-V, Sparc and even IBM’s big s390x z/Architecture.

With such a wide variety of options and settings available you can imagine it will require some digging through the user interface and you’d be wrong. There is no GUI however and the third party ones that exist mostly seem to be from 10 to 2 years out of date and many don’t support Windows at all. The only “up to date” one I found - QtEmu - only supports configuring x86 virtual machines.

This is a shame as although I love the command-line for its scriptability when it comes to exploring valid combinations of options the command-line is mostly awful (the IBM AS/400 command-line and prompting system excluded).

You could try and build this yourself but Stefan Weil has you covered for pre-built QEMU Windows binaries

Please note that Mac sound support is missing here. There are “screamer” forks but the only binaries available are for Mac OS X so you’d have to build it yourself and there will probably be a whole lot of hoops to jump through.

Obtaining an OS install image

To install Mac OS 9 we’re going to need a disk image/ISO to install from.

The nice people over at Mac OS 9 Lives have a Mac OS 9.2.2 Universal Installer ISO which is pre-configured and easy to use - it also conveniently includes a few extra tools and apps you’ll need.

While Mac OS is copyrighted this image has been up for over 6 years so I like to think Apple are turning a blind eye in that people who want to use their legacy stuff can do so without expecting support from Apple - win-win. They also don’t charge for their Operating Systems instead it’s “free” with the hardware and I still have a MacBook Pro 15” so I won’t feel bad about using it. Your mental mileage may vary.

Creating a machine & installing Mac OS 9

First off create a new folder to put your machine config into. (Windows won’t let you stuff it into Program Files). I’ve chosen c:\retro\mac.

Now let’s create an empty hard-drive image file:

cd "c:\program files\qemu"
qemu-img create -f qcow2 c:\retro\mac\MacOS9HD.img 5G

This creates a virtual hard-drive that can grow up to 5 GB in size and will allocate disk as it needs (copy-on-write). After installation this file will grow to about 660 MB.

Now before we go further remember the keyboard-shortcuts:

  • CtrlAltG as you’ll need it to get the mouse back
  • CtrlAltF to get you in and out of full-screen mode

Now lets start the installer:

qemu-system-ppc -cpu "g4" -M mac99,via=pmu -m 512 -hda c:/retro/Mac/MacOS9HD.img -cdrom "c:/retro/Mac/Mac OS 9.2.2 Universal Install.iso" -boot d -g 1024x768x32 -device usb-kbd -device usb-mouse -sdl

This specifies that we want to use:

  • a PowerPC G4 900MHz CPU -cpu g4
  • a PowerMAC based Mac with USB support -M mac99,via=pmu
  • 512 MB of RAM -m 512
  • Our hard-drive image -hda xxx
  • Our Installation CD mounted -cdrom xxx
  • Boot from CD -boot d
  • 1024x768 32-bit display -g 1024x768x32
  • a USB keyboard -device usb-kbd
  • a USB mouse -device usb-mouse
  • SDL display buffer -sdl

There are many other useful config switches available for the PowerPC emulation if you need to troubleshoot or tweak. The final item, SDL, is required because the default GTK emulation, while faster, has major problems on Windows trying to keep the mouse captured.

You should now be presented after a few seconds with a ReadMe. Just close that with the top-left window control then click into the Drive Setup window, select <not initialized>, press the Initialize button then confirm it with the default on the subsequent Initialize window.

Screenshot of Mac OS 9 initializing a drive inside QEMU on Windows

You might now want to click the “untitled” hard drive icon that’s appeared on the desktop, wait a few seconds and you should be able to rename it. Typically Macintosh HD is a popular choice.

Head up to the MacOS9Live CD icon, double click it then double-click on Apple Software Restore. The following Window will appear. You can accept all the defaults or just change Volume Format to Extended - I did this just in case I want to try and mount the image on my MacBook at a later date.

Screenshot of Mac OS 9 Apple Software Restore window inside QEMU on Windows

Clicking Restore, then confirming the dialog will give you a progress bar that is comically fast for installing an operating system (via software emulation no less).

Screenshot of Mac OS 9 Apple Software Restore window inside QEMU on Windows

Now you’ll need to head to the Special menu and choose Shut Down.

Using our virtual Mac OS 9

Finally, we want to start our freshly created machine without booting from the ISO. The command line is mostly the same just omitting the ISO and boot-from-CD options:

qemu-system-ppc -cpu g4 -M mac99,via=pmu -m 512 -hda c:/retro/Mac/MacOS9HD.img -device usb-kbd -device usb-mouse -sdl

You’ll probably want to put that in a shortcut icon.

Here's a PowerMac G4 "Quicksilver" Icon (128 KB) I put together.

When it boots for the first time you’ll get a Register With Apple “wizard”. Just press WindowsQ to quit this and get to that Platinum desktop!

Head to the Control Panel’s Monitors applet to set the screen size/resolution you want. You might also want to head into Appearance applet’s Fonts tab to turn off anti-aliasing so you can enjoy the fonts in their pixel-glory. (You can also switch from the revised Charcoal front back to the classic Chicago font here). You may also have to switch screen-resolution again if you see some odd artifacts/missing/doubled pixels when turning it off. (There’s a quick resolution changer on the control-strip in the lower left, it’s the one with the checkerboard effect)

Remember to always shut-down correctly! Use the switcher at the top-right to “switch” to Finder then go through that Special, Shutdown process each time. QEMU will close several seconds after it’s complete.

On your hard-drive you’ll find an Applications folder, dig into Internet Utilities, Classilla folder and you can launch Classilla which is a port of the Netscape browser made in 2014 (based on Netscape Navigator 1.3.1 Nokia N90 port). It was a valiant effort given how diferent Mac OS development was prior to Mac OS X - there were no Unix libraries/support so ports were difficult and most applications were written in MetroWerks CodeWarrior - the “classic” Mac OS version was discontinued in 2002.

Still Classilla is much better than IE 5.5 which fails to do anything at all. Google works, for example, but many sites don’t render at all because of the push to later versions of SSL the browser does not support.

Screenshot of Classica browser searching for me and also showing quick-resolution switching on Mac OS 9 inside QEMU on Windows

You can find a ton of old Mac software at The Macintosh Repository but there are no more capable browsers.

Still, it’s a fun environment to play with and it’s nice to have 100% accurate references to Geneva, Chicago, Monaco, Espy Sans etc. as most “conversions” tend to be hand-converted and mistakes are a-plenty. I’ve done a few conversions myself this way on FontStruct and know how easy it is to make mistakes when working from screenshots especially when it comes to spacing between letters.

Screenshot of Geneva font on Mac OS 9 inside QEMU on Windows

It’s also nice to see an old friend again. Despite regularly finding myself on retro machines and emulators spanning 8 and 16 bit machines I don’t have (or have the space for) a classic Mac and emulation has been difficult. I think I last used Mac OS 9 in 2000 on an iMac at work before we put the Mac OS X Public Beta on it (I was a bit NeXT/OpenStep fan and wanted to see what they had done to it!)

My thanks to James Badger for his general article on Mac OS 9 on QEMU.


Creating OR expressions in LINQ

As everybody who has read my blog before knows, I love LINQ and miss it when coding in other languages, so it’s nice when I get a chance to use it again. When I come back to it with fresh eyes, I notice some things aren’t as easy as they should be - and this time is no exception.


People often need to build up LINQ expressions at runtime based on filters or criteria a user has selected. Adding criteria is incredibly easy, as you can chain operations together on the IQueryable interface, e.g.

if (customerActive)
  query = query.Where(c => c.IsActive);
if (customerCountry != null)
  query = query.Where(c => c.Country == customerCountry);

This is great if you want to AND things together, which is often the case in building up filters. Other operations, like Contains, are useful in allowing many options against a specific field.


But what about offering a choice involving either of two properties? One way to write it would be:

if (customerActive) {
  if (customerEnterprise) {
    query = query.Where(c => c.IsActive || c.IsEnterprise);
  } else {
    query = query.Where(c => c.IsActive);
  } else {
    if (customerEnterprise) {
      query = query.Where(c => c.IsEnterprise);

This code is already hard to read and is going to increase in complexity for each new option.


With some helper code, we can combine two expression predicates into a single OrElse for use in a Where condition.

class Or {
  public static Expression<Func<T, bool>> Combine<T>(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right) {
    if (left == null && right == null) throw new ArgumentException("At least one argument must not be null");
    if (left == null) return right;
    if (right == null) return left;
    var parameter = Expression.Parameter(typeof(T), "p");
    var combined = new ParameterReplacer(parameter).Visit(Expression.OrElse(left.Body, right.Body));
    return Expression.Lambda<Func<T, bool>>(combined, parameter);

  class ParameterReplacer : ExpressionVisitor {
    readonly ParameterExpression parameter;

    internal ParameterReplacer(ParameterExpression parameter) {
      this.parameter = parameter;

    protected override Expression VisitParameter(ParameterExpression node) {
      return parameter;

Now we can:

Expression<Func<Customer, bool>> criteria = null;
if (customerActive)
    criteria = Or.Combine(criteria, c => c.IsActive == customerActive);
if (customerEnterprise)
    criteria = Or.Combine(criteria, c => c.IsEnterprise == customerEnterprise);

How it works

There are two parts to this working. The first combines the two separate predicates into a single OR lambda, taken care of by the Expression.OrElse method.

That alone can’t quite do what we need because the two expressions each have a separate parameter variable. The visitor replaces both of these with a new parameter we’ll pass into the new Lambda expression we’re creating that combines both conditions.


The code, as it stands, creates an unbalanced tree. Ideally, the tree should be balanced to limit the depth LINQ providers must traverse to translate the query. I doubt you’ll run into the dreaded Stack Overflow (the message, not the site), but if you do, that’s why.

An overload that takes an Enumerable of Expression<Func<T, bool>> and produces a balanced tree is left as an exercise to the reader :p


From CircleCI to GitHub Actions for Jekyll publishing

I’ve been a big fan of static site generation since I switched from WordPress to Jekyll back in 2018. I’m also a big fan of learning new technologies as they come along, and now GitHub Actions are out in the wild; I thought this would be an opportunity to see how I can port my existing custom CircleCI build to Jekyll.

The CircleCI job

A quick recap from part 5 - Hosting & Building, my CircleCI configuration was basically two jobs that have subsequently been tweaked since then. They are:


The build job’s responsibility was to configure a Ruby environment capable of executing Jekyll to build the site, removing the .html extension from the output filenames, and then indexing the content using Algolia. Here is the start of the .circleci/config.yml I was using for that:

version: 2
      - image: circleci/ruby:2.6.1
    working_directory: ~/jekyll
      - JEKYLL_ENV=production
      - JOB_RESULTS_PATH=run-results
      - checkout
      - restore_cache:
          key: jekyll-{{ .Branch }}-{{ checksum "Gemfile" }}
      - run:
          name: Update gems
          command: gem update --system
      - run:
          name: Install dependencies
          command: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
      - save_cache:
          key: jekyll-{{ .Branch }}-{{ checksum "Gemfile" }}
            - "vendor/bundle"
      - run:
          name: Create results directory
          command: mkdir -p $JOB_RESULTS_PATH
      - run:
          name: Build site
          command: bundle exec jekyll build --config _config.yml,_config-publish.yml 2>&1 | tee $JOB_RESULTS_PATH/build-results.txt
      - run:
          name: Remove .html suffixes
          command: find _site -name "*.html" -not -name "index.html" -exec rename -v 's/\.html$//' {} \;
      - run:
          name: Index with Algolia
          command: bundle exec jekyll algolia --config _config.yml,_config-publish.yml
      - store_artifacts:
          path: run-results/
          destination: run-results
      - persist_to_workspace:
          root: ~/jekyll
            - _site


Deploy takes the output from the build job and syncs it with the S3 bucket I use to publish the site. It then applies AWS S3-specific commands using the AWS CLI tool to ensure metadata, redirects, and caching are correctly set using a Python environment.

      - image: circleci/python:2.7
    working_directory: ~/jekyll
      - attach_workspace:
          at: ~/jekyll
      - run:
          name: Install AWS CLI
          command: sudo pip install awscli
      - run:
          name: Deploy to S3
          command: aws s3 sync _site s3://damieng-static/ --delete --content-type=text/html
      - run:
          name: Correct MIME for robots.txt automatically
          command: aws s3 cp s3://damieng-static/robots.txt s3://damieng-static/robots.txt --metadata-directive="REPLACE"
      - run:
          name: Correct MIME for sitemap.xml automatically
          command: aws s3 cp s3://damieng-static/sitemap.xml s3://damieng-static/sitemap.xml --metadata-directive="REPLACE"
      - run:
          name: Correct MIME for Atom feed manually
          command: aws s3 cp s3://damieng-static/feed.xml s3://damieng-static/feed.xml --no-guess-mime-type --content-type="application/atom+xml" --metadata-directive="REPLACE"
      - run:
          name: Redirect /damieng for existing RSS subscribers
          command: aws s3api put-object --bucket damieng-static --key "damieng" --website-redirect-location ""
      - run:
          name: Latest Envy Code R redirect
          command: aws s3api put-object --bucket damieng-static --key "envy-code-r" --website-redirect-location ""
      - run:
          name: Latest Envy Code R redirect #2
          command: aws s3api put-object --bucket damieng-static --key "fonts/envy-code-r" --website-redirect-location ""
      - run:
          name: Latest Envy Code R download
          command: aws s3api put-object --bucket damieng-static --key "downloads/latest/EnvyCodeR" --website-redirect-location ""
      - run:
          name: Correct MIME for CSS files
          command: aws s3 cp s3://damieng-static/css/ s3://damieng-static/css/ --metadata-directive="REPLACE" --recursive

GitHub Actions

So how could I go about this in GitHub Actions? I have to admit I spent far too long poking around and examining existing Jekyll actions. I have a bunch of steps here I need fine control of, especially around Algolia and S3. I finally ended up on what was quite a simple port.

Unlike the CircleCI configuration, I did not split these into two separate jobs because:

  1. There is no exact equivalent to persist_to_workspace and attach_workspace
  2. The alternative of storing and restoring the artifacts leaves large useless artifacts around
  3. I never ended up running the jobs separately
  4. GitHub Actions provides an environment with both Ruby and AWS CLI installed

So, on to the configuration which lives in .github/workflows/jekyll.yml in my case:

name: Build site and deploy

    branches: [ master ]

    name: Build + Deploy
    runs-on: ubuntu-latest
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
          ruby-version: 2.6.1

      - name: Ruby gem cache
        uses: actions/cache@v1
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-
      - name: Install dependencies
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

      - name: Build Jekyll site
        run:  bundle exec jekyll build --config _config.yml,_config-publish.yml

      - name: Remove .html suffixes except for index.html
        run: find _site -name "*.html" -not -name "index.html" | while read f; do mv "$f" "${f%.html}"; done                      

      - name: Index with Algolia
          ALGOLIA_API_KEY: ${{ secrets.ALGOLIA_API_KEY }}
        run:  bundle exec jekyll algolia --config _config.yml,_config-publish.yml
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - name: Sync site with S3
        run:  aws s3 sync _site s3://damieng-static/ --delete --content-type=text/html

      - name: Correct MIME types
        run: |
          aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.txt" --metadata-directive="REPLACE"
          aws s3 cp s3://damieng-static/sitemap.xml s3://damieng-static/sitemap.xml --metadata-directive="REPLACE"
          aws s3 cp s3://damieng-static/feed.xml s3://damieng-static/feed.xml --no-guess-mime-type --content-type "application/atom+xml" --metadata-directive "REPLACE"
          aws s3 cp s3://damieng-static/css/ s3://damieng-static/css/ --metadata-directive "REPLACE" --recursive
          aws s3 cp s3://damieng-static/js/ s3://damieng-static/js/ --metadata-directive "REPLACE" --recursive
      - name: Redirect /damieng for existing RSS subscribers
        run:  aws s3api put-object --bucket damieng-static --key "damieng" --website-redirect-location ""

      - name: Latest Envy Code R redirects
        run:  |
          aws s3api put-object --bucket damieng-static --key "fonts/envy-code-r" --website-redirect-location ""
          aws s3api put-object --bucket damieng-static --key "downloads/latest/EnvyCodeR" --website-redirect-location ""
      - name: Set caching for images at 30 days
        run:  aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.svg" --include "*.ico" --include "*.jpg" --include "*.png" --include "*.webp" --include "*.gif" --recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

      - name: Set caching for CSS and JS at 1 hour
        run:  aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.css" --include "*.js" --recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=3600,public

I also took the opportunity to fix a long-running issue in that my S3 objects would lose my manually-applied cache settings (and new posts and files would not have any).


There were only a couple of bumps in the road once I decided on a straight-port rather than trying to leverage higher-level existing actions:

  1. The secrets configured in the repo settings were not automatically exposed to the commands running in the action. Instead, you have to expose them using the ${{ secrets.KEY_NAME }} syntax.
  2. I was using rename instead of mv. I don’t recall why. Perhaps it was my Windows-ness creeping in. Rename has been dropped in newer distros.


The syntax is surprisingly similar with build-times about the same as CircleCI. It’s just nice to have it in one place. Time to port some other repos over!