As everybody who has read my blog before knows, I love LINQ and miss it when coding in other languages, so it's nice when I get a chance to use it again. When I come back to it with fresh eyes, I notice some things aren't as easy as they should be - and this time is no exception.

Background

People often need to build up LINQ expressions at runtime based on filters or criteria a user has selected. Adding criteria is incredibly easy, as you can chain operations together on the IQueryable interface, e.g.

if (customerActive)
  query = query.Where(c => c.IsActive);
if (customerCountry != null)
  query = query.Where(c => c.Country == customerCountry);

This is great if you want to AND things together, which is often the case in building up filters. Other operations, like Contains, are useful in allowing many options against a specific field.

Problem

But what about offering a choice involving either of two properties? One way to write it would be:

if (customerActive) {
  if (customerEnterprise) {
    query = query.Where(c => c.IsActive || c.IsEnterprise);
  } else {
    query = query.Where(c => c.IsActive);
  } else {
    if (customerEnterprise) {
      query = query.Where(c => c.IsEnterprise);
    }
  }
}

This code is already hard to read and is going to increase in complexity for each new option.

Solution

With some helper code, we can combine two expression predicates into a single OrElse for use in a Where condition.

class Or {
  public static Expression<Func<T, bool>> Combine<T>(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right) {
    if (left == null && right == null) throw new ArgumentException("At least one argument must not be null");
    if (left == null) return right;
    if (right == null) return left;

    var parameter = Expression.Parameter(typeof(T), "p");
    var combined = new ParameterReplacer(parameter).Visit(Expression.OrElse(left.Body, right.Body));
    return Expression.Lambda<Func<T, bool>>(combined, parameter);
  }

  class ParameterReplacer : ExpressionVisitor {
    readonly ParameterExpression parameter;

    internal ParameterReplacer(ParameterExpression parameter) {
      this.parameter = parameter;
    }

    protected override Expression VisitParameter(ParameterExpression node) {
      return parameter;
    }
  }
}

Now we can:

Expression<Func<Customer, bool>> criteria = null;
if (customerActive)
    criteria = Or.Combine(criteria, c => c.IsActive == customerActive);
if (customerEnterprise)
    criteria = Or.Combine(criteria, c => c.IsEnterprise == customerEnterprise);

How it works

There are two parts to this working. The first combines the two separate predicates into a single OR lambda, taken care of by the Expression.OrElse method.

That alone can't quite do what we need because the two expressions each have a separate parameter variable. The visitor replaces both of these with a new parameter we'll pass into the new Lambda expression we're creating that combines both conditions.

Caveats

The code, as it stands, creates an unbalanced tree. Ideally, the tree should be balanced to limit the depth LINQ providers must traverse to translate the query. I doubt you'll run into the dreaded Stack Overflow (the message, not the site), but if you do, that's why.

An overload that takes an Enumerable of Expression<Func<T, bool>> and produces a balanced tree is left as an exercise to the reader :p

[)amien

I've been a big fan of static site generation since I switched from WordPress to Jekyll back in 2018. I'm also a big fan of learning new technologies as they come along, and now GitHub Actions are out in the wild; I thought this would be an opportunity to see how I can port my existing custom CircleCI build to Jekyll.

The CircleCI job

A quick recap from part 5 - Hosting & Building, my CircleCI configuration was basically two jobs that have subsequently been tweaked since then. They are:

Build

The build job's responsibility was to configure a Ruby environment capable of executing Jekyll to build the site, removing the .html extension from the output filenames, and then indexing the content using Algolia. Here is the start of the .circleci/config.yml I was using for that:

{%raw%}version: 2
jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.1
    working_directory: ~/jekyll
    environment:
      - JEKYLL_ENV=production
      - NOKOGIRI_USE_SYSTEM_LIBRARIES=true
      - JOB_RESULTS_PATH=run-results
    steps:
      - checkout
      - restore_cache:
          key: jekyll-{{ .Branch }}-{{ checksum "Gemfile" }}
      - run:
          name: Update gems
          command: gem update --system
      - run:
          name: Install dependencies
          command: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
      - save_cache:
          key: jekyll-{{ .Branch }}-{{ checksum "Gemfile" }}
          paths:
            - "vendor/bundle"
      - run:
          name: Create results directory
          command: mkdir -p $JOB_RESULTS_PATH
      - run:
          name: Build site
          command: bundle exec jekyll build --config _config.yml,_config-publish.yml 2>&1 | tee $JOB_RESULTS_PATH/build-results.txt
      - run:
          name: Remove .html suffixes
          command: find _site -name "*.html" -not -name "index.html" -exec rename -v 's/\.html$//' {} \;
      - run:
          name: Index with Algolia
          command: bundle exec jekyll algolia --config _config.yml,_config-publish.yml
      - store_artifacts:
          path: run-results/
          destination: run-results
      - persist_to_workspace:
          root: ~/jekyll
          paths:
            - _site
{%endraw%}

Deploy

Deploy takes the output from the build job and syncs it with the S3 bucket I use to publish the site. It then applies AWS S3-specific commands using the AWS CLI tool to ensure metadata, redirects, and caching are correctly set using a Python environment.

{%raw%}  deploy:
    docker:
      - image: circleci/python:2.7
    working_directory: ~/jekyll
    steps:
      - attach_workspace:
          at: ~/jekyll
      - run:
          name: Install AWS CLI
          command: sudo pip install awscli
      - run:
          name: Deploy to S3
          command: aws s3 sync _site s3://damieng-static/ --delete --content-type=text/html
      - run:
          name: Correct MIME for robots.txt automatically
          command: aws s3 cp s3://damieng-static/robots.txt s3://damieng-static/robots.txt --metadata-directive="REPLACE"
      - run:
          name: Correct MIME for sitemap.xml automatically
          command: aws s3 cp s3://damieng-static/sitemap.xml s3://damieng-static/sitemap.xml --metadata-directive="REPLACE"
      - run:
          name: Correct MIME for Atom feed manually
          command: aws s3 cp s3://damieng-static/feed.xml s3://damieng-static/feed.xml --no-guess-mime-type --content-type="application/atom+xml" --metadata-directive="REPLACE"
      - run:
          name: Redirect /damieng for existing RSS subscribers
          command: aws s3api put-object --bucket damieng-static --key "damieng" --website-redirect-location "https://damieng.com/feed.xml"
      - run:
          name: Latest Envy Code R redirect
          command: aws s3api put-object --bucket damieng-static --key "envy-code-r" --website-redirect-location "https://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released"
      - run:
          name: Latest Envy Code R redirect #2
          command: aws s3api put-object --bucket damieng-static --key "fonts/envy-code-r" --website-redirect-location "https://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released"
      - run:
          name: Latest Envy Code R download
          command: aws s3api put-object --bucket damieng-static --key "downloads/latest/EnvyCodeR" --website-redirect-location "https://download.damieng.com/fonts/original/EnvyCodeR-PR7.zip"
      - run:
          name: Correct MIME for CSS files
          command: aws s3 cp s3://damieng-static/css/ s3://damieng-static/css/ --metadata-directive="REPLACE" --recursive
{%endraw%}

GitHub Actions

So how could I go about this in GitHub Actions? I have to admit I spent far too long poking around and examining existing Jekyll actions. I have a bunch of steps here I need fine control of, especially around Algolia and S3. I finally ended up on what was quite a simple port.

Unlike the CircleCI configuration, I did not split these into two separate jobs because:

1. There is no exact equivalent to persist_to_workspace and attach_workspace
2. The alternative of storing and restoring the artifacts leaves large useless artifacts around
3. I never ended up running the jobs separately
4. GitHub Actions provides an environment with both Ruby and AWS CLI installed

So, on to the configuration which lives in .github/workflows/jekyll.yml in my case:

{%raw%}name: Build site and deploy

on:
  push:
    branches: [ master ]

jobs:
  build:
    name: Build + Deploy
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: 2.6.1

      - name: Ruby gem cache
        uses: actions/cache@v1
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Install dependencies
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

      - name: Build Jekyll site
        run:  bundle exec jekyll build --config _config.yml,_config-publish.yml

      - name: Remove .html suffixes except for index.html
        run: find _site -name "*.html" -not -name "index.html" | while read f; do mv "$f" "${f%.html}"; done

      - name: Index with Algolia
        env:
          ALGOLIA_API_KEY: ${{ secrets.ALGOLIA_API_KEY }}
        run:  bundle exec jekyll algolia --config _config.yml,_config-publish.yml

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Sync site with S3
        run:  aws s3 sync _site s3://damieng-static/ --delete --content-type=text/html

      - name: Correct MIME types
        run: |
          aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.txt" --metadata-directive="REPLACE"
          aws s3 cp s3://damieng-static/sitemap.xml s3://damieng-static/sitemap.xml --metadata-directive="REPLACE"
          aws s3 cp s3://damieng-static/feed.xml s3://damieng-static/feed.xml --no-guess-mime-type --content-type "application/atom+xml" --metadata-directive "REPLACE"
          aws s3 cp s3://damieng-static/css/ s3://damieng-static/css/ --metadata-directive "REPLACE" --recursive
          aws s3 cp s3://damieng-static/js/ s3://damieng-static/js/ --metadata-directive "REPLACE" --recursive

      - name: Redirect /damieng for existing RSS subscribers
        run:  aws s3api put-object --bucket damieng-static --key "damieng" --website-redirect-location "https://damieng.com/feed.xml"

      - name: Latest Envy Code R redirects
        run:  |
          aws s3api put-object --bucket damieng-static --key "fonts/envy-code-r" --website-redirect-location "https://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released"
          aws s3api put-object --bucket damieng-static --key "downloads/latest/EnvyCodeR" --website-redirect-location "https://download.damieng.com/fonts/original/EnvyCodeR-PR7.zip"

      - name: Set caching for images at 30 days
        run:  aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.svg" --include "*.ico" --include "*.jpg" --include "*.png" --include "*.webp" --include "*.gif" --recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

      - name: Set caching for CSS and JS at 1 hour
        run:  aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.css" --include "*.js" --recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=3600,public
{%endraw%}

I also took the opportunity to fix a long-running issue in that my S3 objects would lose my manually-applied cache settings (and new posts and files would not have any).

Gotchas

There were only a couple of bumps in the road once I decided on a straight-port rather than trying to leverage higher-level existing actions:

1. The secrets configured in the repo settings were not automatically exposed to the commands running in the action. Instead, you have to expose them using the ${{ secrets.KEY_NAME }} syntax.
2. I was using rename instead of mv. I don't recall why. Perhaps it was my Windows-ness creeping in. Rename has been dropped in newer distros.

Summary

The syntax is surprisingly similar with build-times about the same as CircleCI. It's just nice to have it in one place. Time to port some other repos over!

[)amien

I've recently been working on a full HTML5 conversion of the Sinclair Spectrum +3 manual with full canvas-drawn screenshots and diagrams for smooth scaling/high res displays as well as some close font matching and layout as well as cross-reference links all over the place.

My ZX Spectrum +3 Manual conversion is now available!

What I wanted to show today was the odd commands, pokes and outs that were hastily scribbled down in the back of my manual over the years.

Spectrum +3/+2A specific

• You know you can type SPECTRUM to go from +3 BASIC to 48K BASIC but, did you know you can get back? RANDOMIZE USR 23354. It leaves CHR$ 163/164 as UDGs.
• Switch off the +3 drive with POKE 23399,4. In 48K mode, you can use POKE 23611,221 if it somehow stayed on.

128K specific

• 128K machines are very compatible but the last two user-defined graphics (UDGs) T and U have become the SPECTRUM and PLAY command (for switching to 48K mode and using the AY-3-8912 sound chip). You can, however, switch them back to UDGs using POKE 23611,205.
• POKE 23611,205; RANDOMIZE USR 4867; alternative way to switch from 128K to 48K.

A couple of VTX-5000 modem specific hacks:

• OUT 8189,0 to switch the 128K ROM back in.
• POKE 23611,29 also to switch from 48K to 128K mode.

All machines

• POKE 23617,n changes the input mode for INPUT prompts. 0=Alpha (C/L), 1=Extended (E), 2=Graphics (G), 4=Keyword (K).
• POKE 23658,0 turns off CAPS LOCK while POKE 23658,1 turns it on.
• POKE 23692,255 will suppress the Scroll? prompt 255 times so you can auto-scroll.
• RANDOMIZE USR 3582 to scroll the screen one line.
• RANDOMIZE USR 3330 to scroll the entire screen.
• LET A=32768: POKE 23607,A/256: POKE 23606,A-(PEEK 23607 * 256)

A few pokes to stop people messing with your BASIC programs!

• POKE 23756,n to change the first line number to n. If you set this to 0 then, it is not editable without reversing the poke! In 128K mode it lists line 0 over and over and can cause the editor to crash while adding new lines!
• POKE 23613,82 will disable the BREAK key in your BASIC program
• POKE 23613,0 causes BREAK to crash.
• POKE 23755,100 causes the program listing to be invisible on BREAK while running (use POKE 23755,0 to get it back)

Header-less tape loading and saving...

LD A,255
LD IX,@START
LD DE,@LENGTH
SCF
JP 1218 ; Save or
JP 1366 ; Load

Enjoy,

[)amien

Setting the scene

It's 1995, and a wiry-looking engineer in need of a haircut is working at a tech services company in the Channel Islands. The island is Jersey (you can see the French coastline on a clear day), and he's over here for a week or two for training from his nearby Guernsey home.

This company does everything from IBM AS/400 maintenance to custom PC software development - the developer involved was brought in and trained to work on a banking package on those AS/400s but has ended up on the PC development side through a series of improbable events.

Visual Basic 3 has been the company tool of choice for these tasks - supplemented with Microsoft Access for the data-driven projects (usually backed by Microsoft's brand-new SQL Server, which is taking over from Sybase installs). He's been helping work on the in-house "In/Out" software the company uses to track who is in the office vs. who is out on calls (this is 18 months before the first mainstream instant-messenger product - ICQ - will appear and offer similar functionality).

He's also been working on a document imaging and indexing solution (iStation) for two local government organizations (they love to make a solution for one company then tweak it for others like many ISVs), and that is the limit of his VB experience. He's faced the insurmountable wall of Borland C++ and OWL enough times to feel that when it comes to VB... "This is the way."

Arrival

I'm kicking around in the narrow row of desks - unlike the sales team; we're not important enough to get dividers between our desks to turn them into semi-cubes.

I watch and ask questions as he and a few other developers crowd around his giant 17" CRT to check it out. At first glance, it looks like Visual Basic with a few more included controls, a slicker IDE, but a more confusing language. Over the week that follows, he keeps showing us discoveries, and interest increases.

Before the week is up, I have a ZIPped copy of my own to carry home.

Back home, I get past the confusion of unit dependencies and Pascal idiosyncrasies like := for assignment and dig deeper, discovering the beauty of the VCL - Visual Component Library.

My journey is aided by a curious programming book written in a style I have never come across before or since. It is a humorous tale of a private detective also writing Delphi apps on the side. It is called the Delphi Programming Explorer and helps me get over the hump.

The VCL is hard to describe if you've never used it, but I'll give it a shot. It was basically like the VB controls, but they gave you the source code, and there was a proper hierarchy to it. Up to this point in my life, I thought Object-Oriented meant you had classes and instances of classes. I had no idea about interfaces, inheritance, or good OO design, but Delphi and the VCL introduced it.

I was hooked. When Delphi 2 came out, it was time to dig very deep and buy my copy.

Using it

Over time I pumped out an About95 control, then a Delphi Control Pack that included a font combo box, a gradient control, an improved list view, a button that popped out a menu control, a splitter control, a color combo box. Useful functionality that was otherwise missing but often seen in other non-Delphi apps.

I was shocked years later when I picked up the book Developing Custom Delphi 3 Components that my controls, and source, were included on the CD.

It received so many downloads that my local ISP got in contact and asked me very nicely, in the absence of any formal limits on bandwidth back then, to host the ZIP elsewhere. That and a few other tools I'd uploaded to my free personal site were now responsible for a significant percentage of their bandwidth. Double digits if my memory serves me correctly, and so relocated to Demon Internet.

Hot off the heels of that "success," I tried my hand at shareware again with the Envy Delphi Development pack. Polished and extended versions of the previous release now accompanied by a host more. An image view control, a database-bound label, a progress bar, shading control (like Borland C++), a tip-of-the-day control that needed nothing more than a text file of tips, a frame control to allow inset/outset and a bitmap tiling control for more texture. At this point, icons are sufficiently basic in the world that I can mostly get away with designing my own. I was a team of one.

I also put out my first proper shareware tool that provided a trickle of income for a while in the form of my Windows monitoring tool for "Remote Access BBS" called Monitor/RA. At one point, Diamond Multimedia purchased a commercial license to monitor their European support BBS.

Later on, I'd even make some controls that drew themselves looking like MacOS Platinum and NextStep.

My site back then had a heavy Delphi focus and is in love with small fonts, tiny icons, pastel shades, and almost-invisible watermark textures.

Alternatives

I never managed to get into Java desktop development despite many attempts. The screenshots looked good, but it never felt properly native-like Delphi did. Having to install a non-trivial runtime was also off-putting, especially for these small side-projects.

Borland C++ still felt like a large mountain to climb, and I had no reason, and so those books stayed on my shelf for many years in the "I should get to this." Of course, OWL died as Microsoft Visual C++ came up - a product that has never really delivered on the Visual unless you were prepared to commit to another mountain in the form of the MFC.. the mountain might have been climbable with a visual designer but, one never materialized.

Visual Basic was hard to go back to. It immediately felt limiting, and having to learn COM to develop controls was a show-stopper. After I had picked up Delphi 2, in fact, I never touched VB again as far as I can recall. I instead spent a lot of time in Microsoft Access connected to SQL Server, then onto ASP, and then finally ASP.NET.

I remember being excited when WinForms was announced. C# is a language I came to love more than Pascal, and WinForms was a definite improvement over VB - you could now easily write your own controls, but .NET Framework eschewed object-orientated design and patterns and instead just went with some inheritance and helper methods - ironically something Java did better. Comparing IO in .NET vs. IO in Java also highlights this - the latter more than happy to provide wrappers and layers for great composition vs. .NET lumping tons of stuff into static methods on File and providing a bunch of Readers you can never remember.

During my degree, we used Borland C++ builder, which took the VCL and provided you with the ability to use C++ instead of Pascal. I enjoyed using that, and just this year, I've found myself wanting to create a single-executable solution, but C++ was again something nobody in my circles was needing.

Legacy

I kept buying copies of Delphi (at every alternate release, it's my own money) and found myself creating a tool for working with disk images of 3" floppy disks (yes, not 3.5") used by Sinclair, Amstrad, and Tatung machines. It was aimed at the archiving community, and Delphi was the tool I fell back to - I had zero interest in telling people to install large runtime packages. It's still maintained today, although it was ported to the free-software Lazarus alternative.

At one point, I also worked with a team on a ZX Spectrum emulator developed in Delphi. Their diving into x86 assembler at the tip of a hat irked me. I wanted to work on something maintainable with cool features rather than cater to a small vocal crowd of people on lower-powered PCs. I didn't know x86, and I had no desire to learn it - my brief encounter with 8051 had been enough.

Pascal always felt a little awkward to me, but the VCL, IDE, and those self-contained small executables (200KB in initial versions, IIRC) were worth staying for. Living on a small island meant that programming jobs were scarce and almost exclusively limited to COBOL on AS/400's or Microsoft tools on Windows.

The draw for me was always this idea of small standalone exes, which is hard to achieve on Windows with a decent GUI builder. There are third-party options, but every one of them seems to fail to provide the seamlessly enjoyable experience Delphi and the VCL did, and many of them are either poorly maintained or can cost a small fortune.

Interestingly Delphi and C++ Builder and now available in free community editions but confusing bundled together as "RAD Studio." I've wanted to write a specific Windows native app of late that needs to call a bunch of Windows APIs, and so C++ Builder is back on my machine. (I wish the small binaries of old hadn't become megabyte monsters... I'm sure I don't need most of what's in there)

[)amien

I know some of this is a bit gushing; that's nostalgia for you, I guess. I can at least assure you that I receive no compensation/kickback/affiliate from Embarcadero, who currently own Delphi/C++ Builder and RAD Studio (I had to Google their name, seriously).