Posts in category development

Developing a great SDK: Guidelines & Principles

A good SDK builds on the fundamentals of good software engineering but SDKs have additional requirements to consider.

Why is an SDK different?

When developing software as a team a level of familiarity is reached between the team members. Concepts, approaches, technologies, and terminology are shaped by the company and the goals are typically aligned.

Even as new members join the team a number of 1-on-1 avenues exist to onboard such as pairing, mentoring, strategic choices of what to work on etc.

Software to be consumed by external developers is not only missing this shared context but each developer will have a unique context of their own. For example: What you think of as an authorization request as a domain expert is unlikely to match what a user thinks authorization means to their app.

The backgrounds of developers can be diverse with varying abilities and requirements each shaped by their experiences with other software and the industries they’ve worked in.

Onboarding potential customers, developers, or clients with 1-on-1 support simply doesn’t scale and the smallest bump in the road can lead them down a different path and away from your service.

Goals

A guiding principle for developing software is to be user-focused throughout.

This is especially important when developing an SDK and yet is more easily overlooked as it is created by a developer for a developer.

It is important to remember that you are not your audience.

You have in-depth knowledge of the how and why that the user is unlikely to have. Indeed they may not care or even want to learn - they have work of their own to be doing delivering the unique functionality of their solution. If they had the interest, time or experience to deal with the intracacies your library is supposed to take care of they wouldn’t need it.

Some more specific goals to follow are:

Reduce the steps

Every step is another opportunity for the developer to get confused, distracted or disenfranchised.

Success should involve as few steps as possible.

The primary technique for achieving this is to utilize defaults and convention liberally. Default to what works for the majority of people while still being secure and open to customization.

In the case where multiple steps are required consider combining those steps into a use-case specific flow.

Success can be delivered in parts.

If a user can try a default configuration and see that part working it provides encouragement and incentive to keep going on to the next requirement they have. Each success builds upon the previous to keep the user on-track and invested in this solution.

Simplify concepts and terminology

Terminology is essential to a deep understanding of any field however it can become a massive barrier to adoption for those less versed in the topic.

It is important to use phrases, concepts and terminology your audience will understand rather than specific abstract or generic terms defined in underlying RFCs or APIs. This should be reflected in the primary class names, functions, methods, and throughout the documentation.

When it is necessary to expose less-common functionality you should strive to progressively reveal the necessary detail. In cases where this exposure provides facilities close to the underlying implementation then it becomes advantageous to revert back to the terminology used there for advanced operations.

Guide API discovery

Many platforms and languages have facilities that can be utilized to help guide API discovery primarily through autocompletion mechanisms such as Visual Studio’s IntelliSense.

Common functionality should flow-out from the objects or functionality the developer has at that point. If your API has provided a connection to your service you should not then expect them to go and discover all new objects and namespaces.

Many popular pieces of software like to provide the developer with a “context” object. This is an object that exists only for the current request (in web server applications) or current developers instance of the app (in client applications) that could provide access to the current user and the various operations available - authorizing, configuring, performing api requests etc. The act of obtaining this context in an for example an identity SDK could be logging in.

Namespaces can be used to push more advanced functionality away from newer developers allowing them to concentrate on primary use-cases which can be combined into a single common root or default namespace.

The same principle applies to methods and properties especially in strongly-typed languages with rich IDE support. A fluent-style API for optional configuration can not only guide you through the available options but can also prevent you from making incompatible choices right at compile time where it is safe to provide detailed messages in context to the line of code.

Look native

Developers often specialize in just one or two platforms at any one time and become intimately familiar with the design and flavor of those platforms.

SDKs should strive to feel like native citizens of that ecosystem adopting the best practices, naming conventions, calling patterns and integrations that the user. Cross-platform solutions that look the same across platforms are only of interest to other cross-platform developers.

When using your API developers should be able to anticipate how it will function, how error handling, logging, and configuration will work based on their experience with the platform. Take time to understand both what is popular on that platform and which direction things are moving when making choices.

Feeling native further reduces the barrier to entry and can replace it with a feeling of “It just works!”.

Resist the temptation to make your SDKs work the same way across platforms when it goes against the grain of that platform. It might make things easier for you and your team but the pain will be pushed onto consumers of your SDK and waste their resources every time your API behaves in a way that is unintuitive to people familiar with the platform.

Inline documentation

Online documentation provides a great place for both advanced topics that require multiple interactions as well as letting new developers see what is involved before they switch into their IDE or download an SDK.

However, when using the code itself the SDK should put concise documentation about classes, functions and parameters at their fingertips where possible. Many IDEs provide the ability to display code annotations e.g. XML Documentation Comments in C# and JSDoc.

This should be leveraged to keep developers engaged once they start using the SDK. Switching to a browser to read documentation presents them with tabs of other things needing attention or other solutions that don’t involve using your SDK.

Strategies

Consume an API you wish you had

It can be incredibly advantageous to start by writing the code a user might expect to write to perform the operation in their application.

You want the code to be aligned with the goals:

  • Concise — the minimum number of steps
  • Understandable — only well-known terminology
  • Discoverable — if the editor allows it
  • Familiar — it feels like a native SDK

Start with a small fragment that exercises a specific scenario to both prove it and provide a real-world-like snippet for the documentation.

Better yet adopt or develop a small reference application and show the SDK working as a whole. Such an app can also be published itself as a great reference for programmers looking for concrete examples or best practices and form the basis of starters, examples and tutorials.

These applications also have further long-term value in that they can be used to:

  • See if and how the SDK breaks applications when changes are introduced
  • Form ideas about how deprecations are handled
  • Prove (or disprove) how a proposed SDK change improves the experience

Iterative approach

Traditional up-front design requires you trade off how much research you do before you start designing the system. Undoubtedly no matter how much research you do it will never be enough. Either there are use cases that were missed, small details that negatively impact the design, or implementation constraints that go against the design in awkward ways.

It is important to approach writing a library by implementing pieces one at a time continually refining and refactoring as you go along to ensure that you end up with a design that fits both the problem domain and the unseen constraints either the technology or the intricacies the domain imposes that would have no doubt been missed in an up-front design.

Sample applications and references can really help shape the good design as you go by reflecting how changes to the SDK affect your applications.

Breaking clients is to be avoided where possible so a design should be refined as much as possible before initial release given both the current constraints and future direction. If a piece of the SDK is not well baked strongly consider keeping back the unbaked portions so that the primary developers do not take a dependency on it yet.

Other avenues are available to help bake the design and functionality of new components such as forums, private groups, internal teams or other team members both on this SDK or on others. Design shortcomings are much easier to spot when you are distanced from its creation.

Local git branches are a vitally important safety net to aggressive refactoring - commit after each good step.

Note about unit tests

Unit tests are very important for production quality software, ongoing maintenance and even late stage refactoring however developing unit tests too early in the development cycle can work against aggressive refactoring and redesign.

It is difficult to move functionality, fields and methods around between classes and methods when there are a multitude of unit tests expecting them there especially given that the unit tests at these early phases tend to be nothing more than checking the most basic of functionality.

You should also pay attention to how much influence the unit tests themselves are exerting on the design of the SDK components themselves and whether this makes those components a simpler design for consumers or pushes more requirements to them on how to make many small testable pieces work together as a simple single component to solve traditional use-cases.

Layered design

Some of these goals can be difficult to implement in a single design. For example, how do you:

  • Ensure that SDKs that appear so different are approachable by engineers at Auth0?
  • Avoid a multitude of options when there are choices that need to be made?
  • Provide the ability for advanced consumers to go beyond the basics?
  • A dual-layer design can work very well for client SDKs and help solve these problems.

The lower levels of the design are unit-testable building blocks that very much mirror the underlying APIs, concepts, and RFCs. Classes are focused on a specific task or API but take a variety of inputs ensuring the right kind of request is sent with the right parameters and encodings. This gives a solid foundation to build on that is well understood by the team as well as across SDKs.

The high-level components are responsible for orchestrating the lower pieces into a coherent native-friendly easy-to-use experience for a majority of use cases. They form the basis of quick-starts, initial guidance, and tutorials. Other high-level components may in fact form plug-ins to existing extensions provided by the environment.

This layering also helps developers perhaps unfamiliar with the platform clearly see how the underlying familiarly-named elements are utilized in particular environments or flows.

When consumers need to go beyond the capabilities of high-level components the same underlying building blocks used by the high-level components are available to them to compose, reuse and remix as they need.

For example: A C# SDK might include a high-level component for desktop apps that automatically become part of application’s startup and shutdown as well as provides support for opening login windows. Another high-level components might be developed for server-to-server communication and know how to check common configuration patterns such as Azure application settings or web.config files and deal with secure secrets.

Each is tailored specifically to the use case but use the same underlying blocks to achieve the result.

It is also advantageous in environments that support package management to individually package environment-dependent parts with clear labels that describe their use in that environment. This aids in both discovery and ensures the SDK does not bring along additional unneeded sub-dependencies for that environment while bringing along the core shared lower level package.

Google Trends in Vue & Nuxt.JS

I’ve been doing a fair amount of work in Nuxt.JS of late. I’d previously used Next.JS but found React not to my liking and that Vue just fit better with my mental model/workflow.

While they are great for accelerating development there are definitely some areas where it’s made things a little trickier. One such area is when you wish to drop some “simple” client-side JavaScript in - here’s the process I went through to get Google Trends to work.

It looks like this:

The errors

Google Trends is a great way of visualizing what trends to searches are over time. The default embed script tho will not work in your Nuxt.JS app.

If you paste it as-is you will see the error:

Uncaught SyntaxError: expected expression, got '&'

So we should tell Nuxt.JS to run the script only on the client using the <client-only> tag. Now you will see:

Uncaught ReferenceError: trends is not defined

It appears that the second part of the embed code Google provides is trying to run before the first is ready. I’m not sure why this happens inside Nuxt.JS/Vue and not in a normal browser session but we can fix that by moving the initialization code into a function and then calling that function using the onload event on the primary script.

Now you will experience one of two things - either the page will reload with only Google Trends and not your content or you’ll get an error about the DOM being changed. This is due to the use of document.write in the default embed code.

Thankfully Google includes a renderExploreWidgetTo function which leads us to…

The solution

Instead of using the default embed code, adapt this version to your needs replacing just the contents of the initTrends function with the part from your Google Trends embed code and viola!

<template>
  <main>
    <div id="trendChart"></div>
    <client-only>
      <script>
        function initTrendChart() {
          trends.embed.renderExploreWidgetTo(
            document.getElementById("trendChart"),
            // Replace this block with yours
            "TIMESERIES",
            {
              comparisonItem: [
                {
                  keyword: "ZX Spectrum",
                  geo: "",
                  time: "2004-01-01 2021-07-01",
                },
              ],
              category: 0,
              property: "",
            },
            {
              exploreQuery: "date=all&amp;q=ZX+Spectrum",
              guestPath: "https://trends.google.com:443/trends/embed/",
            }
          );
        }
      </script>
      <script
        type="text/javascript"
        src="https://ssl.gstatic.com/trends_nrtr/2578_RC01/embed_loader.js"
        onload="initTrendChart()"
      ></script>
    </client-only>
  </main>
</template>

If you need multiple charts you’ll need to create multiple divs and paste multiple blocks into initTrendChart to ensure they are all initialized. You do not need multiple copies of the embed_loader script.

This also works just fine with markdown files used to render content via Nuxt Content.

Enjoy!

[)amien

Shipping breaking changes

Breaking changes are always work for your users. Work you are forcing them to do when they upgrade to your new version. They took a dependency on your library or software because it saved them time but now it’s costing them time.

Every breaking change is a reason for them to stop and reconsider their options. If your library is paid-for or the preferred way for paying users to access your services then lost users can come with a real financial cost.

Use breaking changes sparingly.

Good reasons for breaking changes

  • A feature is being removed for business reasons
  • Standards are changing how something works
  • The feature did not work as intended and it’s impossible to fix without a break
  • Service or library dependencies you rely upon are forcing a change
  • A small breaking change now prevents actual significant breaking changes later

Even when presented with these you should think not only about whether you can avoid a break but also take the opportunity to think about what you can do now to avoid similar breaks in the future.

Poor reasons for breaking changes

  • It makes the internals of the library tidier
  • Parameter order, method naming or property naming would be ”clearer”
  • Consistency with other platforms or products
  • Personal subjective interpretations of “better”
  • Compatibility with a different library to attract new users

While many of these are admirable goals in of themselves they are not a reason to break your existing customers.

Managing breaking changes

It goes without saying that intentional breaking changes should only occur in major versions with the exception of security fixes that require users of your library take some action.

Here are some thoughts to ease the pain of breaking changes:

  • List each breaking change in a migration guide with a before and after code fragment
  • Summarize breaking changes in the README with a link to the migration guide for more information
  • Keep breaking change count low even on major releases

Users should ideally also be able to find/replace or follow compiler errors. Consider:

  • Platform-specific mechanisms for dealing with breaking changes. e.g. in C# you can use the [Obsolete] attribute to help guide to the replacement API, Java has the @deprecated annotation.
  • Leaving a stub for the old method in place for one whole major release that calls the new method with the right arguments and produces a log warning pointing to the migration guide.

Rewrites

If a package is drastically different users will need to rewrite code. This is always a bigger deal for them than you expect, because:

  1. It is unscheduled and was not on their radar (no they are not monitoring your GitHub issue discussions)
  2. They use your library in ways you likely don’t expect or anticipate
  3. The more they depend on your product then the more work your rewrite involves

Really consider whether your new API is actually better (ideally before you ship it). One way to do this is to produce a set of example usage code for using the old library vs the new library. Put them side-by-side and open them up to feedback. Is the new API genuinely better? Some indicators it might be are:

  • Less code for users to write
  • Plenty of sensible and safe defaults
  • Less specialized terminology and concepts
  • Easier to test and abstract
  • Existing customers prefer the new syntax and think it’s worth changing

Some indicators that it isn’t really any better is: internal staff prefer it, it aligns better with some other platform or just being different.

Sometimes it’s worth doing because it targets a different crowd or comes at the problem from a simpler direction or abstraction. If so, then seriously consider giving it a new package name and put it out for early access.

Make sure users are led to the right library of the two and if there is a lot of code duplication and users on the “old” library then consider making the next version of the old library a compatibility wrapper around the new library.

[)amien

From CircleCI to GitHub Actions for Jekyll publishing

I’ve been a big fan of static site generation since I switched from WordPress to Jekyll back in 2018. I’m also a big fan of learning new technologies as they come along, and now GitHub Actions are out in the wild; I thought this would be an opportunity to see how I can port my existing custom CircleCI build to Jekyll.

The CircleCI job

A quick recap from part 5 - Hosting & Building, my CircleCI configuration was basically two jobs that have subsequently been tweaked since then. They are:

Build

The build job’s responsibility was to configure a Ruby environment capable of executing Jekyll to build the site, removing the .html extension from the output filenames, and then indexing the content using Algolia. Here is the start of the .circleci/config.yml I was using for that:

version: 2
jobs:
  build:
    docker:
      - image: circleci/ruby:2.6.1
    working_directory: ~/jekyll
    environment:
      - JEKYLL_ENV=production
      - NOKOGIRI_USE_SYSTEM_LIBRARIES=true
      - JOB_RESULTS_PATH=run-results
    steps:
      - checkout
      - restore_cache:
          key: jekyll-{{ .Branch }}-{{ checksum "Gemfile" }}
      - run:
          name: Update gems
          command: gem update --system
      - run:
          name: Install dependencies
          command: bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
      - save_cache:
          key: jekyll-{{ .Branch }}-{{ checksum "Gemfile" }}
          paths:
            - "vendor/bundle"
      - run:
          name: Create results directory
          command: mkdir -p $JOB_RESULTS_PATH
      - run:
          name: Build site
          command: bundle exec jekyll build --config _config.yml,_config-publish.yml 2>&1 | tee $JOB_RESULTS_PATH/build-results.txt
      - run:
          name: Remove .html suffixes
          command: find _site -name "*.html" -not -name "index.html" -exec rename -v 's/\.html$//' {} \;
      - run:
          name: Index with Algolia
          command: bundle exec jekyll algolia --config _config.yml,_config-publish.yml
      - store_artifacts:
          path: run-results/
          destination: run-results
      - persist_to_workspace:
          root: ~/jekyll
          paths:
            - _site

Deploy

Deploy takes the output from the build job and syncs it with the S3 bucket I use to publish the site. It then applies AWS S3-specific commands using the AWS CLI tool to ensure metadata, redirects, and caching are correctly set using a Python environment.

  deploy:
    docker:
      - image: circleci/python:2.7
    working_directory: ~/jekyll
    steps:
      - attach_workspace:
          at: ~/jekyll
      - run:
          name: Install AWS CLI
          command: sudo pip install awscli
      - run:
          name: Deploy to S3
          command: aws s3 sync _site s3://damieng-static/ --delete --content-type=text/html
      - run:
          name: Correct MIME for robots.txt automatically
          command: aws s3 cp s3://damieng-static/robots.txt s3://damieng-static/robots.txt --metadata-directive="REPLACE"
      - run:
          name: Correct MIME for sitemap.xml automatically
          command: aws s3 cp s3://damieng-static/sitemap.xml s3://damieng-static/sitemap.xml --metadata-directive="REPLACE"
      - run:
          name: Correct MIME for Atom feed manually
          command: aws s3 cp s3://damieng-static/feed.xml s3://damieng-static/feed.xml --no-guess-mime-type --content-type="application/atom+xml" --metadata-directive="REPLACE"
      - run:
          name: Redirect /damieng for existing RSS subscribers
          command: aws s3api put-object --bucket damieng-static --key "damieng" --website-redirect-location "https://damieng.com/feed.xml"
      - run:
          name: Latest Envy Code R redirect
          command: aws s3api put-object --bucket damieng-static --key "envy-code-r" --website-redirect-location "https://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released"
      - run:
          name: Latest Envy Code R redirect #2
          command: aws s3api put-object --bucket damieng-static --key "fonts/envy-code-r" --website-redirect-location "https://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released"
      - run:
          name: Latest Envy Code R download
          command: aws s3api put-object --bucket damieng-static --key "downloads/latest/EnvyCodeR" --website-redirect-location "https://download.damieng.com/fonts/original/EnvyCodeR-PR7.zip"
      - run:
          name: Correct MIME for CSS files
          command: aws s3 cp s3://damieng-static/css/ s3://damieng-static/css/ --metadata-directive="REPLACE" --recursive

GitHub Actions

So how could I go about this in GitHub Actions? I have to admit I spent far too long poking around and examining existing Jekyll actions. I have a bunch of steps here I need fine control of, especially around Algolia and S3. I finally ended up on what was quite a simple port.

Unlike the CircleCI configuration, I did not split these into two separate jobs because:

  1. There is no exact equivalent to persist_to_workspace and attach_workspace
  2. The alternative of storing and restoring the artifacts leaves large useless artifacts around
  3. I never ended up running the jobs separately
  4. GitHub Actions provides an environment with both Ruby and AWS CLI installed

So, on to the configuration which lives in .github/workflows/jekyll.yml in my case:

name: Build site and deploy

on:
  push:
    branches: [ master ]

jobs:
  build:
    name: Build + Deploy
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: 2.6.1

      - name: Ruby gem cache
        uses: actions/cache@v1
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-
            
      - name: Install dependencies
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

      - name: Build Jekyll site
        run:  bundle exec jekyll build --config _config.yml,_config-publish.yml

      - name: Remove .html suffixes except for index.html
        run: find _site -name "*.html" -not -name "index.html" | while read f; do mv "$f" "${f%.html}"; done                      

      - name: Index with Algolia
        env:
          ALGOLIA_API_KEY: ${{ secrets.ALGOLIA_API_KEY }}
        run:  bundle exec jekyll algolia --config _config.yml,_config-publish.yml
    
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
    
      - name: Sync site with S3
        run:  aws s3 sync _site s3://damieng-static/ --delete --content-type=text/html

      - name: Correct MIME types
        run: |
          aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.txt" --metadata-directive="REPLACE"
          aws s3 cp s3://damieng-static/sitemap.xml s3://damieng-static/sitemap.xml --metadata-directive="REPLACE"
          aws s3 cp s3://damieng-static/feed.xml s3://damieng-static/feed.xml --no-guess-mime-type --content-type "application/atom+xml" --metadata-directive "REPLACE"
          aws s3 cp s3://damieng-static/css/ s3://damieng-static/css/ --metadata-directive "REPLACE" --recursive
          aws s3 cp s3://damieng-static/js/ s3://damieng-static/js/ --metadata-directive "REPLACE" --recursive
      
      - name: Redirect /damieng for existing RSS subscribers
        run:  aws s3api put-object --bucket damieng-static --key "damieng" --website-redirect-location "https://damieng.com/feed.xml"

      - name: Latest Envy Code R redirects
        run:  |
          aws s3api put-object --bucket damieng-static --key "fonts/envy-code-r" --website-redirect-location "https://damieng.com/blog/2008/05/26/envy-code-r-preview-7-coding-font-released"
          aws s3api put-object --bucket damieng-static --key "downloads/latest/EnvyCodeR" --website-redirect-location "https://download.damieng.com/fonts/original/EnvyCodeR-PR7.zip"
        
      - name: Set caching for images at 30 days
        run:  aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.svg" --include "*.ico" --include "*.jpg" --include "*.png" --include "*.webp" --include "*.gif" --recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public

      - name: Set caching for CSS and JS at 1 hour
        run:  aws s3 cp s3://damieng-static/ s3://damieng-static/ --exclude "*" --include "*.css" --include "*.js" --recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=3600,public

I also took the opportunity to fix a long-running issue in that my S3 objects would lose my manually-applied cache settings (and new posts and files would not have any).

Gotchas

There were only a couple of bumps in the road once I decided on a straight-port rather than trying to leverage higher-level existing actions:

  1. The secrets configured in the repo settings were not automatically exposed to the commands running in the action. Instead, you have to expose them using the ${{ secrets.KEY_NAME }} syntax.
  2. I was using rename instead of mv. I don’t recall why. Perhaps it was my Windows-ness creeping in. Rename has been dropped in newer distros.

Summary

The syntax is surprisingly similar with build-times about the same as CircleCI. It’s just nice to have it in one place. Time to port some other repos over!

[)amien