Posts tagged with api

Developing a great SDK: Guidelines & Principles

A good SDK builds on the fundamentals of good software engineering but SDKs have additional requirements to consider.

Why is an SDK different?

When developing software as a team a level of familiarity is reached between the team members. Concepts, approaches, technologies, and terminology are shaped by the company and the goals are typically aligned.

Even as new members join the team a number of 1-on-1 avenues exist to onboard such as pairing, mentoring, strategic choices of what to work on etc.

Software to be consumed by external developers is not only missing this shared context but each developer will have a unique context of their own. For example: What you think of as an authorization request as a domain expert is unlikely to match what a user thinks authorization means to their app.

The backgrounds of developers can be diverse with varying abilities and requirements each shaped by their experiences with other software and the industries they’ve worked in.

Onboarding potential customers, developers, or clients with 1-on-1 support simply doesn’t scale and the smallest bump in the road can lead them down a different path and away from your service.

Goals

A guiding principle for developing software is to be user-focused throughout.

This is especially important when developing an SDK and yet is more easily overlooked as it is created by a developer for a developer.

It is important to remember that you are not your audience.

You have in-depth knowledge of the how and why that the user is unlikely to have. Indeed they may not care or even want to learn - they have work of their own to be doing delivering the unique functionality of their solution. If they had the interest, time or experience to deal with the intracacies your library is supposed to take care of they wouldn’t need it.

Some more specific goals to follow are:

Reduce the steps

Every step is another opportunity for the developer to get confused, distracted or disenfranchised.

Success should involve as few steps as possible.

The primary technique for achieving this is to utilize defaults and convention liberally. Default to what works for the majority of people while still being secure and open to customization.

In the case where multiple steps are required consider combining those steps into a use-case specific flow.

Success can be delivered in parts.

If a user can try a default configuration and see that part working it provides encouragement and incentive to keep going on to the next requirement they have. Each success builds upon the previous to keep the user on-track and invested in this solution.

Simplify concepts and terminology

Terminology is essential to a deep understanding of any field however it can become a massive barrier to adoption for those less versed in the topic.

It is important to use phrases, concepts and terminology your audience will understand rather than specific abstract or generic terms defined in underlying RFCs or APIs. This should be reflected in the primary class names, functions, methods, and throughout the documentation.

When it is necessary to expose less-common functionality you should strive to progressively reveal the necessary detail. In cases where this exposure provides facilities close to the underlying implementation then it becomes advantageous to revert back to the terminology used there for advanced operations.

Guide API discovery

Many platforms and languages have facilities that can be utilized to help guide API discovery primarily through autocompletion mechanisms such as Visual Studio’s IntelliSense.

Common functionality should flow-out from the objects or functionality the developer has at that point. If your API has provided a connection to your service you should not then expect them to go and discover all new objects and namespaces.

Many popular pieces of software like to provide the developer with a “context” object. This is an object that exists only for the current request (in web server applications) or current developers instance of the app (in client applications) that could provide access to the current user and the various operations available - authorizing, configuring, performing api requests etc. The act of obtaining this context in an for example an identity SDK could be logging in.

Namespaces can be used to push more advanced functionality away from newer developers allowing them to concentrate on primary use-cases which can be combined into a single common root or default namespace.

The same principle applies to methods and properties especially in strongly-typed languages with rich IDE support. A fluent-style API for optional configuration can not only guide you through the available options but can also prevent you from making incompatible choices right at compile time where it is safe to provide detailed messages in context to the line of code.

Look native

Developers often specialize in just one or two platforms at any one time and become intimately familiar with the design and flavor of those platforms.

SDKs should strive to feel like native citizens of that ecosystem adopting the best practices, naming conventions, calling patterns and integrations that the user. Cross-platform solutions that look the same across platforms are only of interest to other cross-platform developers.

When using your API developers should be able to anticipate how it will function, how error handling, logging, and configuration will work based on their experience with the platform. Take time to understand both what is popular on that platform and which direction things are moving when making choices.

Feeling native further reduces the barrier to entry and can replace it with a feeling of “It just works!”.

Resist the temptation to make your SDKs work the same way across platforms when it goes against the grain of that platform. It might make things easier for you and your team but the pain will be pushed onto consumers of your SDK and waste their resources every time your API behaves in a way that is unintuitive to people familiar with the platform.

Inline documentation

Online documentation provides a great place for both advanced topics that require multiple interactions as well as letting new developers see what is involved before they switch into their IDE or download an SDK.

However, when using the code itself the SDK should put concise documentation about classes, functions and parameters at their fingertips where possible. Many IDEs provide the ability to display code annotations e.g. XML Documentation Comments in C# and JSDoc.

This should be leveraged to keep developers engaged once they start using the SDK. Switching to a browser to read documentation presents them with tabs of other things needing attention or other solutions that don’t involve using your SDK.

Strategies

Consume an API you wish you had

It can be incredibly advantageous to start by writing the code a user might expect to write to perform the operation in their application.

You want the code to be aligned with the goals:

  • Concise — the minimum number of steps
  • Understandable — only well-known terminology
  • Discoverable — if the editor allows it
  • Familiar — it feels like a native SDK

Start with a small fragment that exercises a specific scenario to both prove it and provide a real-world-like snippet for the documentation.

Better yet adopt or develop a small reference application and show the SDK working as a whole. Such an app can also be published itself as a great reference for programmers looking for concrete examples or best practices and form the basis of starters, examples and tutorials.

These applications also have further long-term value in that they can be used to:

  • See if and how the SDK breaks applications when changes are introduced
  • Form ideas about how deprecations are handled
  • Prove (or disprove) how a proposed SDK change improves the experience

Iterative approach

Traditional up-front design requires you trade off how much research you do before you start designing the system. Undoubtedly no matter how much research you do it will never be enough. Either there are use cases that were missed, small details that negatively impact the design, or implementation constraints that go against the design in awkward ways.

It is important to approach writing a library by implementing pieces one at a time continually refining and refactoring as you go along to ensure that you end up with a design that fits both the problem domain and the unseen constraints either the technology or the intricacies the domain imposes that would have no doubt been missed in an up-front design.

Sample applications and references can really help shape the good design as you go by reflecting how changes to the SDK affect your applications.

Breaking clients is to be avoided where possible so a design should be refined as much as possible before initial release given both the current constraints and future direction. If a piece of the SDK is not well baked strongly consider keeping back the unbaked portions so that the primary developers do not take a dependency on it yet.

Other avenues are available to help bake the design and functionality of new components such as forums, private groups, internal teams or other team members both on this SDK or on others. Design shortcomings are much easier to spot when you are distanced from its creation.

Local git branches are a vitally important safety net to aggressive refactoring - commit after each good step.

Note about unit tests

Unit tests are very important for production quality software, ongoing maintenance and even late stage refactoring however developing unit tests too early in the development cycle can work against aggressive refactoring and redesign.

It is difficult to move functionality, fields and methods around between classes and methods when there are a multitude of unit tests expecting them there especially given that the unit tests at these early phases tend to be nothing more than checking the most basic of functionality.

You should also pay attention to how much influence the unit tests themselves are exerting on the design of the SDK components themselves and whether this makes those components a simpler design for consumers or pushes more requirements to them on how to make many small testable pieces work together as a simple single component to solve traditional use-cases.

Layered design

Some of these goals can be difficult to implement in a single design. For example, how do you:

  • Ensure that SDKs that appear so different are approachable by engineers at Auth0?
  • Avoid a multitude of options when there are choices that need to be made?
  • Provide the ability for advanced consumers to go beyond the basics?
  • A dual-layer design can work very well for client SDKs and help solve these problems.

The lower levels of the design are unit-testable building blocks that very much mirror the underlying APIs, concepts, and RFCs. Classes are focused on a specific task or API but take a variety of inputs ensuring the right kind of request is sent with the right parameters and encodings. This gives a solid foundation to build on that is well understood by the team as well as across SDKs.

The high-level components are responsible for orchestrating the lower pieces into a coherent native-friendly easy-to-use experience for a majority of use cases. They form the basis of quick-starts, initial guidance, and tutorials. Other high-level components may in fact form plug-ins to existing extensions provided by the environment.

This layering also helps developers perhaps unfamiliar with the platform clearly see how the underlying familiarly-named elements are utilized in particular environments or flows.

When consumers need to go beyond the capabilities of high-level components the same underlying building blocks used by the high-level components are available to them to compose, reuse and remix as they need.

For example: A C# SDK might include a high-level component for desktop apps that automatically become part of application’s startup and shutdown as well as provides support for opening login windows. Another high-level components might be developed for server-to-server communication and know how to check common configuration patterns such as Azure application settings or web.config files and deal with secure secrets.

Each is tailored specifically to the use case but use the same underlying blocks to achieve the result.

It is also advantageous in environments that support package management to individually package environment-dependent parts with clear labels that describe their use in that environment. This aids in both discovery and ensures the SDK does not bring along additional unneeded sub-dependencies for that environment while bringing along the core shared lower level package.

Designing a great API

Several years ago I worked on a payroll package developing a core engine that required an API to let third parties write calculations, validations and security gates that would execute as part of it’s regular operation.

We were a small team and I had many conversations with another developer tasked with building a payroll using the API I would provide. Some methods here, classes there, the odd helper function and I had an API and then we had a mini payroll running.

Then he showed me the code he had written and that smug grin dropped off my face. It was awful.

Perhaps this other developer wasn’t as great as I’d thought? Looking at the code though made me realize he had done the best anyone could with a terrible API. I’d exposed parts of this core payroll engine with hooks when it needed a decision. Its job was to run the payroll – a very complex task that involved storage, translation, time periods, users and companies. That complexity and context had leaked out.

Unfortunately it’s not a unique story – many API’s are terrible to use. They’re concerned with their own terminology, limitations and quirks because they are exposed sections of an underlying system developed by those responsible for the underlying system.

If you want others to have a good experience with your product you have to put yourself in their shoes. Whether it’s a UI or an API makes no difference.

You are not the user

That’s the real difference between writing the classes that form your regular implementation and those that make up your public API.

We had time to fix our payroll API. Instead of refining and polishing here and there we took the 20 or so snippets developed for the mini payroll and pruned, cleaned and polished until they looked beautiful. They scanned well and made sense to payroll developers unfamiliar with our package. When a third developer familiar with payrolls but unfamiliar with out package developed the necessary code for a fully-functional jurisdiction in record time with minimal assistance we knew we had hit our goal.

Sure implementing that new API was hard work. Instead of simple methods sticking out of the engine we had a facade over our engine but it was justified. They were two different systems for two different types of user with distinct ideas about what the system was and how it was going to be used.

Code First

Many years later I found myself on a small team of 3 people tasked with putting a brand new API on top of Entity Framework for configuring models with code the .NET world would come to know as Code First. I was determined to use my experience and avoid another complex API surface littered with terminology and leaky abstractions. Parts of EF already suffered from that problem.

So for the first few weeks of that project we didn’t write any of the code that would in fact become Code First.

Instead we decided who our user was – in this case a C# developer who likes writing code, knows LINQ and some database concepts but doesn’t know Entity Framework as people who did were already using Model First or Database First.

Then we wrote tiny sample apps and tried to find simpler and simpler ways to describe them in code. We’d often start on a whiteboard with a scenario and write the complete mapping. We’d then try and find conventions that would remove the need for most of it and then try to write succinct code to configure the rest. As the newest guy to the team I’d fight to keep EF terms away from the main API surface in order to reduce that barrier to entry and help drive adoption.

Finally we’d hit the computer and develop stub classes and methods to make samples compile and let us try the IntelliSense. This isn’t always necessary but if you want to develop a fluent API or provide lots of type-safety such as Code First’s relationship mapping it’s highly recommended.

We’d then revisit the samples later and see if they could be read as easily as they were written and figure out what problems people were likely to run into and whether we could solve them without too much noise. Sometimes this meant having more than one way to do things such as chaining the fluent methods or allowing a bunch of properties to be set (solved with an extension method class providing the fluent API) and how users could manage larger models (solved by sub-classing EntityConfiguration – now EntityTypeConfiguration sigh – and allowing redundant specification for things like relationships that span more than one class).

We finally ended up with succinct code like this with IntelliSense guiding you along the way and preventing you from even being able to specify invalid combinations. The HasMany prompts the properties on Customer and it won’t show you WithRequired unless it is valid. In the case of Required to Required it will ensure that the WithRequired specified which end is principle and dependent. In short it guides you through the process and results in highly readable code.

Entity<Customer>().HasMany(c => c.Orders).WithRequired(o => o.Customer).WillCascadeOnDelete();

This process took a little longer but given the amount of use the API will get that time will be saved by users countless times over.

Code First went down incredibly well with both the target audience and existing EF users and inspired the simpler DbContext interface that became the recommended way of accessing EF.

I think it’s one of the nicer APIs to come out of Microsoft and .NET.

[)amien

PS. Martin Fowler has some great guidance in his book Domain Specific Languages.