Posts tagged with code-snippet

Download files with progress in Electron via window.fetch

Working on Atom lately I need to be able to download files to disk. We have a couple of ways to do this today but they do not show download progress which leads to confusion and sometimes frustration on larger downloads such as updates or big packages.

There are many npm libraries out there but they either don’t expose a progress indicator or they bypass Chrome (thus not using proxy settings, caching and network inspector) by using node directly.

I’m also not a fan of sprawling dependencies to achieve what can be done simply in a function or two.

Hello window.fetch

window.fetch is a replacement for XMLHttpRequest currently shipping in Chrome (and therefore Electron) as well as a whatWG living standard. While there is some documentation around most of it relies on grabbing the entire content as JSON, a blob or text. This is not advised for streaming where the files might be large and you want to not only minimize memory impact but also display a progress indicator to your users.

Thankfully window.fetch has a getReader() function that will give you a ReadableStreamReader although this reads in chunks (32KB on my machine) and isn’t compatible with Node’s streams, pipes and data events.

Download function

With a little work though we can wire these two things up to get us a file downloader that has no extra dependencies outside of Electron, honors the Chrome cache, proxy and network inspector and best of all is incredibly easy to use;

import fs from 'fs';

export default async function download(sourceUrl, targetFile, progressCallback, length) {
  const request = new Request(sourceUrl, {
    headers: new Headers({'Content-Type': 'application/octet-stream'})
  });

  const response = await fetch(request);
  if (!response.ok) {
    throw Error(`Unable to download, server returned ${response.status} ${response.statusText}`);
  }

  const body = response.body;
  if (body == null) {
    throw Error('No response body');
  }

  const finalLength = length || parseInt(response.headers.get('Content-Length' || '0'), 10);
  const reader = body.getReader();
  const writer = fs.createWriteStream(targetFile);

  await streamWithProgress(finalLength, reader, writer, progressCallback);
  writer.end();
}

async function streamWithProgress(length, reader, writer, progressCallback) {
  let bytesDone = 0;

  while (true) {
    const result = await reader.read();
    if (result.done) {
      if (progressCallback != null) {
        progressCallback(length, 100);
      }
      return;
    }
    
    const chunk = result.value;
    if (chunk == null) {
      throw Error('Empty chunk received during download');
    } else {
      writer.write(Buffer.from(chunk));
      if (progressCallback != null) {
        bytesDone += chunk.byteLength;
        const percent = length === 0 ? null : Math.floor(bytesDone / length * 100);
        progressCallback(bytesDone, percent);
      }
    }
  }
}

A FlowType annotated version is also available.

Using it

Using it is simplicity - call it with a URL to download and a local file name to save it as along with an optional callback that will receive download progress.

Downloader.download('https://download.damieng.com/fonts/original/EnvyCodeR-PR7.zip', 'envy-code-r.zip', (bytes, percent) => console.log(`Downloaded ${bytes} (${percent})`));

Caveats

Some servers do not send the Content-Length header. You have two options if this applies to you;

  1. Don’t display a percentage just the KB downloaded count (percentage will be null in the callback)
  2. Bake-in the file size if it’s a static URL - just pass in as final parameter to the download function

Enjoy!

[)amien

Rails-style controllers for ASP.NET

Rob Conery has been putting together some great screen casts on SubSonic and his latest on generating controllers pointed out that ASP.NET doesn’t support the Rails-style http://site//controller/method style of execution.

This got me quite excited and I’ve put together a proof-of-concept web project that demonstrates mapping the path to controller methods using an IHttpHandler and reflection.

How it works

It registers the ControllerHttpHandler via the web.config:

<httpHandlers>
  <add path="/ctl/\*/\*" verb="POST,GET,HEAD" type="ControllerHttpHandler" />
</httpHandlers>

There is a very basic Controller abstract base class that just provides a simple way of accessing the context for dealing with request/response for now.

public abstract class Controller
{
  protected System.Web.HttpContext context;

  internal Controller(System.Web.HttpContext context) {
    this.context = context;
  }
}

We then have a test controller or two that implement from this with a couple of methods and the required constructor:

public class TestController : Controller
{
  public TestController(System.Web.HttpContext context) : base(context) { }

  public void Index() {
    context.Response.Write("This is the index");
  }

  public void Welcome() {
    context.Response.Write("Welcome to the TestController");
  }
}

Finally the magic that joins them up is the ControllerHttpHandler:

using System;
using System.Web;
using System.Reflection;

public class ControllerHttpHandler : IHttpHandler
{
  public void ProcessRequest(HttpContext context) {
    string[] parts = context.Request.Path.Split('/');
    if (parts.Length < 4) {
      context.Response.Write("No controller & member specified");
      return;
    }

    string controllerName = parts[2];
    string methodName = parts[3];
    Type potentialController = Type.GetType(controllerName);
    if (potentialController != null && potentialController.IsClass && potentialController.IsSubclassOf(typeof(Controller))) {
      MethodInfo potentialMethod = potentialController.GetMethod(methodName);
      if (potentialMethod != null) {
        Controller controller = (Controller) Activator.CreateInstance(potentialController, context);
        potentialMethod.Invoke(controller, null);
      }
      else
        context.Response.Write(String.Format("Method '{0}' not found on controller '{1}'", methodName, controllerName));
    }
    else
      context.Response.Write(String.Format("Controller '{0}' not found", controllerName));
  }

  public bool IsReusable {
    get { return false; }
  }
}

That’s it!

Limitations

The controllers and methods are mapped at run-time using reflection. This would probably be too slow for production. Also it currently has to be in a top-level folder because I can’t figure out how to pass the HTTP request back to ASP.Net to try with the rest of the stack if we don’t have a matching controller/method.

One option might be to have no httpHandlers in the web.config and add the exact controller/method maps at build or run-time. This solves both the top-level problem and potentially the speed.

Another option to address just the speed of reflection would be to cache the path/method strings to the actual method and type so the only reflection would be the Activator.CreateInstance. If that is slow then we could look at pooling the controller instances themselves.

Going forward

Parameters for a method could be extracted and parsed from the query-string – they are currently ignored.

Response is raw output – we could do something very similar to rhtml.

I’m going to chat things over with the Subsonic team and see if we can come up with anything from here.

[)amien

Parameterising the IN clause of an SQL SELECT in .NET

I’m a fan of parameterized queries with a strong dislike for building SQL (or other magic strings). Encoding, escaping errors, localization formatting problems and injection can run rampant when you think everything is a string.

Problem

Today I found myself a quandary as I needed to SELECT records based on a list of values I had. e.g.

SELECT * FROM Products WHERE ProductCode IN ('ABC123', 'DEF456', 'GHI789')

At first glance the relevant parameterized version might look like:

SELECT * FROM Products WHERE ProductCode IN (@productlist)

The problem here however is that if you put a comma-separate list of items into a string parameter named @productlist then it sends this to the database server:

SELECT * FROM Products WHERE ProductCode IN ('ABC123, DEF456, GHI789')

That’s not what we want at all.

This hack isn’t pretty and it has some limitations:
  • Only works with named parameters
  • Could upset some DB providers that don’t like having command parameters removed or command text modified
  • Parameter name to replace must be totally unique – i.e. not exist as a subset of another parameter name
  • Only preserves the basic IDbCommand declared properties

It should however work across DB providers and types.

Usage

The previous example would mean we use exactly the expected parameterized version with @productlist in place.

Add the parameter as you’d expect but instead of assigning a string/numeric to it assign something IEnumerable.

Finally call this method against the command and parameter before you execute it for the ‘magic’ to happen:

The ‘magic’

public void ExpandDbArrayParameter(IDbCommand cmd, IDbDataParameter parameter) {
  if (parameter.Value is IEnumerable) {
    int index = 0;
    StringBuilder newParameterSQL = new StringBuilder();
    foreach(Object value in (IEnumerable) parameter.Value) {
      String valueParameterName = String.Format("{0}{1}", parameter.ParameterName, ++index);
      IDataParameter valueParameter = cmd.CreateParameter();
      valueParameter.DbType = parameter.DbType;
      valueParameter.Direction = parameter.Direction;
      valueParameter.ParameterName = valueParameterName;
      valueParameter.SourceColumn = parameter.SourceColumn;
      valueParameter.SourceVersion = parameter.SourceVersion;
      valueParameter.Value = value;
      cmd.Parameters.Add(valueParameter);

      if (index == 1)
        newParameterSQL.Append(valueParameterName);
      else
        newParameterSQL.Append("," + valueParameterName);
      }
      cmd.Parameters.Remove(parameter);
      cmd.CommandText = cmd.CommandText.Replace(parameter.ParameterName, newParameterSQL.ToString());
   }
}

[)amien

.NET quick samples: Up-times, ages, rounding to n places

Just a few quick .NET samples for performing some common tasks that the .NET Framework doesn’t do for you:

System uptime

using System.Diagnostics;

public TimeSpan GetUptime() {
    var systemUpTime = new PerformanceCounter("System", "System Up Time");
    systemUpTime.NextValue(); // Required to work!
    return TimeSpan.FromSeconds(systemUpTime.NextValue())));
}

Calculating age

public int GetAge(DateTime birthday) { 
    int years = DateTime.Now.Year - birthday.Year;
    return (birthday.DayOfYear >= DateTime.Now.DayOfYear) ? years : years - 1;
}

Rounding to n decimal places

public decimal ArithmeticRound(decimal d, int decimals) {
    var power = (decimal)Math.Pow(10, decimals);
    return (decimal.Floor((Math.Abs(d) * power) + 0.4m) / power) * Math.Sign(d);
}

[)amien