Look before you file

Head to Bing, Google, official support sites and bug database if you have access to it (Microsoft Connect, Apple Radar, Bugzilla for Firefox etc.) to see if others have run into this issue. Searching for the error message can yield good results but remove any elements of the message specific to your project (e.g. class names, property names etc.)

The reasons for this are:

• you may have made a mistake (likely if the software has seen considerable use)
• behave differently to your expectations (the dreaded ‘by design’)
• already be fixed (in the next release and possibly as a patch)
• have an acceptable workaround (why accept a workaround?)

If you are running into problems with developer tools also check out:

• Official forums (MSDN Forums & ASP.NET Forums for Microsoft tools)
• Unofficial respected sites (StackOverflow)

Consider how likely it is you’ve discovered a bug given the complexity of what you are doing, how unusual it is and how mature the software is. Attempting something simple on an established piece of software likely means you’ve made a mistake or misunderstood the documentation.

A good bug report

So you decided to go ahead and report the bug (we all want better software after all).

The essence of a good bug report is having just the right amount of information to identify the bug without any unnecessary detail.

Let’s break it down.

What happened

The most important element is describing what happened and there are four major possibilities.

Error message

You told the software to do something and it displayed an error message instead.

On a good day the error message lets you know why it can’t do what you asked and lets you know what to do to make it work. Given you’re filing a bug report it isn’t one of those days.

The error message is likely cryptic, doesn’t tell you how to get what you want or is just plain wrong. We’ll need that message in its entirety so:

• Copy Windows message box contents as text by pressing control-c when it’s in focus
• When there is a lot of text take a screen print (printscreen on Windows, command-shift-3 on Mac OS X)
• Localized error messages may slow down support responses – switch the app back to their language first

Exceptions

When a piece of a program (called a method or function) can’t do what it claims it throws an ‘exception’ back up to the piece that asked (called) it. This exception is like an error message but with enough lot of technical detail that travels back up the program until something deals with it (known as a catch).

When you see an exception then nothing wanted to deal with it.

You will see an error message that contains the exception and possibly a list of program pieces that couldn’t deal with it. This is called a stack trace and is invaluable to the person investigating your report so include it.

Developers seeing exceptions in their own program need to determine if they should be catching that exception or whether it shouldn’t be occurring. Feel frim the stack trace so your own methods aren’t included but too much is better than too little.

Unexpected behaviour

The software should have done what you wanted but did something you didn’t expect instead.

Report what you thought should happen, what the software did instead, how this is different and the reason why you think it should be that way.

Other people may not agree with your change and in the case of shared programming libraries or frameworks a fix for you can become a break for others that rely on existing behaviour.

Terminated

The software just vanished from the screen without a trace. Crashed, terminated or unexpectedly quit and perhaps took some of your work with it :(

If you’re really unlucky the software that crashed is your operating system. The blue screen of death (BSOD) on Windows, the grey screen on Mac OS X.

Often there are logs left behind you can examine to identify what went wrong.

On Mac OS X fire up Console from Application > Utilities and see what you can find. iPhones, iPods and iPads do this quite often and iTunes will likely offer to send detailed information to Apple who will hopefully share it with the application developer if it’s not their own.

Windows users will want to head to the Event Viewer and find the error and any additional messages that appear to relate to it and included these too.

Steps to reproduce

Ideally the minimal number of steps to reproduce the error every time. You want this to be reliable as possible as companies have limited resources too and won’t spend days trying to reproduce a low-impact bug.

This can be a very important step in understanding the scope of the problem and it’s impact. If you can’t reproduce it in a minimal number of steps there’s always the possibility you’re overlooking some other aspect that could be causing the problem – bad data or rogue code elsewhere!

For an application this may be a data file or a number of manual steps a user must manually perform via the user interface.

For a framework a small self-contained project file with a minimal amount of code to reproduce the failing scenario.

Again here, if you can make the steps clear and ideally in the language of the company producing the software your bug report is not going to hit extra delays.

Environment

Bugs are often sensitive to their environment and you should always include the version number of the software you are using as well as pertinent platform details,  including:

• What operating system, version & service pack
• What processor architecture (x86, x64, IA-64)
• What language & locale your machine is running in (e.g. US English (en-US), Brazil Portuguese (pr-BR))

In the specific case of Visual Studio and .NET bugs:

• What version of Visual Studio you are using (including any service packs)
• What processor architecture you are compiling for
• What language and compiler version you are using (e.g. C# compiling for 4.0)

You may need to include details for your desktop or developer machine and details of the server it is connecting to if any are involved.

What you’ve tried

Chances are you tried several things before filing a bug report. Let us know if you tried:

• Alternate routes through the user interface
• Entering data in a different format or order
• A different computer or environment

Letting them know this means they can avoid suggesting things you’ve already tried or waste time trying them too.

[)amien

In LINQ this can be a big issue as associations are properties and it’s easy to end up issuing a query every time you hit one. This is referred to as the SELECT N+1 problem and tools like LINQ to SQL Profiler can help you find them.

An example

Consider the following section of C# code that displays a list of blog posts and also wants the author name.

foreach(Post post in db.Posts)
  Console.WriteLine("{0} {1}", post.Title, post.Author.Name);

This code looks innocent enough and will issue a query like “SELECT * FROM [Posts]” but iterating over the posts causes the lazy-loading of the Author property to trigger and each one may well issue a query similar to “SELECT * FROM [Authors] WHERE [AuthorID] = 1″.

In the case of LINQ to SQL it’s not always an extra load as it will check the posts AuthorID foreign key in its internal identity map (cache) to see if it’s already in-memory before issuing a query to the database.

LINQ to SQL’s LoadWith

Most object-relational mappers have a solution for this – Entity Framework’s ObjectQuery has an Include operator (that alas takes a string), and NHibernate has a fetch mechanism. LINQ to SQL has LoadWith which is used like this:

var db = new MyDataContext();
var dlo = new DataLoadOptions();
dlo.LoadWith<Posts>(p => p.Blog);
db.LoadOptions = dlo;

This is a one-time operation for the lifetime of this instance of the data context which can be inflexible and LoadWith has at least one big bug with inheritance issuing multiple joins.

A flexible alternative

This got me thinking and I came up with a useful extension method to provide Include-like facilities on-demand in LINQ to SQL (and potentially other LINQ providers depending on what they support) in .NET 4.0.

public static IEnumerable<T> Include<T, TInclude>(this IQueryable<T> query, Expression<Func<T, TInclude>> sidecar) {
   var elementParameter = sidecar.Parameters.Single();
   var tupleType = typeof(Tuple<T, TInclude>);
   var sidecarSelector =  Expression.Lambda<Func<T, Tuple<T, TInclude>>>(
      Expression.New(tupleType.GetConstructor(new[] { typeof(T), typeof(TInclude) }),
         new Expression[] { elementParameter, sidecar.Body  },
         tupleType.GetProperty("Item1"), tupleType.GetProperty("Item2")), elementParameter);
   return query.Select(sidecarSelector).AsEnumerable().Select(t => t.Item1);
}

To use simply place at the end of your query and specify the property you wish to eager-load, e.g.

var oneInclude = db.Posts.Where(p => p.Published).Include(p => p.Blog));
var multipleIncludes = db.Posts.Where(p => p.Published).Include(p => new { p.Blog, p.Template, p.Blog.Author }));

How it works

How it works is actually very simple – it projects into a Tuple that contains the original item and all additional loaded elements and then just returns the query back the original item. It is a dynamic version of:

var query = db.Posts.Where(p => p.Published).Select(p => new Tuple(p, p.Blog)).Select(t => t.Item1);

This is why it has to return IEnumerable<T> and belong at the end (and the use of Tuple is why it is .NET 4.0 only although that should be easy enough to change). Not all LINQ providers will necessarily register the elements with their identity map to prevent SELECT N+1 on lazy-loading but LINQ to SQL does :)

[)amien

Last week I needed to expose RSS feeds and checked out some examples online but was very disappointed.

If you find yourself contemplating writing code to solve technical problems rather than the specific business domain you work in you owe it to your employer and fellow developers to see what exists before churning out code to solve it.

The primary excuse (and I admit to using it myself) is “X is too bloated, I only need a subset. I can write that quicker than learn their solution.” but a quick reality check:

• Time – code always takes longer than you think.
• “Bloat” – indicates the problem is more complex than you realize.
• Growth – todays requirements will grow tomorrow.
• Maintenance – fixing code outside your business domain.
• Isolation – nobody coming in will know your home-grown solution.

The RSS examples I found had their own ‘feed’ and ‘items’ classes and implemented flaky XML rendering by themselves or as MVC view pages.

If these people had spent a little time doing some research they would have discovered .NET’s built in SyndicatedFeed and SyndicatedItem class for content and two classes (Rss20FeedFormatter and Atom10FeedFormatter )  to handle XML generation with correct encoding, formatting and optional fields.

All that is actually required is a small class to wire up these built-in classes to MVC.

using System;
using System.ServiceModel.Syndication;
using System.Text;
using System.Web;
using System.Web.Mvc;
using System.Xml;

namespace MyApplication.Something
{
    public class FeedResult : ActionResult
    {
        public Encoding ContentEncoding { get; set; }
        public string ContentType { get; set; }

        private readonly SyndicationFeedFormatter feed;
        public SyndicationFeedFormatter Feed{
            get { return feed; }
        }

        public FeedResult(SyndicationFeedFormatter feed) {
            this.feed = feed;
        }

        public override void ExecuteResult(ControllerContext context) {
            if (context == null)
                throw new ArgumentNullException("context");

            HttpResponseBase response = context.HttpContext.Response;
            response.ContentType = !string.IsNullOrEmpty(ContentType) ? ContentType : "application/rss+xml";

            if (ContentEncoding != null)
                response.ContentEncoding = ContentEncoding;

            if (feed != null)
                using (var xmlWriter = new XmlTextWriter(response.Output)) {
                    xmlWriter.Formatting = Formatting.Indented;
                    feed.WriteTo(xmlWriter);
                }
        }
    }
}

In a controller that supplies RSS feed simply project your data onto SyndicationItems and create a SyndicationFeed then return a FeedResult with the FeedFormatter of your choice.

public ActionResult NewPosts() {
    var blog = data.Blogs.SingleOrDefault();
    var postItems = data.Posts.Where(p => p.Blog = blog).OrderBy(p => p.PublishedDate).Take(25)
        .Select(p => new SyndicationItem(p.Title, p.Content, new Uri(p.Url)));

    var feed = new SyndicationFeed(blog.Title, blog.Description, new Uri(blog.Url) , postItems) {
        Copyright = blog.Copyright,
        Language = "en-US"
    };

    return new FeedResult(new Rss20FeedFormatter(feed));
}

This also has a few additional advantages:

1. Unit tests can ensure the ActionResult is a FeedResult
2. Unit tests can examine the Feed property to examine results without parsing XML
3. Switching to Atom format involved just changing the new Rss20FeedFormatter to Atom10FeedFormatter

[)amien

Don’t forget to check out The Visual Studio Blog for more tips, tricks and tools.

Theme VS itself

Color themes for the VS editor have been available and popular for some time but the Visual Studio Color Theme Editor adds colour themes to the VS shell letting you customize it to the most intimate detail as well as providing a bunch of pre-defined themes like Aero and shades of XP.

A bucket and a mop

CodeMaid lets you clean up your code more thoroughly and quickly including removing extra empty lines and whitespace and automatically triggering VS’s cleanup steps too (format document, remove unused strings, sort usings) as well as quick switching between project sub-items, quick-jump to complex methods etc.

Ceasefire on indentation war

The Indentation Matcher Extension detects the indentation style used when you open a file and sets your VS settings to match meaning you can just edit existing projects and solutions without a care in the world.

As it should be – or at least until Elastic tabstops gets ported to VS2010 which might now be possible.

Stylistic comments

My hacked-version of Envy Code R marked italic as bold to trick VS into using it which made a lot of people, myself included, happy. But for those who preferred Consolas it wasn’t much help (there was no way I could redistribute a modified version of Consolas but believe me it looked sweet).

VS 2010 curiously still spurns italic fonts but the pluggable editor means extensions like ItalicComments can get you there although you’ll need to grab the source from gitHub to set it to your coding font of choice given the curious decision to hard-code Lucida Sans.

My Engrish is gud

Until Windows gets an OS-level spell checker (OS X had one in 2000) we’ll have to be content with each major app having it’s own or in the case of VS, none.

The aptly-named Spell Checker extension adds English spell checking to comments, HTML text, strings etc. and you too can avoid embarrassing mistakes preserved in source control for all to see.

[)amien

Lazy loading with stored procedures

LINQ to SQL supports stored procedures for retrieving entities, insert, update and delete operations but you can also use them to perform lazy-loading of navigation properties.

Lets show an example of a bi-directional relationship between a Post and a Comment. We have two stored procedures shown below and we bring them into the DBML by dragging them from Server Explorer into the LINQ to SQL designer surface and we set the return type property for each to the expected entity (Post and Comment respectively).

CREATE PROCEDURE LoadPost (@PostID int) AS SELECT * FROM Posts WHERE ID = @PostID
CREATE PROCEDURE LoadComments(@PostID int) AS SELECT * FROM Comments WHERE Parent_Post_ID = @PostID

This generates two method stubs named LoadPost and LoadComments that we can use to programatically retrieve entities:

var post = dataContext.LoadPost(1).First();
Console.WriteLine("{0}", post.Title);

Now to replace LINQ to SQL’s lazy-loading query generation we add  methods to the data context subclass with a specific signature.

partial class DataClasses1DataContext {
    protected IEnumerable<Comment> LoadComments(Post post) {
        return this.LoadComments(post.ID);
    }

    protected Post LoadParentPost(Comment comment) {
        return this.LoadPost(comment.Post_ID).First();
    }
}

To get the signature of the method names right:

1. Visibility can be anything (protected or private is recommended)
2. Return type must be the type of the other side of the association (wrapped in IEnumerable<T> when that side can be many)
3. Method name must start with the word “Load”
4. Method name must then continue with the name of the navigation property you want to intercept
5. Parameter type must be the type that has the named navigation property (step 4)

Storing and retrieving binary files

LINQ to SQL supports the SQL Server’s varbinary type but storing something practical like a file in there isn’t so clear. Map your varbinary(max) column from your table into your entity which will expose the column as the special System.Data.Linq.Binary type (effectively a wrapper for a byte array but better change tracking).

File to database

To store a file in the database just read those bytes in and assign them to the property (Binary knows how to create itself from a byte array automatically). e.g.

string readPath = @"c:\test.jpg";
var storedFile = new StoredFile();
storedFile.Binary = File.ReadAllBytes(readPath);
storedFile.FileName = Path.GetFileName(readPath);
data.StoredFiles.InsertOnSubmit(storedFile);

I recommend storing the file name as well as the binary contents for two reasons. Firstly writing the file back to disk or streaming it to a browser will require you know the file type (e.g. .jpg or image/jpeg) and secondly nobody likes downloading a a file called ‘download’ or ’1′ :)

Database to file

Writing the file back to disk is just as easy although you have to use the ToArray() method of System.Data.Linq.Binary to turn it back into a byte array.

string writePath = @"c:\temp";
var storedFile = data.StoredFiles.First();
File.WriteAllBytes(Path.Combine(writePath, storedFile.FileName), storedFile.Binary.ToArray());

Multiple databases with a single context

Contrary to popular belief you can in fact access entities from multiple databases with a single data context providing they live on the same server. This isn’t supported but I’ve used it on my own projects without issue :)

The first part is the tricky bit which involves getting the definition of your entity into your DBML. You have two options here:

Create a temporary view

If you have the rights you can temporarily create views in your primary database for each table in your non-primary database.

CREATE VIEW MyOtherTable AS SELECT * FROM MyOtherDatabase.dbo.MyOtherTable

Once the views are created add them to your DBML by dragging them from Server Explorer into the LINQ to SQL designer surface and delete the views you created from the database.

Create a temporary DBML

If you can’t or don’t want to create temporary views then add a second (temporary) LINQ to SQL classes file (DBML) to your project. Use Server Explorer to find your secondary database and drag all the tables you will want to access to the LINQ to SQL designer surface.

Now save & close open files and use the right-mouse-button context menu to Open With… and choose XML Editor on your original DBML and the new temporary one. Head to the Window menu and select New Vertical Tab Group to make the next step easier.

Looking through the DBML you will see each entity has a <Table> block inside the <Database>. Select all the Table tags and their children (but not Database or Connection) and copy/paste them into your existing DBML file. Then close the files and check all looks well in the designer again.

If it does, delete the temporary DBML file you created. If not go back and check the DBML file for duplicate names, mismatched XML etc.

Finally, the easy bit

Open the designer and for each table that comes from the other database select it and change the Source property in the Properties window from dbo.MyOtherTable to MyOtherDatabase.dbo.MyOtherTable.

Hit play and run!

[)amien

SQL Server like all databases goes through a number of steps when it receives a command. Besides parsing and validating the command text and parameters it looks at the database schema, statistics and indexes to come up with a plan to efficiently query or change your data.

You can view the plan SQL Server comes up with for a given query in SQL Management Studio by selecting Include Actual Execution Plan from the Query menu before running your query.

A query plan in SQL Managment Studio

Show me the cache!

Query plans are cached so subsequent identical operations can reuse them for further performance gains. You can see the query plans in use on your server with the following SQL:

SELECT objtype, p.size_in_bytes, t.[text], usecounts
     FROM sys.dm_exec_cached_plans p
     OUTER APPLY sys.dm_exec_sql_text (p.plan_handle) t
     WHERE objtype IN ('Prepared', 'Adhoc')
     ORDER BY usecounts DESC

Hitting the cache

DBAs know the value in hitting the query plan often and this is one of the reasons they like stored procedures. You can however achieve the same thing with parameterized queries providing the query text and the parameter definitions are identical so you can execute the same thing over and over again just with different parameters.

If your ORM uses parameterized queries then it too can take advantage of it but it is important to remember the query definition and parameters need to be identical for this to happen.

How this applies to ORMs

In .NET 3.5SP1 both LINQ to SQL and Entity Framework did not set the length of variable type parameters (varchar, nvarchar, text, ntext and varbinary) so SQL Client sets it to the actual content length. This means the cache is often missed and instead populated with plans that are different only in the parameter lengths.

In .NET 4.0 variable length parameters now honour the defined length in both LINQ to SQL and Entity Framework where possible or fall back to the maximum length when the actual content doesn’t fit in the defined length.

[)amien

While useful it had a few issues:

• Getting start/end blocks mixed up resulted in unpredictable behaviour
• Files were rewritten even when content did not change
• Did not play well with source control
• Files not always deleted in VS
• Failed in Visual Studio’s project-less Web Sites

This helper class forms the basis of multiple file output for Entity Framework templates in .NET 4.0 and the LINQ to SQL templates on CodePlex so we (Jeff Reed, Andrew Peters and myself) made the following changes.

Improvements

Simpler block handling

The header, footer and file blocks can now be completed with EndBlock (EndHeader and EndFooter are gone), although it will automatically end the previous block when it hits a new one or the final Process method.

Skip unchanged files

Files are now only written to disk if the contents are different with the exception of the original T4 output file (we can’t stop that, sorry).

There is additional overhead reading and comparing files we believe unmodified files keeping their dates and source control status are worth it.

Automatic checkout

When the template detects it is running in Visual Studio and that the file it needs to write to is currently in source control but not checked out it will check the file out for you.

Predictable clean-up

All files that were not part of the generation process but are nested under the project item will now be deleted when running inside Visual Studio.

Outside of Visual Studio files are no longer deleted – this was destructive and it couldn’t know which files it generated on a previous run to clean-up correctly anyway.

Website projects fall back to single file generation

Visual Studio has both web sites and web applications with the former being project-less leading to very messy multi-file generation so it forces single file generation.

Internal improvements

Source is now simpler to read and understand with less public visibility and faster and more robust VS interop by batching the files & deletes to a single invoke at the end to avoid conflicts with other add-in’s that might be triggered by the changes.

Usage

Initialization

You’ll need to get the code into your template – either copy the code in or reference it with an include directive. Then declare an instance of the Manager class passing in some environmental options such as the desired default output path. (For Visual Studio 2010 remove the #v3.5 portion from the language attribute)

<#@ template language="C#v3.5" hostspecific="True"
#><#@include file="Manager.ttinclude"
#><# var manager = Manager.Create(Host, GenerationEnvironment); #>

File blocks

Then add one line before and one line after each block which could be split out into it’s own file passing in what the filename would be if split. The EndBlock is optional if you want it to carry through to the next one :)

<# manager.StartNewFile("Employee.generated.cs"); #>
public class Employee { … }
<# manager.EndBlock(); #>

Headers & footers

Many templates need to share a common header/footer for such things as comments or using/import statements or turning on/off warnings. Simply use StartHeader and StartFooter and the blocks will be emitted to the start and end of all split files as well as being left in the original output file.

<# manager.StartHeader(); #>
// Code generated by a template
using System;

<# manager.EndBlock(); #>

Process

At the end of the template call Process to handle splitting the files (true) or not (false). Anything not included in a specific StartNewFile block will remain in the original output file.

<# manager.Process(true); #>

Revised Manager class

<#@ assembly name="System.Core"
#><#@ assembly name="System.Data.Linq"
#><#@ assembly name="EnvDTE"
#><#@ assembly name="System.Xml"
#><#@ assembly name="System.Xml.Linq"
#><#@ import namespace="System"
#><#@ import namespace="System.CodeDom"
#><#@ import namespace="System.CodeDom.Compiler"
#><#@ import namespace="System.Collections.Generic"
#><#@ import namespace="System.Data.Linq"
#><#@ import namespace="System.Data.Linq.Mapping"
#><#@ import namespace="System.IO"
#><#@ import namespace="System.Linq"
#><#@ import namespace="System.Reflection"
#><#@ import namespace="System.Text"
#><#@ import namespace="System.Xml.Linq"
#><#@ import namespace="Microsoft.VisualStudio.TextTemplating"
#><#+

// Manager class records the various blocks so it can split them up
class Manager {
    private class Block {
        public String Name;
        public int Start, Length;
    }

    private Block currentBlock;
    private List<Block> files = new List<Block>();
    private Block footer = new Block();
    private Block header = new Block();
    private ITextTemplatingEngineHost host;
    private StringBuilder template;
    protected List<String> generatedFileNames = new List<String>();

    public static Manager Create(ITextTemplatingEngineHost host, StringBuilder template) {
        return (host is IServiceProvider) ? new VSManager(host, template) : new Manager(host, template);
    }

    public void StartNewFile(String name) {
        if (name == null)
            throw new ArgumentNullException("name");
        CurrentBlock = new Block { Name = name };
    }

    public void StartFooter() {
        CurrentBlock = footer;
    }

    public void StartHeader() {
        CurrentBlock = header;
    }

    public void EndBlock() {
        if (CurrentBlock == null)
            return;
        CurrentBlock.Length = template.Length - CurrentBlock.Start;
        if (CurrentBlock != header && CurrentBlock != footer)
            files.Add(CurrentBlock);
        currentBlock = null;
    }

    public virtual void Process(bool split) {
        if (split) {
            EndBlock();
            String headerText = template.ToString(header.Start, header.Length);
            String footerText = template.ToString(footer.Start, footer.Length);
            String outputPath = Path.GetDirectoryName(host.TemplateFile);
            files.Reverse();
            foreach(Block block in files) {
                String fileName = Path.Combine(outputPath, block.Name);
                String content = headerText + template.ToString(block.Start, block.Length) + footerText;
                generatedFileNames.Add(fileName);
                CreateFile(fileName, content);
                template.Remove(block.Start, block.Length);
            }
        }
    }

    protected virtual void CreateFile(String fileName, String content) {
        if (IsFileContentDifferent(fileName, content))
            File.WriteAllText(fileName, content);
    }

    public virtual String GetCustomToolNamespace(String fileName) {
        return null;
    }

    public virtual String DefaultProjectNamespace {
        get { return null; }
    }

    protected bool IsFileContentDifferent(String fileName, String newContent) {
        return !(File.Exists(fileName) && File.ReadAllText(fileName) == newContent);
    }

    private Manager(ITextTemplatingEngineHost host, StringBuilder template) {
        this.host = host;
        this.template = template;
    }

    private Block CurrentBlock {
        get { return currentBlock; }
        set {
            if (CurrentBlock != null)
                EndBlock();
            if (value != null)
                value.Start = template.Length;
            currentBlock = value;
        }
    }

    private class VSManager: Manager {
        private EnvDTE.ProjectItem templateProjectItem;
        private EnvDTE.DTE dte;
        private Action<String> checkOutAction;
        private Action<IEnumerable<String>> projectSyncAction;

        public override String DefaultProjectNamespace {
            get {
                return templateProjectItem.ContainingProject.Properties.Item("DefaultNamespace").Value.ToString();
            }
        }

        public override String GetCustomToolNamespace(string fileName) {
            return dte.Solution.FindProjectItem(fileName).Properties.Item("CustomToolNamespace").Value.ToString();
        }

        public override void Process(bool split) {
            if (templateProjectItem.ProjectItems == null)
                return;
            base.Process(split);
            projectSyncAction.EndInvoke(projectSyncAction.BeginInvoke(generatedFileNames, null, null));
        }

        protected override void CreateFile(String fileName, String content) {
            if (IsFileContentDifferent(fileName, content)) {
                CheckoutFileIfRequired(fileName);
                File.WriteAllText(fileName, content);
            }
        }

        internal VSManager(ITextTemplatingEngineHost host, StringBuilder template)
            : base(host, template) {
            var hostServiceProvider = (IServiceProvider) host;
            if (hostServiceProvider == null)
                throw new ArgumentNullException("Could not obtain IServiceProvider");
            dte = (EnvDTE.DTE) hostServiceProvider.GetService(typeof(EnvDTE.DTE));
            if (dte == null)
                throw new ArgumentNullException("Could not obtain DTE from host");
            templateProjectItem = dte.Solution.FindProjectItem(host.TemplateFile);
            checkOutAction = (String fileName) => dte.SourceControl.CheckOutItem(fileName);
            projectSyncAction = (IEnumerable<String> keepFileNames) => ProjectSync(templateProjectItem, keepFileNames);
        }

        private static void ProjectSync(EnvDTE.ProjectItem templateProjectItem, IEnumerable<String> keepFileNames) {
            var keepFileNameSet = new HashSet<String>(keepFileNames);
            var projectFiles = new Dictionary<String, EnvDTE.ProjectItem>();
            var originalFilePrefix = Path.GetFileNameWithoutExtension(templateProjectItem.get_FileNames(0)) + ".";
            foreach(EnvDTE.ProjectItem projectItem in templateProjectItem.ProjectItems)
                projectFiles.Add(projectItem.get_FileNames(0), projectItem);

            // Remove unused items from the project
            foreach(var pair in projectFiles)
                if (!keepFileNames.Contains(pair.Key) && !(Path.GetFileNameWithoutExtension(pair.Key) + ".").StartsWith(originalFilePrefix))
                    pair.Value.Delete();

            // Add missing files to the project
            foreach(String fileName in keepFileNameSet)
                if (!projectFiles.ContainsKey(fileName))
                    templateProjectItem.ProjectItems.AddFromFile(fileName);
        }

        private void CheckoutFileIfRequired(String fileName) {
            var sc = dte.SourceControl;
            if (sc != null && sc.IsItemUnderSCC(fileName) && !sc.IsItemCheckedOut(fileName))
                checkOutAction.EndInvoke(checkOutAction.BeginInvoke(fileName, null, null));
        }
    }
} #>

[)amien

One solution is to write methods by hand, another is to code generate them but it would be nice to be able to do:

var customers = new SqlCommand("SELECT ID, Name FROM Customer", connection)
  .As(r => new Customer { CustomerID = r.GetInt32(0), Name = r.GetString(1) }).ToList();

So for any DbCommand object you can turn it into a bunch of classes by specifying the new pattern.

The tiny helper class to achieve this is:

public static class DataHelpers {
    public static List<T> ToList<T>(this IEnumerable<T> enumerable) {
        return new List<T>(enumerable);
    }

    public static IEnumerable<T> As<T>(this DbCommand command, Func<IDataRecord, T> map) {
        using (var reader = command.ExecuteReader())
            while (reader.Read())
                yield return map(reader);
    }
}

It might even be possible to do some cool caching/materialization. I should look into that :)

[)amien

It shows the syntax for a number of common query operations, manipulations and attributes and can be a very useful quick reference :)

Download LINQ to SQL cheat sheet (PDF) (76 KB)

[)amien

private static Dictionary<string,Employee> employees;
…
public static Employee GetByName(string name) {
    Employee employee;
    if (!employees.TryGetValue(name, out employee)) {
        employee = new Employee(whatever);
        employees.Add(name, employee);
    }
    return employee;
}

It’s not that it is particularly difficult but it can be a bit error prone and when you’re doing it over and over. What would be nicer is something that let you do:

public static Employee GetByName(string name) {
    return employees.GetOrAdd(name, () => new Employee(whatever));
}

Here’s an extension method to drop-in to a static class of your choosing that achieves just that.

public static TDictionaryValue GetOrAdd<TKey, TDictionaryValue>(this IDictionary<TKey, TDictionaryValue> dictionary, TKey key, Func<TDictionaryValue> newValue)
{
    TDictionaryValue value;
    if (!dictionary.TryGetValue(key, out value)) {
        value = newValue.Invoke();
        dictionary.Add(key, value);
    }
    return value;
}

[)amien

If you write properties on the client you can’t use them in remote LINQ operations.

The problem occurs because these properties can’t be translated and sent to the server as they have been compiled into intermediate language (IL) and not LINQ expression trees that are required for translation by IQueryable implementations. There is nothing available in .NET to let us reverse-engineer the IL back into the methods and syntax that would allow us to translate the intended operation into a remote query.

This means you end up having to write your query in two parts; firstly the part the server can do, a ToList or AsEnumerable call to force that to happen and bring the intermediate results down to the client, and then the operations that can only be evaluated locally. This can hurt performance if you want to reduce or transform the result set significantly.

What we came up (David, Colin Meek and myself) is a provider-independent way of declaring properties just once so they can be used in both scenarios. Computed properties for LINQ to SQL, LINQ to Entities and anything else LINQ enabled with little effort and it works great on .NET 3.5 SP1 :)

Before example

Here we have extended the Employee class to add Age and FullName. We only wanted to people with “da” in their name but we are forced to pull down everything to the client in order to the do the selection.

partial class Employee {
	public string FullName {
		get { return Forename + " " + Surname; }
	}

	public int Age {
		get { return DateTime.Now.Year - BirthDate.Year -
			(DateTime.Now.Month < BirthDate.Now.Month
			|| DateTime.Now.Month == BirthDate.Now.Month && DateTime.Now.Day < BirthDate.Now.Day) ? 1 : 0);
		}
	}
}
...
var employees = db.Employees.ToList().Where(e => e.FullName.Contains("da")).GroupBy(e => e.Age);

After example

Here using our approach it all happens server side… and works on both LINQ to Entities and LINQ to SQL.

partial class Employee {
    private static readonly CompiledExpression<Employee,string> fullNameExpression
     = DefaultTranslationOf<Employee>.Property(e => e.FullName).Is(e => e.Forename + " " + e.Surname);
    private static readonly CompiledExpression<Employee,int> ageExpression
     = DefaultTranslationOf<Employee>.Property(e => e.Age).Is(e => DateTime.Now.Year - e.BirthDate.Value.Year - ((DateTime.Now.Month < e.BirthDate.Value.Month || (DateTime.Now.Month == e.BirthDate.Value.Month && DateTime.Now.Day < e.BirthDate.Value.Day)) ? 1 : 0)));

    public string FullName {
        get { return fullNameExpression.Evaluate(this); }
    }

    public int Age {
        get { return ageExpression.Evaluate(this); }
    }
}
...
var employees = db.Employees.Where(e => e.FullName.Contains("da")).GroupBy(e => e.Age).WithTranslations();

Getting started

Check out this download which includes the necessary project to drop-in to your solution. The caveat to the usage technique shown above is you need to ensure your class has been initialized before you write queries to it. If this is a problem check out the usage considerations section below.

The other major caveat is obviously the expression you register for a property must be able to be translated to the remote store so you will need to constrain yourself to the methods and operators your IQueryable provider supports.

Usage considerations

There are a few alternative ways to use this rather than the specific examples above.

Registering the expressions

You can register the properties in the class itself as shown in the examples which means the properties themselves can evaluate the expressions without any reflection calls. Alternatively if performance is less critical you can register them elsewhere and have the methods look up their values dynamically via reflection. e.g.

...
DefaultTranslationOf<Employee>.Property(e => e.FullName).Is(e => e.Forename + " " + e.Surname);
var employees = db.Employees.Where(e => e.FullName.Contains("da")).GroupBy(e => e.Age).WithTranslations();
...
partial class Employee {
    public string FullName { get { return DefaultTranslationOf<Employees>.Evaluate<string>(this, MethodInfo.GetCurrentMethod());} }
}

If performance of the client-side properties is critical then you can always have them as regular get properties with the full code in there at the expense of having the calculation duplicated, once in IL in the property and once as an expression for the translation.

Different maps for different scenarios

Sometimes certain parts of your application may want to run with different translations for different scenarios, performance etc. No problem!

The WithTranslations method normally operates against the default translation map (accessed with DefaultTranslationOf) but there is also another overload that takes a TranslationMap you can build for specific scenarios, e.g.

var myTranslationMap = new TranslationMap();
myTranslationMap.Add<Employees, string>(e => e.Name, e => e.FirstName + " " + e.LastName);
var results = (from e in db.Employees where e.Name.Contains("martin") select e).WithTranslations(myTranslationMap).ToList();

Not specifying WithTranslation everywhere

If you are happy to always have the default translation applied simply add the following statement to the top of your file which will bring in a bunch of extensions methods for the usual LINQ query operators that already apply WithTranslation for you :)

using Microsoft.Linq.Translations.Auto;

With .NET 4.0

You should be able to drop the ExpressionVisitor class in the download and reference the built-in one.

How it works

CompiledExpression<T, TResult>

The first thing we needed to do was get the user-written client-side “computed” properties out of IL and back into expression trees so we could translate them. Given that we also want to evaluate them on the client we need to compile them at run time so CompiledExpression exists which just takes an expression of Func<T, TResult>, compiles it and allows evaluation of objects against the compiled version.

ExpressiveExtensions

This little class provides both the WithTranslations extensions methods and the internal TranslatingVisitor that unravels the property accesses into their actual registered Func<T, TResult> expressions via the TranslationMap so that the underlying LINQ provider can deal with that instead.

TranslationMap

We need to have a map of properties to compiled expressions and for that purpose TranslationMap exists. You can create a TranslationMap by hand and pass it in to WithTranslations if you want to programmatically create them at runtime or have different ones for different scenarios but generally you will want to use…

DefaultTranslationOf

This helper class lets you register properties against the default TranslationMap we use when nothing is passed to WithTranslations. It also allows you to lookup what is already registered so you can evaluate to that although there is a small reflection performance penalty for that:

public int Age { get { return DefaultTranslationOf<Employees>.Evaluate<int>(this, MethodInfo.GetCurrentMethod()); } }

AutoTranslation

Remembering to specify WithTranslations everywhere can be a pain so if you always want to go via translation and use the default TranslationMap you can include the Microsoft.Linq.Translations.Auto namespace in your code which includes extension methods for all the regular LINQ operations that saves you having to remember.

Have fun!

[)amien

Templates

• Entity Developer
  Add-in for Visual Studio that provides a replacement designer with code templates. Commercial ($99.95)
• LLBLGen Pro
  This ORM templating and design tool has a set of LINQ to SQL templates available for generating C# code including file per entity that can be used in conjunction with their designer. Templates are free but require LLBLGen Pro licence (€179-249).
• L2ST4
  My templates use Visual Studio’s built in T4 engine and provide a great starting point to customizing the code generation process. Everything that SQL Metal/LINQ to SQL designer can generate is handled including C# and VB.NET generation and are freely licensed under the MS-PL.
• PLINQO
  These templates for Eric Smith’s ever-popular CodeSmith templating environment include a whole host of extra functionality beyond the standard generation including entity clone & detach, enum’s from tables, data annotations, batching, auditing and more. Templates are free but require CodeSmith licence ($79-$299).
• T4 Toolbox
  Oleg Sych has put together a suite of useful T4 templates including ones for producing LINQ to SQL entities, data contexts as well as SQL scripts for altering the schema to reflect changes. C# only, MS-RL.

Note: While the T4 templating language is built-in to Visual Studio 2008/2010 it does not come with syntax highlighting or IntelliSense. Check out either:

• Clarius Visual T4
  Basic ‘community’ version available for free, commercial ‘pro’ version with IntelliSense, sub-templates, preview, user preferences etc. is $99.
• Tangible T4
  Free version available with limited IntelliSense, commercial ‘pro’ version with UML modelling etc. available for $99.

Blogs

• David DeWinter
  David is a dev in test who recently joined our team and hit the ground running with testing, blogging and helping out on the forums.
• Roger Jennings
  Roger over at OakLeaf Systems publishes regular articles and roundups of some of the best .NET data access content from around the web including LINQ to SQL.
• Scott Guthrie
  Our Corporate Vice President for the .NET Developer Platform takes a very active role in getting his hands dirty and publishes a series of useful LINQ to SQL articles.
• Sidar Ok
  Sidar is a regular forum helper and has written a number of great posts on LINQ to SQL including some good POCO coverage.

Tools

• Devart dotConnect
  Devart’s dotConnect family are database providers for Oracle, MySQL, Postgres and SQLite that also include LINQ support to enable LINQ to SQL like functionality on other platforms. Some database basic versions are free, professional versions are commercial and vary in price (~$99-$209).
• Hugati DBML/EDMX Tools
  Add-in for Visual Studio that provides comparison/update facilities between the database and the DBML as well as standardising names and generating interfaces. Commercial ($49-$119, free 30 day trial).
• Hugati LINQ to SQL Profiler
  Profiling tool to help optimize your LINQ to SQL based applications. Commercial ($49-$119, free 45 day trial).
• LINQpad
  This invaluable tool helps you write and visualise your LINQ queries in a test-bench without compilation and includes a version for the .NET 4.0 beta. Free to use, auto-completion add-on available ($19).
• LINQ to SQL Cheat Sheet
  PDF download of the most popular query and update syntax for C# and VB.NET.

Official guides

• Samples
  The official samples includes a whopping 101 snippets showing how to use many of the features and syntax.
• Programming Guide
  Includes steps on how to get started, querying, making changes, debugging and background information.
• Whitepaper
  Single document describing the architecture, query capabilities, change tracking and lifecycle, multi-tier entities, external mapping etc.
• Changes in .NET 4.0
  List of the changes made to LINQ to SQL to .NET 4.0 including some possible breaking changes

Books

• Essential LINQ
• LINQ in Action
• Pro ADO.NET 3.5 with LINQ
• Pro LINQ Object Relational Mapping
• Programming Microsoft LINQ

Support

• Official MSDN forums
  Great way to get access to the product team directly as well as knowledgeable and experienced users if you have a question or a problem.
• Official LINQ to SQL FAQ
  Coverage is a little thin on the ground but it has some useful tips.
• StackOverflow’s LINQ to SQL tag
  StackOverflow has rapidly become a leader in questions and answers for a wide variety of developer topics and covers LINQ to SQL (which the site uses for it’s data access too!)

[)amien

25 Aug 2009 – Updated with additional changes, some of which are new in beta 2.

Change list

Performance

• Query plans are reused more often by specifically defining text parameter lengths (when connecting to SQL 2005 or later)
• Identity cache lookups for primary key with single result now includes query.Where(predicate).Single/SingleOrDefault/First/FirstOrDefault
• Reduced query execution overhead when DataLoadOptions specified (cache lookup considers DataLoadOptions value equivalency)

Usability

• ITable<T> interface for additional mocking possibilities
• Contains with enums automatically casts to int or string depending on column type
• Associations can now specify non-primary-key columns on the other end of the association for updates
• Support list initialization syntax for queries
• LinqDataSource now supports inherited entities
• LinqDataSource support for ASP.NET query extenders added

Query stability

• Contains now detects self-referencing IQueryable and doesn’t cause a stack overflow
• Skip(0) no longer prevents eager loading
• GetCommand operates within SQL Compact transactions
• Exposing Link<T> on a property/field is detected and reported correctly
• Compiled queries now correctly detect a change in mapping source and throw
• String.StartsWith, EndsWith and Contains now correctly handles ~ in the search string (regular & compiled queries)
• Now detects multiple active result sets (MARS) better
• Associations are properly created between entities when using eager loading with Table-Valued Functions (TVFs)
• Queries that contain sub-queries with scalar projections now work better

Update stability

• SubmitChanges no longer silently consumes transaction rollback exceptions
• SubmitChanges deals with timestamps in a change conflict scenario properly
• IsDbGenerated now honors renamed properties that don’t match underlying column name
• Server-generated columns and SQL replication/triggers now work instead of throwing SQL exception
• Improved binding support with the MVC model binder

General stability

• Binary types equate correctly after deserialization
• EntitySet.ListChanged fired when adding items to an unloaded entity set
• Dispose our connections upon context disposal (ones passed in are untouched)

Database  control

• DeleteDatabase no longer fails with case-sensitive database servers

SQL Metal

• Foreign key property setter now checks all affected associations not just the first
• Improved error handling when primary key type not supported
• Now skips stored procedures containing table-valued parameters instead of aborting process
• Can now be used against connections that use AttachDbFilename syntax
• No longer crashes when unexpected data types are encountered

LINQ to SQL class designer

• Now handles a single anonymously named column in SQL result set
• Improved error message for associations to nullable unique columns
• No longer fails when using clauses are added to the partial user class
• VarChar(1) now correctly maps to string and not char
• Decimal precision and scale are now emitted correctly in the DbType attributes for stored procedures & computed columns
• Foreign key changes will be picked up when bringing tables back into the designer without a restart
• Can edit the return value type of unidentified stored procedure types
• Stored procedure generated classes do not localize the word “Result” in the class name
• Opening a DBML file no longer causes it to be checked out of source control
• Changing a FK for a table and re-dragging it to the designer surface will show new FK’s

Code generation (SQL Metal + LINQ to SQL class designer)

• Stored procedures using original values now compiles when the entity and context namespaces differ
• Virtual internal now generates correct syntax
• Mapping attributes are now fully qualified to prevent conflicts with user types
• KnownTypeAttributes are now emitted for DataContractSerializer with inheritance
• Delay-loaded foreign keys now have the correct, compilable, code generated
• Using stored procedures with concurrency no longer gets confused if entities in different namespace to context
• ForeignKeyReferenceAlreadyHasValueException is now thrown if any association is loaded not just the first

Potentially breaking changes

We worked very hard to avoid breaking changes but of course any potential bug fix is a breaking change if your application was depending on the wrong behavior. The ones I specifically want to call out are:

Skip(0) is no longer a no-op

The special-casing of 0 for Skip to be a no-op was causing some subtle issues such as eager loading to fail and we took the decision to stop special casing this. This means if you had syntax that was invalid for a Skip greater than 0 it will now also be invalid for skip with a 0. This makes more sense and means your app would break on the first page now instead of subtlety breaking on the second page. Fail fast :)

ForeignKeyReferenceAlreadyHasValue exception

If you are getting this exception where you weren’t previously it means you have an underlying foreign key with multiple associations based on it and you are trying to change the underlying foreign key even though we have associations loaded.Best thing to do here is to set the associations themselves and if you can’t do that make sure they aren’t loaded when you want to set the foreign key to avoid inconsistencies.

[)amien

Take full control of  the TSQL

There are times when LINQ to SQL refuses to cook up the TSQL you wanted either because it doesn’t support the feature or because it has a different idea what makes an optimal query.

In either case the Translate method allows you to deliver your own TSQL to LINQ to SQL to process as if it were its own with execution, materialization and identity mapping still honored. For example:

var db = new PeopleContext();
if (db.Connection.State == System.Data.ConnectionState.Closed)
    db.Connection.Open();
var cmd = db.GetCommand(db.Persons.Where(p => p.CountryID == 1));
cmd.CommandText = cmd.CommandText.Replace("[People] AS [t0]", "[People] AS [t0] WITH (NOLOCK)");
var results = db.Translate<Person>(cmd.ExecuteReader());

Complex stored procedures

When working with stored procedures the LINQ to SQL designer and SQLMetal tools need a way of figuring out what the return type will be. In order to do this without actually running the stored procedure itself they use the SET FMTONLY command set to ON so that SQL Server will just parse the stored procedure instead.

Unfortunately this parsing does not extend to dynamic SQL or temporary tables so you must change the return type from the scalar integer to one of the known entity types by hand. You could use the following command at the start to let it run regardless given the subsequent warning.

SET FMTONLY OFF

If your stored procedure can not safely handle being called at any time with null parameters set the return type by hand instead.

Cloning an entity

There are many reasons you might want to clone an entity – you may want to create many similar ones, you could want to keep it around longer than the DataContext it came from – whatever your reason implementing a Clone method can be a pain but taking advantage of the DataContractSerializer can make light work of this providing your DBML is set to enable serialization.

If you use discriminator subclassing you will need to either ensure your type is cast to its concrete type or use my L2ST4 templates for now as .NET 3.5 SP1 doesn’t emit the necessary KnownType attributes to make this automatically happen (fixed in .NET 4.0). Add a simple method to serialize in-memory like this:

public static T Clone<T>(T source) {
    var dcs = new System.Runtime.Serialization.DataContractSerializer(typeof(T));
    using (var ms = new System.IO.MemoryStream()) {
        dcs.WriteObject(ms, source);
        ms.Seek(0, System.IO.SeekOrigin.Begin);
        return (T)dcs.ReadObject(ms);
    }
}

And then to clone simply:

var source = myQuery.First();
var cloned = Clone(source);

Be aware that this comes with a little overhead in the serialization and deserialization process.

If that is a problem for you then why not grab those templates and make your entities implement ICloneable!

[)amien

Loading a delay-loaded property

LINQ to SQL lets you specify that a property is delay-loaded meaning that it is not normally retrieved as part of normal query operations against that entity. This is particularly useful for binary and large text fields such as a photo property on an employee object that is rarely used and would cause a large amount of memory to be consumed on the client not to mention traffic between the SQL and application.

There are times however when you want all these binaries returned in a single query, say for example returning all the photos for the company photo intranet page:

var db = new NorthwindContext();
var loadOptions = new DataLoadOptions();
loadOptions.LoadWith<Employee>(e => e.Photo);
db.LoadOptions = loadOptions;

Multiple entity types from a single stored procedure

It is actually possible to return multiple entity types from a single stored procedure in LINQ to SQL although this is not well known as the LINQ to SQL designer doesn’t actually support it. Indeed to generate the code it is necessary to hand-edit the DBML and then use SQL Metal (or my T4 template) to generate the required method signature.

In fact it is just much easier to write the code yourself and add it to the non-generated portion of your data context. If you imagine a stored procedure GetStaticData that looks like this:

CREATE PROCEDURE GetStaticData AS
  SELECT * FROM Region
  SELECT * FROM Categories
  SELECT * FROM Territories

Then all you need to do is write a method signature that looks like this (the sequence of result type attributes must match the order in the stored procedure):

[Function(Name=@"dbo.DynamicContractsActiveBetween")]
[ResultType(typeof(Region))]
[ResultType(typeof(Category))]
[ResultType(typeof(Territory))]
public IMultipleResults GetStaticData() {
   return (IMultipleResults) ExecuteMethodCall(this, (MethodInfo) MethodInfo.GetCurrentMethod()).ReturnValue;
}

Intercepting create, update and delete operations

There are times it is useful to be able to listen in to when these events happen and perform your own logic, perhaps auditing or logging for some scenarios. The easiest way to do this is to implement some specially-named methods on your data context, perform your action and then to dispatch the call back to LINQ to SQL.

The format of these specially-named methods is [Action][Entity] and then you should pass back control to LINQ to SQL using ExecuteDynamic[Action] where [Action] is either Insert, Update or Delete. One example of such usage might be:

partial class NorthwindContext {
   partial void InsertEmployee(Employee instance) {
      instance.CreatedBy = CurrentUser;
      instance.CreatedAt = DateTime.Now;
      ExecuteDynamicInsert(instance);
   }

   partial void UpdateEmployee(Employee instance) {
      AuditEmployeeOwnerChange(instance);
      instance.LastModifiedAt = DateTime.Now;
      ExecuteDynamicUpdate(instance);
   }

   partial void DeleteEmployee(Employee instance) {
      AuditDelete(instance, CurrentUser);
      ExecuteDynamicDelete(instance);
   }
}

[)amien