25 Aug 2009 – Updated with additional changes, some of which are new in beta 2.

Change list

Performance

• Query plans are reused more often by specifically defining text parameter lengths (when connecting to SQL 2005 or later)
• Identity cache lookups for primary key with single result now includes query.Where(predicate).Single/SingleOrDefault/First/FirstOrDefault
• Reduced query execution overhead when DataLoadOptions specified (cache lookup considers DataLoadOptions value equivalency)

Usability

• ITable<T> interface for additional mocking possibilities
• Contains with enums automatically casts to int or string depending on column type
• Associations can now specify non-primary-key columns on the other end of the association for updates
• Support list initialization syntax for queries
• LinqDataSource now supports inherited entities
• LinqDataSource support for ASP.NET query extenders added

Query stability

• Contains now detects self-referencing IQueryable and doesn’t cause a stack overflow
• Skip(0) no longer prevents eager loading
• GetCommand operates within SQL Compact transactions
• Exposing Link<T> on a property/field is detected and reported correctly
• Compiled queries now correctly detect a change in mapping source and throw
• String.StartsWith, EndsWith and Contains now correctly handles ~ in the search string (regular & compiled queries)
• Now detects multiple active result sets (MARS) better
• Associations are properly created between entities when using eager loading with Table-Valued Functions (TVFs)
• Queries that contain sub-queries with scalar projections now work better

Update stability

• SubmitChanges no longer silently consumes transaction rollback exceptions
• SubmitChanges deals with timestamps in a change conflict scenario properly
• IsDbGenerated now honors renamed properties that don’t match underlying column name
• Server-generated columns and SQL replication/triggers now work instead of throwing SQL exception
• Improved binding support with the MVC model binder

General stability

• Binary types equate correctly after deserialization
• EntitySet.ListChanged fired when adding items to an unloaded entity set
• Dispose our connections upon context disposal (ones passed in are untouched)

Database  control

• DeleteDatabase no longer fails with case-sensitive database servers

SQL Metal

• Foreign key property setter now checks all affected associations not just the first
• Improved error handling when primary key type not supported
• Now skips stored procedures containing table-valued parameters instead of aborting process
• Can now be used against connections that use AttachDbFilename syntax
• No longer crashes when unexpected data types are encountered

LINQ to SQL class designer

• Now handles a single anonymously named column in SQL result set
• Improved error message for associations to nullable unique columns
• No longer fails when using clauses are added to the partial user class
• VarChar(1) now correctly maps to string and not char
• Decimal precision and scale are now emitted correctly in the DbType attributes for stored procedures & computed columns
• Foreign key changes will be picked up when bringing tables back into the designer without a restart
• Can edit the return value type of unidentified stored procedure types
• Stored procedure generated classes do not localize the word “Result” in the class name
• Opening a DBML file no longer causes it to be checked out of source control
• Changing a FK for a table and re-dragging it to the designer surface will show new FK’s

Code generation (SQL Metal + LINQ to SQL class designer)

• Stored procedures using original values now compiles when the entity and context namespaces differ
• Virtual internal now generates correct syntax
• Mapping attributes are now fully qualified to prevent conflicts with user types
• KnownTypeAttributes are now emitted for DataContractSerializer with inheritance
• Delay-loaded foreign keys now have the correct, compilable, code generated
• Using stored procedures with concurrency no longer gets confused if entities in different namespace to context
• ForeignKeyReferenceAlreadyHasValueException is now thrown if any association is loaded not just the first

Potentially breaking changes

We worked very hard to avoid breaking changes but of course any potential bug fix is a breaking change if your application was depending on the wrong behavior. The ones I specifically want to call out are:

Skip(0) is no longer a no-op

The special-casing of 0 for Skip to be a no-op was causing some subtle issues such as eager loading to fail and we took the decision to stop special casing this. This means if you had syntax that was invalid for a Skip greater than 0 it will now also be invalid for skip with a 0. This makes more sense and means your app would break on the first page now instead of subtlety breaking on the second page. Fail fast :)

ForeignKeyReferenceAlreadyHasValue exception

If you are getting this exception where you weren’t previously it means you have an underlying foreign key with multiple associations based on it and you are trying to change the underlying foreign key even though we have associations loaded.Best thing to do here is to set the associations themselves and if you can’t do that make sure they aren’t loaded when you want to set the foreign key to avoid inconsistencies.

[)amien

A little background

LINQ was a new feature in .NET 3.5 that provides a store-agnostic query language syntax using a provider model.

As part of that initiative the C# team delivered LINQ to SQL – a LINQ provider to SQL Server with additional update and access management via DataContext and mapping tools such as the designer and SQL Metal. The result is a great lightweight solution that is easy to get started with and a good data access solution where your database and objects are closely aligned.

Meanwhile Data Programmability worked on an initiative to provide abstraction between conceptual and physical models of databases to allow applications to operate independently of the database vendor and underlying physical schema. LINQ support was also added in the form of LINQ to Entities and it shipped with it’s management tools in .NET 3.5 SP1.

Which now means we have two object-relational mapping options that overlap in some areas but with very different backgrounds.

The story so far

Some time before I started in May LINQ to SQL was handed over to Data Programmability.

One of the first things you notice is that the overlap between the two object-relational mapping solutions Microsoft causes confusion and hesitation.

Do I want something lightweight and easy or do I need the extra abstraction with a richer feature set?
If I want to start with LINQ to SQL today but know that the database won’t stay under my control how would I move to Entity Framework?

These are the questions that many developers face and our team spent a lot of time looking at the differences between the two stacks, how they behave and which features were missing if you wanted to migrate from LINQ to SQL to Entity Framework v1. We put together code samples, some helpers and documentation which are now being serialized on the ADO.NET blog and left us with a better understanding of the disparity – some of which the EF team have already addressed in v2.

Where next

The decision has been made that Entity Framework is the recommended solution for LINQ to relational scenarios but we are committed to looking after our users and are approaching this in two ways.

Firstly we are going to make sure LINQ to SQL continues to operate as it should. This doesn’t just mean making sure what we had works in .NET 4.0 but also fixing a number of issues that have arisen as people pick it up for more advanced projects and put it into production environments.

Secondly we will evolve LINQ to Entities to encompass the features and ease of use that people have come to expect from LINQ to SQL. In .NET 4.0 this already includes additional LINQ operators and better persistence-ignorance.

This isn’t to say LINQ to SQL won’t ever get new features. The communities around LINQ to SQL are a continuous source of ideas and we need to consider how they fit the minimalistic lightweight approach LINQ to SQL is already valued for. Where these suggestions fit with this strategy we will be working hard to get them into the product. Some enhancements like the T4 templates can be released independently but runtime elements need to stick to the .NET Framework schedule.

In conclusion

DON’T PANIC
(in large, friendly letters)

LINQ to SQL will continue to work and EF will better address the needs of LINQ to SQL users with each new release.

[)amien

damienguard:
Methods returning "this" is a hack for fluency. Let’s get ".." added to the C# compiler to operate on previous object. a.This()..That()

lazycoder:
@damienguard I can’t decide if that’s genius or insanity. Should we add the "~" operator to refer back to the top of the inherit. chain? ;)

LostInTangent:
@damienguard I’ve started using Envy Code R for most of my applications (not just VS) and I have to say I’m loving it.

damienguard:
@LostInTangent: Envy Code R PR8 soon – Greek chars, improved hinting and some glyph revisions subscript/fractions & *96 redone.

damienguard:
Statically typed languages are not flexible enough to develop dynamically linked libraries.

damienguard:
Renaming your wifi router StupidRouter does not alas shame it into being more reliable.

damienguard:
@command_tab: Am I the only one who finds paying for pretty UI’s to leverage free software that took much more effort to develop offensive?

damienguard:
Just took delivery on my Alps-switched keyboard… feels good so far… but let’s see if co-workers complain about the noise.

Plip:
@damienguard I CAN’T HEAR MYSELF THINK FOR THAT INFERNAL CLICKING !

damienguard:
@lancefisher The alps keyboard was from DSI USA… but don’t order one, terrible 2-key limits prevent fast typing.

damienguard:
Apple should add hobbyist to its OS X line-up. Make kernel easier to switch, remove the h/w lockdown and no support.

damienguard:
Standard windows font smoothing’s real problem is lack of scales. Convert a ClearType rendering to greyscale in Photoshop…

[)amien

While SQL Metal does a good job of turning your SQL schema into a set of classes for you it doesn’t let you customize the code generation process.

Usefully there is now a templating system built into Visual Studio 2008 called Text Templates (T4 for short).

Here is a short (369 line) experimental proof-of-concept T4 template I wrote last night that will generate a data context and associated entity classes as a starting point similar to that produced by SqlMetal.

Download of this old version no longer available, see the newer article!

Once downloaded unzip and drop the DataContext.cs.tt into your project and edit line 17 to set the connection string. You can also edit lines 18 and 19 to set the namespace and class name. The lightweight wrappers around database, table and column can be found at the end of the file – they simply wrap the SQL Server Information_Schema views as briefly as possible.

Within seconds Visual Studio should have created a code-behind file for the DataContext named DataContext.cs.cs with your generated code ready to use :) If you don’t like the way the template generates your context you can change it :)

[)amien

Delicious Library is a DVD, game and book organisation tool I’ve been using since my PowerBook G4 and a 2.0 version has been dangling from Wil Shipley’s mouth longer than I care to remember.

Windows users however will find Libra a very interesting clone and it features some of the same great features such as bar-code scanning via a web cam, tracking loans, a rendered virtual shelf and fast queries.

Unlike Delicious Library 1.x it also features sharing your library on-line, tweaking the types and rendering and a more advanced query engine and is available free for non-commercial use.

E Text Editor (like TextMate)

TextMate is a programming editor for the Mac that can be extended through the use of Bundles to provide additional syntax highlighting, menu options and command processing. It is fast, feels lightweight and therefore incredibly customisable all of which contribute to it’s success.

E Text Editor is a Windows clone of TextMate that doesn’t just mimic the user interface but also provides compatibility with TextMate bundles allowing you to take advantage of some of the many great enhancements available and at $34 is almost half the price although it doesn’t feel as snappy as it’s Mac counterpart.

Digsby (like Adium)

Adium is my instant messaging client of choice allowing me a single app to manage MSN, ICQ and Google Talk (I wish they would get basic Skype support in there too).

Digsby provides similar functionality whilst also throwing social networking (Facebook, Twitter) and email notification (Hotmail, GMail, Yahoo Mail etc.) into the mix.

Dash (like Quicksilver)

QuickSilver provides a quick keyboard-based entry system for performing a wide variety of tasks and selections within Mac OS X and what it doesn’t do can often be added with plug-ins.

Dash achieves a similar effect on Windows but I have to admit I’m not really sold on either yet. I think the movement from keyboard to mouse and back every now and then must be a good break for your hands if not for your productivity…

[)amien

My Boot Camp partition was running Vista Ultimate x86 which felt sluggish, ignored the last 1GB and bugged me with UAC. One Windows update kept failing to install which also prevented SP1 from completing.

Apple’s Boot Camp doesn’t support 64-bit Windows (except on the Mac Pro) and my 64-bit experiences have been unpleasant so far (no Flash for IE x64, limited 64-bit shell extensions, Live! refusing to install, drivers etc.) The increased x64 memory consumption would also be an issue when running in a 1.5GB virtual machine via Parallels or VMware Fusion.

Windows XP was one option but losing IIS7 and DirectX 10 would see me reinstalling Vista within weeks so I decided to try Windows 2008 Server x86.

Boot Camp happily accepted the 2008 Server x86 CD where I chose the BOOTCAMP partition, formatting it as NTFS and electing for a standard installation. The Boot Camp drivers subsequently installed without complaint, all 4GB of RAM was accessible and there are no 64-bit compatibility issues.

Microsoft are giving away 1 year evaluation copies of Windows 2008 Enterprise Server x86 as part of their Heroes Happen Here launch program for Windows 2008, SQL Server 2008 and Visual Studio 2008 if you don’t happen to have an MSDN subscription to hand. There are however a few tweaks you need to do to get a more desktop-like experience:

Install desktop features

Head into Server Manager and Add Features then choose Desktop Experience to install Windows Media Player, Aero etc.

Go into Services and set the Themes service to Automatic and Start it to make themes available and then choose Browse… from the Theme Settings in Personalisation to select %windir%\Resources\Themes\Aero.theme

Install wireless networking

This one had me stumped for a while as I thought my wireless card/drivers weren’t working. The reality is that 2008 Server has wireless networking removed by default so head into Server Manager > Add Features > Wireless LAN Service to install it.

Enabling hibernate

Open a command prompt and enter:

powercfg.exe /hibernate on

Remove annoying shutdown

Head into the registry to HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability and change the ShutdownReasonOn DWORD key to 0.

Relaxing local password policy

A controversial change I’m sure but I’d rather choose something complex and unique that will last 90+ days than something memorable every 30. Head into Local Security Policy > Account Policies > Password Policy > Maximum password age and change it to something more reasonable.

Going further

A great guide with screen-shots on additional tweaks for a more workstation-like experience also exists – wish I known about that earlier!

[)amien

In February I travelled out to Redmond for three days of interviews (one position grew to two, then three). Having read the Microsoft Jobs Blog I was prepared for long hard days but in reality the process was incredibly enjoyable and exciting.

So much so I wanted to find a desk and move in right then.

With some luck I also found myself at Hanselman’s geek dinner which involved some great discussions and the chance to meet Scott himself, Brad Wilson and Nikhil Kothari who I knew from .NET on-line community as well as some 35 other developers from both within Microsoft and the outside world. It was one fun evening and my thanks go to Scott for kindly driving me back to my hotel in Redmond town centre.

Many white-boards and a few lunches later (including an unexpected one with Phil Haack, Nikhil and two more guys from ASP.NET team – I wish I could remember all the names of the people I met!) I found myself with the hard task of choosing a position.

I settled on a developer role within the LINQ to SQL team starting mid-May and am counting down the days…

[)amien

We at the Guernsey Software Developer Forum are hosting a community event in conjunction with Microsoft to celebrate the launch of Visual Studio 2008, Windows Server 2008 and SQL Server 2008.

The event starts at 6:30pm on March 27th (next Thursday) at La Rocquettes hotel in St. Peter Port, Guernsey.

We will be giving away a bunch of goodies including the Heroes Happen Here launch packs with promotional copies of Visual Studio 2008 Standard, Windows 2008 Enterprise Server and SQL Server 2008 Developer as well as demonstrating some of the new features of Visual Studio 2008 and .NET 3.5.

I’m hoping this event will be bigger than normal GSDF events given the cool stuff we are giving away and the interest from non-developers in Windows 2008 and SQL Server 2008 so tell your friends (if you’re in Guernsey of course)

[)amien

It’s been a while since I touched VBA and Access but the DLookup function was quite useful so here it is for Excel. Read the warnings below before you use it!

Public Function DLookup(expression As String, domain As String, criteria As String, Optional connectionString As String)

    Dim con As ADODB.connection
    Set con = New ADODB.connection
    If IsMissing(connectionString) Then
        con.Open "Driver={SQL Server};Server=abc;Database=def;Trusted_Connection=Yes;"
    Else
        con.Open connectionString
    End If

    Dim rs As ADODB.Recordset
    Dim sql As String
    sql = "SELECT TOP 1 " + expression + " FROM " + domain + " WHERE " + criteria
    Set rs = con.Execute(sql)
    If Not rs.EOF Then DLookup = rs(0)
    rs.Close
    Set rs = Nothing

    con.Close
    Set con = Nothing

End Function

Copy and paste the code above into an Excel module and replace the “Driver=… with what you want to be your default connection.

Some examples of use:

To select the Name column from the Customers table where the numeric ID is held in column 2.

=DLookup("Name","Customers","ID=" & A2)

To return Forename and Surname columns joined together with a space from the Contacts table in the crm database using the postcode from column 5

=DLookup("Forname + ' ' + Surname","crm..Contact","PostCode='" & A5 & "'")

To return Description from a Products table held in another connection using a partial like match from column 1:

=DLookup("Description","Products","OrderCode LIKE '" & A1 & "%'", "DSN=SecondaryDSN")

I think you get the idea!

If that doesn’t scare you then give it a shot. As usual no warranty expressed or implied!

[)amien

I am not a lawyer but as far as I understand this means you are free to implement the standards with a promise that Microsoft will not use any patents under its control that are required to implement the specification against you.

Hopefully Apple will now address Keynote’s PowerPoint support bug so exported PPT’s works with Office 2007.

Now that the .NET Framework 3.5 source is available (for reference) and Scott Guthrie (now VP) announcing the MVC framework will be user-buildable and patchable (not redistributable) and include project templates for a number of open-source testing frameworks the future is looking very rosy.

[)amien

Former Apple engineer Jens Alfke believes Apple’s external image has been polished until featureless. The restrictive staff blogging policies, the veil of secrecy around future plans and a carefully orchestrated three-person spokes-team of Jobs, Schiller and Ive lead to a very impersonal closed business.

It certainly wasn’t always this way. The original Mac team appeared in Rolling Stone magazine with credit in about boxes, a practice that was continued at NeXT but abolished by Mac OS X Beta. Jobs makes regular comparisons between engineers and artists and touted individual thinking in the Think Different campaign and artists like recognition with signatures on art and credits on film.

Conversely Apple’s Mac OS X operating system is built on open software and standards. The kernel is derived from open elements bundled up as Darwin which Apple provides back along with compilers, debug tools, programming language, command line tools, Bonjour, device driver kit and a bunch of drivers. All are open.

The web rendering technology in Safari (WebKit based on KHTML) is also open and changes rolled back to the communities often reveal unannounced insights into Apple’s plans (e.g. Safari for Windows).

And yet how many engineers write or talk about Apple? Do you know the names of any product managers? Could you find any out with Google? (LinkedIn doesn’t count ;-)

These aren’t academic questions, what if you have a great idea for a feature you’d like to see added? How can you discuss how a product could evolve to fit your needs? What about a simple bug report or advanced access to technology? (The answers are “send it to feedback@apple.com and don’t hold your breath”, “you can’t” and “join the developer program”)

Heaven forbid you do actually find out what their plans might be – you could find yourself talking to their lawyers like the ill-fated ThinkSecret site that featured rumors, speculation and the occasional insider info.

Microsoft

Jens makes a passing mention to Microsoft’s relaxed blogging policies.

Microsoft is a company that rarely provides the source, never ships or builds upon existing free software and yet not only discusses plans and roadmaps but actively solicits feedback in the design process through conferences, user groups, forums, mailing lists and even on-site review teams. Employees such as Scott Guthrie and Brad Abrams have become quite well known within .NET communities often being the first to break announcements and provide quick feedback through their blogs.

The centre of this effort is engineering thanks to sites like Channel 9 providing regular interviews, Microsoft Research providing experiments to play with and CodePlex hosting open projects.

But they aren’t the only ones reaching out.

Microsoft’s HR & recruiting team and individuals are also putting up interesting insights and thoughts on how the company operates and head of the Xbox Live! is so active in this area that the name Major Nelson is known to any serious 360 owner.

Being open

How strange that Apple embraces open technologies yet keeps communication closed and Microsoft’s technologies are still quite closed yet communication is very much open.

What does it mean to be open and where will each company’s approach lead them?

[)amien

We utilised SQL Server’s publisher-subscriber replication on the required databases which, given their legacy nature, had some cross-database dependencies that were added without due consideration.

The reporting SQL pair needed to be rebuilt so the subscriptions were removed whilst they were being rebuilt. Within three days our primary servers were out of disk space and we had a problem that needed to be solved quickly.

Our first step was to remove the new publications. This released a small amount of space in the distribution database (now 100GB) and allowed the replicated databases logs to be truncated and shrunk. We now had a little breathing room.

One publication would not remove in a reasonable time (10+ hours) and even trying to remove individual articles caused a slow-running operation sat on PAGEIOLATCH_SH (which Microsoft describe as waiting on disk IO). A quick investigation revealed a complex query on internal system tables with no indexes – presumably for fast insert performance.
In the mean-time this slow-running operation would block the replication agents which are also used by IBM’s DataMirror product to replicate data back to the iSeries (AS/400) in real time. This blocking would prevent us shipping orders and was therefore unacceptable.

The database in question is a home-grown data warehousing application rebuilt each night and on reflection was a terrible idea to replicate using this method. Each night it generated over twice it’s own size in replication activity. Given we have a 72 hour replication retention and the size of the database that alone was enough to wipe out the disk space on our server within 3 days.

Reinitialize All Subscriptions was the magic bullet which ran in just a minute allowing the publication to be subsequently instantly deleted. The replicated databases transaction logs were then truncated and shrunk giving us ample breathing space on the storage front.

The distribution database was still huge but heading into Replication Monitor’s Common Jobs and starting the Distribution clean up: distribution job was our final step.

The job might take all day but did not block any other replication activity :)

[)amien

Activity Monitor showed a number of processes on the distribution database used for SQL Server’s replication were blocked by another process which was in turn blocked by a SQL Agent – TSQL Job executing on the distribution database.

This process’s job GUID that did not match any of our jobs and appeared to be an internal job that regularly executes:

EXEC dbo.sp_MSdistribution_cleanup @min_distretention = 0, @max_disretention = 32760

This would sit on the PAGEIOLATCH_SH status for over 10 minutes at a time most most likely because it was operating on an MSrepl_commands table with 3.4 million records and a MSrepl_transactions table of 600k records.

The default max_disretention is 72 (hours) and specifies how long to hold the replication history for. This means if a subscriber is down for longer than this period it will loose transactions.

Quite how this system obtained the value 32760 (3.7 years) is a bit of a mystery and I’m assuming that it actually uses a lower value as there have been more than 3.4 million updates in that period.

Setting this value to 84 hours (3.5 days) in Management Studio’s Replication > Distributor Properties > Transaction Retention soon dropped those large MS tables down to 406 records and 300 respectively and resulted in sp_MSdistribution_cleanup executing in a couple of seconds.

It’s a very specific scenario but perhaps this blog post will be of use to somebody else!

[)amien

Windows Live Writer has finally been released but refuses to install on Windows x64 (64-bit) despite the promises made in the release notes of the previous beta. In fact it has an incredibly confusing installer.

This is especially annoying considering it’s such a great tool and that the new version finally supports a UK spell checker.

[)amien

• Jonathan Carter
• Kevin Huffman
• Aaron Jensen
• Dimitar Kapitanov
• James Kovaks
• Julia Lerman
• Jeffrey Palmero
• Vassil Terziev

And from the Microsoft side of the fence

• Brad Abrams
• Pablo Castro
• Simon Calvert
• Scott Guthrie
• Eilon Lipton
• Lance Olson

Congratulations go out to Rob Conery who is now getting paid by Microsoft to work on SubSonic!

How cool is that!

[)amien