Just got out of a presentation by Brian in downtown Chicago. Basically, Brian talked through how Microsoft does software development in the Developer Division from the perspective of using TFS 2010. I think the major point in the presentation was driving at the value that TFS 2010 added to Microsoft’s development effort – given that they are an extremely large commercial software development shop. Overall, it was a great presentation about the value the tool brings + with customizations. Out of the box Microsoft includes an Agile process template and a CMMI one. The metrics and reports that Brian talked through were primarily – it seemed out of Microsoft’s custom process template that they use internally. I think this talk is a great example of what-is-possible with Team Foundation Server – if not somewhat about what the tools do for you out of the box. As a consultant that focuses on the development process in teams, this was FASCINATING. The following consists of a few high points in his talk.
How Microsoft Developer Division [DevDiv] Does Planning
Traceability – Brian talked a great deal about how the process has improved at Microsoft’s DevDiv since the releases of 2005/2008. They have a hierarchical set of requirements traceability from core principles down to deliverables. The high level core principles were called “Pillars.” With 2010 – the core principles were Quality in Development and Testing. So you can imagine many of the features that we’ve heard about – lots of them roll up into those two high level things. While discussing “Requirements Management” – Brian really talks about three key parts of requirements management process:
Elicitation – there is not much support within Microsoft TFS to handle that. The teams use 3rd Party or other MS applications in this phase still.
Analysis/Definition – the act of translating what was elicited into a solution – of what the applications could be. Some synthesis and analysis happens here to transform problems into solutions for sign off/buy-in by business stake-holders.
Management – understanding traceability and completeness of the development efforts as compared to the requirements that were documented.
Brian covered a lot of the product realignment and SKU changes – many of which are have been talked about a great deal in Microsoft’s published documentation.
Peanut Butter – In talking about the context of how the process has evolved at Microsoft over the past 10 years, he described an effect where by Microsoft would say – ‘let’s release 2003 or 2005’ and the various distributed teams in the division would go back to their teams and decide what they could get in there, and as a result – the release of a product would consist of a lot of incremental improvements, but there was no real key pillar of why someone should go out and buy the toolset. I think this has something to do with the way that Microsoft as they’ve grown have evolved to include vision as a key component of the planning. Many enterprise teams today struggle here as well – and determine what is a success criteria for why a project succeeds or fails. With 2010 – they clearly took a big look back if you look at all the improvements and understand what are going to be a part of the key business value improvements of the product. Peanut Butter is a great term to describe this. i have been involved on a few larger teams where synthesizing product releases took an hour to boil down. It wasn’t because of distributed walls, but it was due to a lack of message from the beginning in the release.
No Repro Bugs – In talking through the development process there, Brian took a case study of how they improved testing and development quality – by wanting to eliminate “No Repro” bugs. Those are the bugs filed by testers that the developer just can’t reproduce. The developer says so and it just goes away – time wasted. Brian estimated 30% time-waste due to this factor within the Microsoft Dev Div team. Some of the key features around removing this situation that Brian talked about:
- Video Recording – the tester as they are using Testing Professional [a tool designed explicitly for testers that is not based on Visual Studio] – a video is automatically attached to the newly filed Defect which gets sent to the developer.
- Automatic System Information – Also as defects are automatically filed, the environment [CPU, RAM, OS, Browser] information are all automatically attached to the work item when this happens.
- Intellitrace – While the tester is running through an application – the stack / heap is being recorded, and automatically attached to the bug – the way that Brian described was that this was better than an eventual stack dump/crash dump because it is dynamic and you can open up the intellitrace in an interactive debugger, essentially playing back the entire session of a testing situation to understand the cause that variables are some value or another. This is like being able to run the application within debugger in Visual Studio – all the time – without actually installing VS on testing environments. This one feature has definitely been talked about a lot – i was impressed how Brian talked about this one. I really question any one that says that Microsoft does not innovate – because this really seems an amazing feature that will help countless developers.
- Symbols Server – When a build is deployed and developers can step through the code execution after the fact, the source code that was used for that build can be used to step through. You don’t have a mismatch between the code a developer has on their local machine and the build that was tested on. Someone asked if there was a feature to automatically compare that set of code ‘with latest’ – and there wasn’t at this time, but maybe in another release.
Visual Studio Fun Fact #1 – there are 65 Million Lines of Code in VS 2010 today.
Layer Diagram – Apparently, we learned today there was some code in 2005 (TFS) on the server-side [that’s the part that runs in IIS as ASP.NET web services] – that was referring to the namespace System.Windows.Forms [yikes!]. The Layer Diagram could have prevented this by having a team establish limitations on the code and validating that the code written would conform. I did ask Brian if the BCL (base class library) classes could be used in this diagram – the answer is ‘yes’ there. I think most demos you see constraints on the application code, so it was nice to see you can enforce architecture for specific BCL namespaces and assemblies.
Multiple Process Methodologies – I think one of the things that I really liked was the fact – and given Microsoft is so big – that just as we as consultants don’t say that one process works for everyone, Microsoft released that this was true – even within the same organization. I think they mostly have two types of features/deliverables. Some are called ‘Dark-Tunnel-Features’ and others are called ‘Value-Added-Features.’ Dark Tunnel are those features that once you start, you can’t really stop. They represented things that you can’t have ‘shippable iterations’ of. An example of this in VS 2010 was the editor – and it makes sense, Microsoft couldn’t release a half-editor – it’d have to be committed to, or it wasn’t. For those things – they mostly run those waterfall-esque (Microsoft Project, Phases, etc), for other features, Brian said the teams mostly developed with some Scrum Variant [not the same for all the teams, but each team did things probably a little different in each team].
What Was Not Good About 2010
So – I did ask what could be improved today within DevDiv – and Brian definitely had an answer. Performance, Performance, Performance. Here’s the big irony [and Brian also has said this in his blog], that this release of 2010 had more performance focus throughout the entire process than any other release of Visual Studio in history. Yet when the Beta 2 came out, it was universally panned by the community for having many performance problems. One problem that was attributing to this was the fact that the team, when developing performance tests, they geared towards making them repeatable, and in an attempt to remove the variations that can happen throughout running the application like [network, disk] and things that you can’t really control. So in effect the team wrote their performance tests at the ‘unit’ level – small repeatable tests such as (i’m over simplifying):
- Boot up Computer
- Open Visual Studio 2010
- Open Text File [5 lines of code].
- Compile
- Save
Those tests were reporting throughout the entire development lifecycle that they were on track. So – I think – this was akin to ‘writing the test to succeed’ – that can happen in functional unit testing if you’re not careful, or what arguably happens when you’re not writing unit tests in an automated fashion [developers just clicking through the happy path when testing]. Microsoft was planning on reviewing how they do performance testing for future versions of Visual Studio and TFS.
