MSBuild – BeatificaBytes

Drop Builds’ primary output in a “Latest” folder

vletroye — Sat, 27 Oct 2012 17:07:10 +0000

As briefly mentioned in another post, we have a specific “Reference Assemblies” folder on the Build Machines containing the latest version of each assembly issued from a successful Build. Assemblies in that folder, know as the “Latest” folder, can be used for “Continuous Integration” Builds. This post is about the pragmatic solution implemented to drop only the primary output of the Builds in that folder.

Click to Read MoreClick to Close

The output of a Build contains not only the assemblies and satellite assemblies issued from the compilation of the Visual Studio Projects (I.e.: assemblies part of the “primary output” as named in the Microsoft Setup Projects). It contains also a copy of all the referenced assemblies (file references) with the property “CopyLocal”=”True”.

It’s important for our purpose to only drop the primary output into the “Latest” folder, otherwise we could override the latest version of some referenced assemblies with a copy of the specific version found for the Build (e.g.: when targeting the Integration Environment, we use the promoted version on, the assemblies which are possibly not the latest).

We may not set “CopyLocal”=”False” on all the “file references” because MSTest needs a copy of those in the bin folder to be able to run the unit Tests (That would not be the case if we could find for MSTest an equivalent of the “ReferencePath” parameter of MSBuild).

We don’t have access to methods (or well described “algorithms”) to retrieve the exact list of assemblies part of the “primary output”. Such methods are only implemented in the Microsoft Setup Projects (Projects not supported, by the way, by MSBuild).

We don’t want all our developers to add MSBuild Scripts in their Visual Studio Projects to drop the “Targets” in the “Latest” folder.

Ex. copy “$(TargetDir)$(TargetName).???” “C:\RefAssemblies\Latest”

This is not only too error prone (like everything you request to a human developer), but it’s also not robust enough. It’s in our opinion impossible to maintain this with enough guarantee taking into account that developers can for example add support for new languages at any time. For each Project, in addition to the assembly $(TargetDir)$(TargetName).dll, we also have to drop the satellite assemblies (e.g.: \.resources.dll”), etc…

The most generic solution we found to identify the assemblies part of the primary output, although quick and dirty, consists in parsing the FileListAbsolute.txt file generated by MSBuild itself and available in the “obj” folder. We “rely” so on direct output of Microsoft (the content of that file) and avoid to implement (and maintain) our own generic (and most probably complex) algorithm (“complex” because I have no idea how to detect satellite assemblies in a Visual Studio Project).

Notice: We may only drop the primary output of Builds in the “Latest” folder for “Continuous Integration” Builds or Builds targeting the Integration Environment (both using the latest version of the sources). Builds targeting Qualification, Pre-Production or Production use always a specific version of the sources instead of the latest version. Their purpose is indeed to fix a bug in the assemblies deployed in the targeted environment and build with older sources.

Click to Read MoreClick to Close

[csharp] using System;
using System.IO;
using System.Linq;
using System.Activities;
using System.Reflection;
using System.Diagnostics;
using System.ComponentModel;
using System.Collections.Generic;
using Microsoft.TeamFoundation.Build.Client;
using Microsoft.Build.Evaluation;

namespace AG.SCRM.TeamBuild.Activity
{
/// <summary>
/// Collect Build’s Primary Output Items
/// </summary>
[BuildActivity(HostEnvironmentOption.All)] public sealed class GetPrimayOutput : CodeActivity
{
/// <summary>
/// List of Build’s Primary Output Items
/// </summary>
[RequiredArgument] public InOutArgument<List<FileInfo>> PrimaryOutputItems { get; set; }

/// <summary>
/// Build’s Configuration Parameter
/// </summary>
[RequiredArgument] public InArgument<string> Configuration { get; set; }

/// <summary>
/// Build’s Platform Parameter
/// </summary>
[RequiredArgument] public InArgument<string> Platform { get; set; }

/// <summary>
/// Local path of Build’s project file
/// </summary>
[RequiredArgument] public InArgument<string> LocalProject { get; set; }

[RequiredArgument] public InArgument<string> OutDir { get; set; }

/// <summary>
/// Collect Build’s Primary Output Items
/// </summary>
/// <param name="context"></param>
protected override void Execute(CodeActivityContext context)
{
var fileListAbsolute = PrimaryOutputItems.Get(context);
var localProject = LocalProject.Get(context);
var configuration = Configuration.Get(context);
var platform = Platform.Get(context);

// Default Configuration is "Debug"
if (string.IsNullOrEmpty(configuration)) configuration = "Debug";

// Initialize the list of Build’s Primary Output Items if required.
// Otherwise, add Build’s Primary Output Items to the provided list
// to possibly support a loop on all Visual Studio Solution files in
// Build’s
if (fileListAbsolute == null) fileListAbsolute = new List<FileInfo>();

if (Path.GetExtension(localProject).Equals(".sln", StringComparison.InvariantCultureIgnoreCase))
{
// Parse the Visual Studio Solution file
SolutionParser sln = new SolutionParser(localProject);
string root = Path.GetDirectoryName(localProject);
foreach (SolutionProject project in sln.MSBuildProjects)
{
localProject = Path.Combine(root, project.RelativePath);
CollectOutputItems(context, fileListAbsolute, configuration, platform, localProject);
}
}
else
{
// Validate that the file is a Visual Studio Project
var projectCollection = new ProjectCollection();
try
{
Project project = projectCollection.LoadProject(localProject);
string projectDirectoryPath = Path.GetDirectoryName(localProject);
CollectOutputItems(context, fileListAbsolute, configuration, platform, localProject);
}
catch (Exception ex)
{
throw new Exception(string.Format("Project file ‘{0}’ is not a valid Visual Studio project.", localProject), ex);
}
projectCollection.UnloadAllProjects();

}

context.SetValue(PrimaryOutputItems, fileListAbsolute);
}

private static void CollectOutputItems(CodeActivityContext context, List<FileInfo> items, string configuration, string platform, string project)
{
string projectFileName = Path.GetFileName(project);
string projectName = Path.GetFileNameWithoutExtension(project);
string projectPath = Path.GetDirectoryName(project);
string fileListAbsolute = GetFileListAbsolute(configuration, platform, projectFileName, projectPath);

if (!string.IsNullOrEmpty(fileListAbsolute))
{
System.IO.StreamReader file = null;
string line;

string parent = Path.GetDirectoryName(fileListAbsolute);

// Read the file and parse it line by line.
using (file = new System.IO.StreamReader(fileListAbsolute))
{
while ((line = file.ReadLine()) != null)
{
// Ignore obj folder’s local items.
if (!line.StartsWith(parent, StringComparison.OrdinalIgnoreCase))
{
FileInfo item = new FileInfo(line);
if (item.Exists)
{
// We are actually only interested in .dll and .exe + their .pdb, .xml and .config
if (item.Extension.Equals(".dll", StringComparison.OrdinalIgnoreCase) || item.Extension.Equals(".exe", StringComparison.OrdinalIgnoreCase))
{
//MessageHelper.DisplayInformation(context, string.Format("Build output of {0} contains: {1}.", projectName, item.Name));
items.Add(item);

var pdb = new FileInfo(Path.Combine(item.DirectoryName, Path.GetFileNameWithoutExtension(item.FullName) + ".pdb"));
if (pdb.Exists)
{
items.Add(item);
}

/// <summary>
///
/// </summary>
/// <param name="configuration">Build’s Configuration: Debug or Release</param>
/// <param name="platform">Build’s Target Platform: Any CPU or x86</param>
/// <param name="projectFileName">Visual Studio Project filename</param>
/// <param name="projectPath">Visual StudioProject path</param>
/// <returns>The path of the FileListAbsolute.txt file.</returns>
/// <remarks>This file should be located under /obj/{platform}/{configuration/}.
/// But we also check the parent folders if the file is not found where expected.
/// We did indeed experienced problem to locate this file for Builds with "Mixed Plateform" Target.</remarks>
private static string GetFileListAbsolute(string configuration, string platform, string projectFileName, string projectPath)
{
var fileListAbsoluteName = projectFileName + ".FileListAbsolute.txt";
var fileListAbsolutePath = Path.Combine(projectPath, "obj", platform, configuration, fileListAbsoluteName);
if (!File.Exists(fileListAbsolutePath))
{
fileListAbsolutePath = Path.Combine(projectPath, "obj", configuration, fileListAbsoluteName);
if (!File.Exists(fileListAbsolutePath))
{
fileListAbsolutePath = Path.Combine(projectPath, "obj", fileListAbsoluteName);
if (!File.Exists(fileListAbsolutePath))
{
fileListAbsolutePath = null;
}
}
}

return fileListAbsolutePath;
}
}

/// <summary>
/// This Visual Studio Solution file Parser can be used to retrieve lists of projects in a Solution.
/// The first list contains all projects that can be built with MSBuild.
/// The second list contains all the other projects.
/// </summary>
/// <remarks>
/// Based on http://stackoverflow.com/questions/707107/library-for-parsing-visual-studio-solution-files.
/// It’s a wrapper on Microsoft Build’s internal class "SolutionParser"
/// </remarks>
public class SolutionParser
{
static readonly Type s_SolutionParser;
static readonly PropertyInfo s_SolutionParser_solutionReader;
static readonly MethodInfo s_SolutionParser_parseSolution;
static readonly PropertyInfo s_SolutionParser_projects;

public List<SolutionProject> MSBuildProjects { get; private set; }
public List<SolutionProject> OtherProjects { get; private set; }

static SolutionParser()
{
s_SolutionParser = Type.GetType("Microsoft.Build.Construction.SolutionParser, Microsoft.Build, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a", false, false);
if (s_SolutionParser != null)
{
s_SolutionParser_solutionReader = s_SolutionParser.GetProperty("SolutionReader", BindingFlags.NonPublic | BindingFlags.Instance);
s_SolutionParser_projects = s_SolutionParser.GetProperty("Projects", BindingFlags.NonPublic | BindingFlags.Instance);
s_SolutionParser_parseSolution = s_SolutionParser.GetMethod("ParseSolution", BindingFlags.NonPublic | BindingFlags.Instance);
}
}

public SolutionParser(string solutionFileName)
{
if (s_SolutionParser == null)
{
throw new InvalidOperationException("Cannot find type ‘Microsoft.Build.Construction.SolutionParser’ are you missing a assembly reference to ‘Microsoft.Build.dll’?");
}
var solutionParser = s_SolutionParser.GetConstructors(BindingFlags.Instance | BindingFlags.NonPublic).First().Invoke(null);
using (var streamReader = new StreamReader(solutionFileName))
{
s_SolutionParser_solutionReader.SetValue(solutionParser, streamReader, null);
s_SolutionParser_parseSolution.Invoke(solutionParser, null);
}
MSBuildProjects = new List<SolutionProject>();
OtherProjects = new List<SolutionProject>();
var array = (Array)s_SolutionParser_projects.GetValue(solutionParser, null);
for (int i = 0; i < array.Length; i++)
{
SolutionProject project = new SolutionProject(array.GetValue(i));

if (project.ProjectType == "KnownToBeMSBuildFormat")
MSBuildProjects.Add(project);
else
OtherProjects.Add(project);
}
}
}

/// <summary>
/// This class represent a Visual Studio Project member of a Solution.
/// </summary>
/// <remarks>
/// It’s a wrapper on Microsoft Build’s internal class "ProjectInSolution"
/// </remarks>
[DebuggerDisplay("{ProjectName}, {RelativePath}, {ProjectGuid}, {ProjectType}")] public class SolutionProject
{
static readonly Type s_ProjectInSolution;
static readonly PropertyInfo s_ProjectInSolution_ProjectName;
static readonly PropertyInfo s_ProjectInSolution_RelativePath;
static readonly PropertyInfo s_ProjectInSolution_ProjectGuid;
static readonly PropertyInfo s_ProjectInSolution_ProjectType;

static SolutionProject()
{
s_ProjectInSolution = Type.GetType("Microsoft.Build.Construction.ProjectInSolution, Microsoft.Build, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a", false, false);
if (s_ProjectInSolution != null)
{
s_ProjectInSolution_ProjectName = s_ProjectInSolution.GetProperty("ProjectName", BindingFlags.NonPublic | BindingFlags.Instance);
s_ProjectInSolution_RelativePath = s_ProjectInSolution.GetProperty("RelativePath", BindingFlags.NonPublic | BindingFlags.Instance);
s_ProjectInSolution_ProjectGuid = s_ProjectInSolution.GetProperty("ProjectGuid", BindingFlags.NonPublic | BindingFlags.Instance);
s_ProjectInSolution_ProjectType = s_ProjectInSolution.GetProperty("ProjectType", BindingFlags.NonPublic | BindingFlags.Instance);
}
}

public string ProjectName { get; private set; }
public string RelativePath { get; private set; }
public string ProjectGuid { get; private set; }
public string ProjectType { get; private set; }

public SolutionProject(object solutionProject)
{
this.ProjectName = s_ProjectInSolution_ProjectName.GetValue(solutionProject, null) as string;
this.RelativePath = s_ProjectInSolution_RelativePath.GetValue(solutionProject, null) as string;
this.ProjectGuid = s_ProjectInSolution_ProjectGuid.GetValue(solutionProject, null) as string;
this.ProjectType = s_ProjectInSolution_ProjectType.GetValue(solutionProject, null).ToString();
}
}

/// <summary>
/// List of known project type Guids from http://www.mztools.com/articles/2008/mz2008017.aspx
/// + BizTalk: http://winterdom.com/2008/12/biztalkserver2009msbuildtasks
/// + Workflow 4.0
/// </summary>
public enum ProjectType
{
[Description("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}")] Windows_CSharp,
[Description("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}")] Windows_VBNET,
[Description("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}")] Windows_VisualCpp,
[Description("{349C5851-65DF-11DA-9384-00065B846F21}")] Web_Application,
[Description("{E24C65DC-7377-472B-9ABA-BC803B73C61A}")] Web_Site,
[Description("{F135691A-BF7E-435D-8960-F99683D2D49C}")] Distributed_System,
[Description("{3D9AD99F-2412-4246-B90B-4EAA41C64699}")] Windows_Communication_Foundation_WCF,
[Description("{60DC8134-EBA5-43B8-BCC9-BB4BC16C2548}")] Windows_Presentation_Foundation_WPF,
[Description("{C252FEB5-A946-4202-B1D4-9916A0590387}")] Visual_Database_Tools,
[Description("{A9ACE9BB-CECE-4E62-9AA4-C7E7C5BD2124}")] Database,
[Description("{4F174C21-8C12-11D0-8340-0000F80270F8}")] Database_other_project_types,
[Description("{3AC096D0-A1C2-E12C-1390-A8335801FDAB}")] Test,
[Description("{20D4826A-C6FA-45DB-90F4-C717570B9F32}")] Legacy_2003_Smart_Device_CSharp,
[Description("{CB4CE8C6-1BDB-4DC7-A4D3-65A1999772F8}")] Legacy_2003_Smart_Device_VBNET,
[Description("{4D628B5B-2FBC-4AA6-8C16-197242AEB884}")] Smart_Device_CSharp,
[Description("{68B1623D-7FB9-47D8-8664-7ECEA3297D4F}")] Smart_Device_VBNET,
[Description("{14822709-B5A1-4724-98CA-57A101D1B079}")] Workflow_30_CSharp,
[Description("{D59BE175-2ED0-4C54-BE3D-CDAA9F3214C8}")] Workflow_30_VBNET,
[Description("{06A35CCD-C46D-44D5-987B-CF40FF872267}")] Deployment_Merge_Module,
[Description("{3EA9E505-35AC-4774-B492-AD1749C4943A}")] Deployment_Cab,
[Description("{978C614F-708E-4E1A-B201-565925725DBA}")] Deployment_Setup,
[Description("{AB322303-2255-48EF-A496-5904EB18DA55}")] Deployment_Smart_Device_Cab,
[Description("{A860303F-1F3F-4691-B57E-529FC101A107}")] Visual_Studio_Tools_for_Applications_VSTA,
[Description("{BAA0C2D2-18E2-41B9-852F-F413020CAA33}")] Visual_Studio_Tools_for_Office_VSTO,
[Description("{F8810EC1-6754-47FC-A15F-DFABD2E3FA90}")] SharePoint_Workflow,
[Description("{6D335F3A-9D43-41b4-9D22-F6F17C4BE596}")] XNA_Windows,
[Description("{2DF5C3F4-5A5F-47a9-8E94-23B4456F55E2}")] XNA_XBox,
[Description("{D399B71A-8929-442a-A9AC-8BEC78BB2433}")] XNA_Zune,
[Description("{EC05E597-79D4-47f3-ADA0-324C4F7C7484}")] SharePoint_VBNET,
[Description("{593B0543-81F6-4436-BA1E-4747859CAAE2}")] SharePoint_CSharp,
[Description("{A1591282-1198-4647-A2B1-27E5FF5F6F3B}")] Silverlight,
[Description("EF7E3281-CD33-11D4-8326-00C04FA0CE8D")] BizTalk,
[Description("32f31d43-81cc-4c15-9de6-3fc5453562b6")] Workflow_40
};

/// <summary>
/// Helper Class to manage Visual Studio Project’s types
/// </summary>
public static class ProjectTypeExtensions
{
public static Guid ToGuid(this ProjectType val)
{
DescriptionAttribute[] attributes = (DescriptionAttribute[])val.GetType().GetField(val.ToString()).GetCustomAttributes(typeof(DescriptionAttribute), false);
return attributes.Length > 0 ? Guid.Parse(attributes[0].Description) : Guid.Empty;
}

public static ProjectType Parse(string val)
{
return Parse(Guid.Parse(val));
}

public static ProjectType Parse(Guid val)
{
ProjectType? type = null;
FieldInfo[] fis = typeof(ProjectType).GetFields();
foreach (FieldInfo fi in fis)
{
DescriptionAttribute[] attributes = (DescriptionAttribute[])fi.GetCustomAttributes(typeof(DescriptionAttribute), false);
if (attributes.Length > 0)
{
if (Guid.Parse(attributes[0].Description) == val)
{
if (Enum.IsDefined(typeof(ProjectType), fi.Name))
type = (ProjectType)Enum.Parse(typeof(ProjectType), fi.Name);
break;
}
}
}
if (type.HasValue)
return type.Value;
else
throw new FormatException(string.Format("'{0}’ is not a valid Project Type’s Guid", val.ToString()));
}

public static List<ProjectType> GetMSBuildProjectTypes(string localProject)
{
var projectCollection = new ProjectCollection();
Project project;
try
{
project = projectCollection.LoadProject(localProject);
}
catch (Exception ex)
{
throw new Exception(string.Format("Project Type cannot be determined as ‘{0}’ is not a valid VS project.", localProject), ex);
}

var projectTypes = GetMSBuildProjectTypes(project);

projectCollection.UnloadAllProjects();

return projectTypes;
}

public static List<ProjectType> GetMSBuildProjectTypes(Project project)
{
try
{
var projectTypeGuids = (from property in project.Properties
where property.Name == "ProjectTypeGuids"
select property.EvaluatedValue).FirstOrDefault();

List<ProjectType> projectTypes;
if (string.IsNullOrEmpty(projectTypeGuids))
projectTypes = new List<ProjectType>();
else
projectTypes = (from guid in projectTypeGuids.Split(‘;’) select Parse(guid)).ToList();

return projectTypes;
}
catch (Exception ex)
{
throw new Exception(string.Format("Unable to determine the project type of ‘{0}’ due to: {1}", Path.GetFileNameWithoutExtension(project.FullPath), ex.Message));
}
}
}
}
[/csharp]

Target multiple environments with only one TFS Build Server

vletroye — Sun, 21 Oct 2012 18:05:24 +0000

I had customize our Build Process Template to support multiple target environments on a single Build Machine. I.e.: to resolve the “file references” at Build time depending on the environment targeted for the deployment.

Click to Read MoreClick to Close

At work, we have mainly four distinct environments: Integration, Qualification, Pre-Production and Production. Each application (and its referenced assemblies) are promoted from one environment to the next one via tasks (Release Distribution jobs) scheduled according to a predefined monthly Release Plan…

TFS Build Machines not only compile the applications to be deployed via the standard Release Distribution, but also the urgent fixes to be deployed directly in Pre-Production without passing through Integration and Qualification environments. It also happens, although really quite seldom, that a Build skips the Integration and goes directly to Qualification.

It’s also to be noticed that none of our applications (assemblies) are strong named and deployed in the GAC, neither on development workstations nor on servers. Instead, all referenced assemblies are always copied within the application’s bin folders, also on the servers.

Therefore, basically, we would need Build Machines dedicated for each target environment with the adequate assemblies made available on them, for reference resolution purpose. That would be a pity to have such Build Machines (and the related setup, maintenance, backup costs, …) for at most one build per month (Fix in Production and Qualification are fortunately not common).

To avoid that, I did customize our Build Process Template to take a Reference Path as input and to pass it to MSBuild. Actually, when editing a Build Definition, the Builder can select the target Environment, and the Build will simply receive a path to the location containing the related assemblies.

Ex.: MSBuild mySolution.sln /p:ReferencePath=”c:\RefAssemblies\Qualification\”

How can I be sure that this Reference Path won’t interfere with any Hint Paths defined in the Visual Studio projects?

First, note that the location provided to MSBuild via the “ReferencePath” parameter will be probed to resolve all “file references” before any other location on the standard probing path. But we also pay attention to not make the assemblies available on the Hint Path on the Build Machines:

On Development Workstations, all our assemblies are made available in a single “Reference Assemblies” folder. Developers add references on assemblies in there for development and local testing purpose. They can also start a task at will to update this “Reference Assemblies” folder with the latest version of the assemblies (the versions used in the Integration Environment) or with the version currently deployed in the Qualification or Production Environments (e.g.: for bug fixing purpose).

On the Build Machines, there is one “Reference Assemblies” folder per environment (i.e.: updated by the Release Distribution with the assemblies currently in use in that environment). None of those folders is located at the same path as the “Reference Assemblies” folder of the Development Workstations. As a result, MSBuild cannot use the Hint Paths found in the Visual Studio projects to locate the referenced assemblies. Instead, it uses the path of the “Reference Assemblies” folder passed to it via its parameter /p:ReferencePath.

In addition to the “Reference Assemblies” folders per environment, we also have one extra folder containing the output of each latest successful Build. This one is used for “Continuous Integration“. No need to update the references in the Visual Studio Projects, MSBuild always find in that folder the latest version of each assembly recently built on any Build Machine (if requested to do so via the Build Definition). This is by the way the default “target environment” for all “Rolling Builds” defined on “Development Branches”, so any breaking features in a new version of a referenced assembly is immediately detected. Builds “candidate” to be promoted use by default the Reference Path with the assemblies from the Integration environment.