CategoryContinuous Delivery

Running gulp tasks on a buildserver

With the newest ASP.NET release coming, Microsoft is removing their own optimization framework and pushes developers to use Gulp, NPM and Bower.
I do not want to manually minify and bundle my css and js files, so I want a Gulp task to do it.
My NPM file (package.json) looks like:

{
  "version": "1.0.0",
  "name": "ASP.NET",
  "private": true,
  "devDependencies": {
    "bower": "1.7.7",
    "gulp": "3.9.1",
    ....
  }
}

My bower file (bower.json) looks like

{
  "name": "ASP.NET",
  "private": true,
  "dependencies": {
    "jquery": "2.2.3",
    "jquery-validation-unobtrusive": "3.2.6",
    "bootstrap": "3.3.6",
    ....
  }
}

I also do not want my bundles to be source controlled.
It is a task of the buildserver to prepare my solution for release.
This means that the buildserver should be able to run the same Gulp tasks as we do in our development environment.

The following software should be installed on the buildserver to let the buildserver run Gulp tasks:

When installing Git, set install option to “Run GIT from the Windows Command Prompt”.
I’d like to have all my configuration source controlled, so I create a targets file which contains the targets for running Npm, Gulp and Bower and I import this file in my web project.

  <!-- Gulp -->
  <Target Name="RunGulpTask" AfterTargets="BowerInstall" Condition="'$(RunGulpTasks)' != ''">
    <Message Text="Running gulp task 'default'" Importance="high" />
    <Exec Command="node_modules\.bin\gulp" WorkingDirectory="$(ProjectDir)" />
  </Target>

  <!--Bower -->
  <Target Name="BowerInstall" AfterTargets="NpmInstall" Condition="'$(RunGulpTasks)' != ''">
    <Message Text="Running bower install" Importance="high"/>
    <Exec Command="node_modules\.bin\bower install" WorkingDirectory="$(ProjectDir)" />
  </Target>

  <!--Npm -->
  <Target Name="NpmInstall" BeforeTargets="BeforeBuild" Condition="'$(RunGulpTasks)' != ''">
    <Message Text="Running npm install" Importance="high"/>
    <Exec Command="npm install" WorkingDirectory="$(ProjectDir)" />
  </Target>

I do not want these tasks to run in my development environment (because the TaskRunner of Visual Studio takes care of it), so I added a RunGulpTask parameter. When this is provided (by adding /p:RunGulpTask=true to the msbuild command), the targets will be run before the solution is built.

My gulpfile.js looks like:

var gulp = require('gulp'),
.....
gulp.task('default', ['bundleCss', 'minCss', 'bundleJs', 'minJs'], function() {});

I did not provide a Gulp task to run, so Gulp will run the default task by convention. My default task has a dependency to all tasks I want to be run on the buildserver.
The buildserver now bundles and minifies my css and js files by using the same Gulp tasks.

TFS: Discard changesets when merging to branches

When changes are branch specific and should not be merged (back) to other branches, these changes should be discarded.

The following TFS command will discard changsets:

tf merge $/Project/SourceBranch $/Project/TargetBranch /discard /recursive /version:C10000~C10000

This example command discards pending merge changesets from SourceBranch to TargetBranch.
It discards changeset 10000. The version is a from ~ to, so you can discard multiple changesets at once.

When the command has finished, you still need to check in the merge.

Setting up Git source control on a QNAP NAS

When I start a new project, the first thing I do is set up source control. Source control is key!
I know all my source code is safe and when I make mistakes I can easily do a rollback.

So why not take it serious and use Git?
GitHub is great, but it is not free when using private repositories. Tho I’m a big fan of open source, not all my projects are open source.

So I wanted to configure Git on my NAS, an old QNAP TS-410 (currently running on firmware 4.2.0).
This is how I configured Git for a QNAP NAS.

Install Git

First of all, install the QNAP package from the app center (currently version 2.1.0) and make sure it is turned on.

There seems to be something wrong with the QNAP Git package, because a manual action is required.
open a ssh connection to your NAS.

If you’re not familiar with ssh, you can download a client (f.e. putty) and open a new connection by entering the IP of your NAS.

now login with your admin account and enter the following command:


#  cd /usr/bin
#  ln -s /Apps/git/bin/git-upload-pack
#  ln -s /Apps/git/bin/git-receive-pack

This fixes an issue with the git-upload-pack and git-receive-pack not being found.

Hosting your repositories

Next, create a new share for your repositories.
I created a new share named ‘git’, but you’re free to choose.

Again, open an ssh connection and go to the just created share:

#  cd /share/MD0_DATA/git

if this does not work, the MD0_DATA folder is probably different. go to the /share folder and check the folder name with the following command:

#  ls -la

This will show a full list of all items and you can figure out what the right name is.

in the ‘git’ folder, enter the following command to create a new repository:

git init --bare NameOfMyRepository

This creates a new repository with the name ‘NameOfMyRepository’. It will automatically create a new subfolder with an identical name.

Cloning the repository

On your development machine, open your git tool and go to the directory where you want to work.
Now enter the following command:

git clone admin@YourIP:/share/git/NameOfMyRepository

This will ask for the admin’s password.
You can also use auto login by generating a ssh keyfile, but I do not want that for security reasons.

Once entered, the repository is cloned in a folder named ‘NameOfMyRepository’ and you’re good to go!

The Git controls integrated in VS2015 do not work with ssh yet, but the guys are working on it. You can read about it here

Powershell CmdLets

Powershell is a powerful language and can be used in several situations.
One of these situations is the deployment process (continuous delivery). It is also integrated in several systems, f.e. NuGet. NuGet uses powershell for post package installation processing.

I use powershell for the following cases (in continuous delivery):

  •  Replace/rename config files
  • Replace variables in config files
  • Call/Post to webservices
  • Run SQL commands

Note: External Modules need to be imported for calling webservices and running SQL commands. This can be done by calling the Import-Module CmdLet.

Powershell can be used as scripting language, but you can also create CmdLets. These commands can be invoked from the command line in the powershell environment.

CmdLet

A CmdLet is a command which can be called from the powershell command line.

Cmdlets are created by inheriting your class from Cmdlet.
This is available in the System.Management.Automation namespace.

CmdLets use the Verb-Noun naming convention. The Verb and Noun are provided as argument in the CmdLet attribute which decorates your CmdLet class.
The VerbsCommunications class comes with the following fields:

  • Connect
  • Disconnect
  • Read
  • Receive
  • Send
  • Write

You are not restricted to these Verbs, you can also use custom Verbs.

The ProcessRecord method is called. The method is called for every item in the pipeline.

[Cmdlet(VerbsCommunications.Send, "Greeting")]
public class SendGreetingCommand : Cmdlet
{
	[Parameter(Mandatory=true)]
	public string Name { get; set; }

	protected override void ProcessRecord() 
	{
	      WriteObject("Hello " + name + "!");
	}
}

This CmdLet example can be called from powershell command line as follows:

Send-Greeting –Name "Vincent" // Outputs: Hello Vincent!

Command line arguments are automatically bound to properties which are decorated with the Parameter attribute. Parameters are mandatory when the Mandatory argument of Parameters are set to true.

Maybe it’s interesting to know that the package manager console in Visual Studio also uses powershell and commands like Add-Migration and Update-Database are all CmdLets.

So if you haven’t given powershell a try, you really should!

Add msbuild parameters for NuGet to the TFS build process template

NuGet packages can be a great artifact of a build and it can easily be integrated in your build process.
Make sure the NuGet targets file is imported in the projects which should generate a NuGet package. Please read my previous post about NuGet to see how you can integrate NuGet in your solution.
By default, NuGet registers itself in a project file as follows:

<Import Project="$(SolutionDir)\.nuget\NuGet.targets" Condition="Exists('$(SolutionDir)\.nuget\NuGet.targets')" />

The targets file contain a BuildPackage target which can be triggered by setting an msbuild parameter named BuildPackage to true.
Now we need to create a property in the Build  template which sets the BuildPackage parameter in the msbuild command in the build template.

Open Visual Studio and open the Team Explorer window. Click on Builds to display your TFS Builds. When you right click on a Build, you can select “Edit Build definitions”. This will bring up the edit screen for the selected Build.
Now click on the “Process” tab. This will display which Build template is associated with the Build and what the parameters are.

Click the “show details” icon in the top right corner to see the template.
The template is a Windows WorkFlow Foundation (WWF) template.  The xaml file is an xml based file which creates workflows in applications.
Edit the .xaml build process template in an xml editor, I prefer notepad++.
Find <x:Members> and add the following (do not remove any existing child elements):


<x:Members>
   ....
   <x:Property Name="NuGet_BuildPackage" Type="InArgument(x:Boolean)" />
</x:Members>

Now that we’ve added the properties, we can add them to a section.
Find the <this:Process.Metadata> element and add the following items (and again, do not delete any existing child elements):


<this:Process.Metadata>
    .....
    <mtbw:ProcessParameterMetadataCollection>
    <mtbw:ProcessParameterMetadata BrowsableWhen="Always" Category="#400 NuGet" Description="Set this to true to create NuGet packages" DisplayName="NuGet : Build Package" ParameterName="NuGet_BuildPackage" />
    </mtbw:ProcessParameterMetadataCollection>
</this:Process.Metadata>

This will create a section called “NuGet” and add the parameters to this section. The #400 sets the priority of the section, it will be positioned in fourth place (just beneath the default sections).

build-process-parameters

Now that we can set the parameters in the build process, we need to provide them as build arguments to msbuild.
Switch from the xml editor to the Workflow editor and find “Run MsBuild for project“.

Edit the command arguments in the properties window and add the new build property to the CommandLineArguments property:


String.Format("/p:SkipInvalidConfigurations=true {0} /p:BuildPackage={1}", MSBuildArguments, NuGet_BuildPackage)

commandline-arguments

You can now enable or disable NuGet package Generation from the build process window of your build. Creating NuGet packages is triggered for every project which has the NuGet targets file imported.

When you want to push the NuGet packages to the NuGet repository, take a look at NuGets push command.  This can also be implemented by an msbuild task.

Continuous delivery, NuGet package generation and integration with TFS build server

Continuous Integration is getting more important and more popular and that is a good thing!
A couple of days ago, I was automating the NuGet package generation process. NuGet packages can be used for deployments to environments.

NuGet integrates in your solution when package restore is enabled.

enable-package-restore

This adds the following files to the .nuget folder in your solution:

  • NuGet.exe
  • NuGet.targets
  • NuGet.config

It also adds an import command for the NuGet.targets file to the project file:

<Import Project="$(SolutionDir)\.nuget\NuGet.targets" Condition="Exists('$(SolutionDir)\.nuget\NuGet.targets')" />

One target of the targets file is BuildPackage. This target can be triggered by running the following msbuild command from the command line:


msbuild mysolution.sln /t:Build /p:BuildPackage=true

This will create a NuGet package for all projects which have imported the NuGet.targets file.
The following files will automatically be added to the NuGet package:

  • The project assembly
  • Files with build action set to content

For simple projects, this behavior is fine. The NuGet package holds the exact content of the project.
In my case, this was not really what I wanted. I wanted one NuGet package for multiple projects and the content files had to be placed in a different structure and location.

That’s when a nuspec file comes to the rescue!

The NuSpec file is the manifest file for NuGet packages. This file holds all information required for the NuGet packages, for example;

  • Package meta data (name, author, copyright, etc.)
  • Dependencies
  • Included Files

For a complete reference of the nuspec file, see http://docs.nuget.org/docs/reference/nuspec-reference.

I added a reference to the assembly1 and assembly2 project and added a nuspec file to the root of the project.
In the nspec file, I added the required assemblies and content files. The nuspec files section looked something like:

  <files>
    <file src="bin\assembly1.dll" target="lib\net40" />
    <file src="bin\assembly2.dll" target="lib\net40" />
    <file src="..\assembly1\Scripts\**\*.*" target="Content\Scripts" />
    <file src="..\assembly2\Scripts\**\*.*" target="Content\Scripts" />
  </files>

I ran the msbuild command on my dev environment and a package was created successfully and contained all files.
It also contains the dll of the NuGet project itself. This is not what I wanted.
I created a naming convention for all NuGet projects, the projects end with NuGet. This gives me the option to exclude the *nuget.dlls from the packages. This can be added to the build command I will describe later in this post.

Files added to the Content folder of a NuGet package are installed on the root of the target the project (the project which the NuGet package is installed to).
Assemblies added to the lib folder will be referenced in the target project. A subfolder in the lib folder can be used for providing different assemblies for a framework (.Net 3.5, .Net 4, etc).

I configured the TFS build server to run the same command and started a build.
The build failed, because NuGet could not find some of the referenced files in the files section of the nuspec file.
When I used RDP to login to the build server, I noticed that the output folder (binaries folder as it is called in the TFS build workflow template) is changed by TFS. It does not use the bin folder of every project, but it changes it to a different folder.
That is why bin\assembly1.dll and bin\assembly2.dll could not be found.

Creating packages should work on both dev environment and on the build server, so referencing the absolute output path on the build server is not an option.
The output folder is stored in the OutDir parameter. This parameter needs to be available in the nuspec file. This can be achieved by adding it to the properties list of the build command.
The build command is stored in the targets file, add “OutDir=$(OutDir.Trim(‘\\’))\\” to the property parameter of the build command. You can also exclude files by adding an Exclude parameter. This is what I have done to exclude the *Nuget.dll files.
The updated BuildCommand is as follows:

<BuildCommand>$(NuGetCommand) pack "$(ProjectPath)" -Properties "Configuration=$(Configuration);Platform=$(Platform);OutDir=$(OutDir.Trim('\\'))\\" $(NonInteractiveSwitch) -OutputDirectory "$(PackageOutputDir)" -symbols -Exclude **\*NuGet.dll</BuildCommand>

It is now possible to use the OutDir variable in the nuspec file:

  <files>
    <file src="$OutDir$assembly1.dll" target="lib\net40" />
    <file src="$OutDir$assembly2.dll" target="lib\net40" />
    <file src="..\assembly1\Scripts\**\*.*" target="Content\Scripts" />
    <file src="..\assembly2\Scripts\**\*.*" target="Content\Scripts" />
  </files>

This solves the output issue on the build server and the build still works on the dev environment.

Setting a version for an assembly with MSBuild

When you have multiple customers and common libraries, it is very important that you can track down the assemblies. You want to be able to tell at all time what the assemblies exactly contain.
Setting versions to an assembly helps a lot with that.
The version of an assembly is stored in the assembly.info file. You can find this file under the Properties folder in your project.
You can also set some meta data of the assembly in this file (like company name, copyright, etc).
The assembly version is stored in the attribute AssemblyVersion

[assembly: AssemblyVersion("1.0.0.0")]

For me, the  ideal situation of the version number should be as follows:

{major version}.{minor version}.{build number}.{revision}

The major and minor versions are defined by the user.
The build number is defined by the build server, a number which is incremented every time a build has run.
The revision is defined by source control;  this is the revision number of the latest commit in this build.

This makes it possible to determine when this build is created and what the source of the assemblies is.

I created a msbuild task for this, which sets the version number. I have used MsBuild Community Tasks for this. This has an AssemblyInfo Task which makes it very easy to set an assembly version:


<Import Project="tasks\MSBuild.Community.Tasks.Targets"/>

<PropertyGroup>
 <AssemblyInfoFile>.\Properties\AssemblyInfo.cs</AssemblyInfoFile>
 <AssemblyMajorVersion>1</AssemblyMajorVersion>
 <AssemblyMinorVersion>0</AssemblyMinorVersion>
 <AssemblyVersion>$(AssemblyMajorVersion).$(AssemblyMinorVersion).$(BUILD_NUMBER).$(SVN_REVISION)</AssemblyVersion>
<PropertyGroup>

<Target Name="SetAssemblyVersion">
 <AssemblyInfo CodeLanguage="CS"
 OutputFile="$(AssemblyInfoFile)"
 AssemblyCompany="Your company here"
 AssemblyProduct="Your product here"
 AssemblyCopyright="Your copyright here"
 ComVisible="false"
 AssemblyVersion="$(AssemblyVersion)"
 AssemblyFileVersion="(AssemblyVersion)" />
</Target>

Note that the BUILD_NUMBER and SVN_REVISION variables are environment variables set by the build server (Hudson in this case).

When you want to set the same version number for multiple assemblies, just simply add the assmblies as a link to the one assembly which will have the correct version number.