Continuous package delivery with Azure DevOps

Background

Since my team started to work with Azure DevOps, we've been exploring its potential progressively : Boards, Repos, Pipelines ...
During the first phase we became accustomed to how each of these unitary services work and the entire team now feels confortable with the them.
At this moment, I consider that the team's efficiency is equal to what it was before using Azure except we are using different tools now.

Stopping our exploration there would have been too bad considering the amount of manual labor that can be automated with Azure. Our development process should ideally converge to what is commonly called a "software factory" where most of the steps from code commits to software package delivery are automated.
I am not saying CI/CD here but the philosophy is equivalent. The continuous deployment of our built system is (at the time of writing) incompatible with the business. The packages are deployed manually in production by running an installer from a USB key. There is no automatic update support and the machines connectivity is limited to workshop automation. Not the ideal situation for DevOps but I believe we will get there sooner or later.

Our first step toward continuous deployment is focused on how each of the system components are developped in the team. The source code now resides in Azure Repo but the integrator is still checking-out the code to build the entire solution at once on his computer which means : build scripts to be updated, versions to be bumped and all binaries packed into a single installer. There is no opening for code reusability between teams: our repositories are private and the components are never archived anywhere.

As I used to work with NuGet packages in the past, all my developments are always packed and published to Azure Artifact. These packages are automatically fetched by visual studio at build time and hence, consumed as binaries and not code. My team really wants to do the same but they have limited knowledge with NuGet, pipeline configuration and artifact management. Besides, the integrator complains about the versioning because he's the only one to know how to increment the versions. I seized that opportunity to think about a solution where my team would only focus on code without ever caring about versioning, packing and publishing of the artifact.

Automation stages

Automation = Azure pipeline and the steps listed below will need to be scripted in an azure-pipelines.yml file

Versioning

It is critical to have a consistent versioning strategy among the development team to easily dissociate a major (i.e. breaking) change from a minor (i.e. compatible) by looking at the version. Besides, the version plays a key role in maintainability as it shall point the developers to the right code snapshot in history to fix whatever bug is found on client site.

I have oriented my team to Semantic Versioning 2.0 (SemVer 2) which is compatible with our traditional versioning strategy.
The version is delimited by the classic 3 digits plus additional metadata as shown below.

Major.Minor.Path-PreRelease.Counter+Build

Part	Reason for change	Nature
Major	Incompatible changes made to public API	Mandatory version number
Minor	New features added but backward compatibility preserved	Mandatory version number
Path	Bug fixes with backward compatibility preserved	Mandatory version number
PreRelease	Pre-release alphanumeric tag to denote the version	Optional build metadata
Counter	Pre-release version counter	Optional build metadata
Build	Build alphanumeric tag to denote the version	Optional build metadata

This versioning pattern is already supported by Azure Artifacts service and NuGet packaging technology since NuGet 4.3.0+ and Visual Studio 2017.

Now that we have the strategy, we have to define how to stamp this version in the binaries.

GitVersion

GitVersion is one of the tools that promises to be an automated SemVer 2.0 versioning system which will generate the right version for your current code depending on the branch you are on and by looking back at the history of the code.

GitVersion comes as a command line tool that can be executed from any git repository. It works out-of-the-box with GitHubFlow and GitFlow branching strategies and will generate a version without ever modifying the code. The version can then be published as an environment variable or injected into AssemblyInfo.

Below is an overview of the versions generated by GitVersion with the default configuration.

On this diagram, each arrow represents a branching or merging operation. The black labels show the version that is returned by GitVersion when executed from each branch.

Here are the default metadata settings built into GitVersion.

	master	hotfix	releases	dev	features
Versioning rule	Last tag used as base version	Patch incremented on creation	Branch name used as next version	Minor incremented when merging from master or release branch
PreRelease		beta	beta	alpha	branch name
Counter		Incremented and reset automatically by gitversion based on the references of a branch. Will keep incrementing until a higher digit gets incremented.
Build	Incremented on each commit since last tag

Note: To be used in .NET projects, the attributes AssemblyVersion and AssemblyFileVersion shall be deleted from AssemblyInfo and a dependency shall be added to GitVersionTask package.

Build and unit tests

Nothing special to be said on that part.
My team was already using pipelines with that setup and nothing needs to be modified for continuous package delivery.

Pack

GitVersion published multiple versions as environment variables. The output usually looks like this when printed in a json format:

{
  "Major":2,
  "Minor":3,
  "Patch":0,
  "PreReleaseTag":"alpha.2",
  "PreReleaseTagWithDash":"-alpha.2",
  "PreReleaseLabel":"alpha",
  "PreReleaseNumber":2,
  "WeightedPreReleaseNumber":2,
  "BuildMetaData":"",
  "BuildMetaDataPadded":"",
  "FullBuildMetaData":"Branch.dev.Sha.b5753b8ab047485908674e7a0c956009abff5528",
  "MajorMinorPatch":"2.3.0",
  "SemVer":"2.3.0-alpha.2",
  "LegacySemVer":"2.3.0-alpha2",
  "LegacySemVerPadded":"2.3.0-alpha0002",
  "AssemblySemVer":"2.3.0.0",
  "AssemblySemFileVer":"2.3.0.0",
  "FullSemVer":"2.3.0-alpha.2",
  "InformationalVersion":"2.3.0-alpha.2+Branch.dev.Sha.b5753b8ab047485908674e7a0c956009abff5528",
  "BranchName":"dev",
  "Sha":"b5753b8ab047485908674e7a0c956009abff5528",
  "ShortSha":"b5753b8",
  "NuGetVersionV2":"2.3.0-alpha0002",
  "NuGetVersion":"2.3.0-alpha0002",
  "NuGetPreReleaseTagV2":"alpha0002",
  "NuGetPreReleaseTag":"alpha0002",
  "VersionSourceSha":"0f42b52188fcda73f3e407063db85695ce4ace1a",
  "CommitsSinceVersionSource":2,
  "CommitsSinceVersionSourcePadded":"0002",
  "CommitDate":"2020-02-28"
}

There is a version string especially dedicated to NuGet packages : NuGetVersion. So all there is to do here is to inject that value into the packing task :

 # Package assemblies
  - task: NuGetCommand@2
    displayName: 'Packaging the artifact'
    inputs:
      command: 'pack'
      packagesToPack: '**/*.csproj;!**/*Tests.csproj'
      versioningScheme: 'byEnvVar'
      versionEnvVar: GitVersion.NuGetVersion
      includeReferencedProjects: true
      configuration: 'Release'

Publish

When a build completes, the created package will reside in what Azure defines as the staging directory, which is where the repository has been cloned for the build. This location is not accessible and if the team wants to share the package within the organization, they have to publish the artifact.

In Azure, the artifacts are stored in Feeds. A Feed is a repository for specific types of packages (npm, pypi, NuGet,…). All teams in Azure are free to create one or several Feeds depending on their needs.

Each Feed can have several Views. A View acts as an overlay of the Feed and is intended to filter the content. This concept has been originally introduced to defined several stages before releasing an artifact. By default, each Feed comes with 3 Views : @Local, @PreRelease and @Release, which store respectively development, release candidates and production artifacts. The diagram below summarizes these concepts.

By default, all packages are published into @Local. This View shall only be visible by developers to avoid interlocks during development.
When a release candidate is ready, the integrator shall promote the package from @Local to @PreRelease. The package becomes visible to the testers for verification and validation.
When a package is finally validated, the integrator will generate a new package that he will promote to @Release. The package becomes visible to all stakeholders within the organization.

Each Feed can define a maximum retention time for the package it stores. When the delay expires, the package is deleted. This retention delay is only applied to @Local and promoted packages won't be deleted by the defined policy.

It is up to each team to configure the permission levels for each view.

After configuring our Azure Artifact feed with the proper permission levels and retention time, we were ready to rollout the first automated package publication.
It worked as expected for the test project. One of my input requirements was that my team needs to focus on code only which means that they should never configure the pipeline for their project. As the pipeline configuration file stands in the project repository, I looked for a way of reusing existing pipeline configuration files ...

Pipeline template

Since December 2019, Azure supports templating with the reuse of pipeline config files located in external repositories. My team and I arrived just in time ! :)

Below is the template that I have pushed to a 'TeamProcess' repository:

# File : base-netfull-pipeline.yml
#
# Azure pipeline configuration to build .NET Framework projects and publish
# them as NuGet artifacts into GF.MS.LAS.Machine Azure feed
parameters:
# Solution path in repository
- name: 'solution'
  default: '**/*.sln'
  type: string
# Target build platform
- name: 'buildPlatform'
  default: 'Any CPU'
  type: string
# Build configuration
- name: 'buildConfiguration'
  default: 'Release'
  type: string
# Build virtual image
- name: 'vmImage'
  default: 'windows-latest'
  type: string
# Source feed
- name: 'feed'
  default: '7ea4c5d0-fe57-441e-9fac-f026c9bb1207'
  type: string
# Packages to pack
- name: 'packagesToPack'
  default: '**/*.csproj;!**/*Tests.csproj'
  type: string
# Packages to push
- name: 'packagesToPush'
  default: '$(Build.ArtifactStagingDirectory)/**/*.nupkg;!$(Build.ArtifactStagingDirectory)/**/*.symbols.nupkg'
  type: string
# Does NuGet shall include all dependencies as reference package and/or dlls in the artifact ?
- name: packageAddReferences
  type: boolean
  default: true


jobs:
- job: Build
  pool:
    vmImage: ${{ parameters.vmImage }}
  steps:
  # Install NuGet utility
  - task: NuGetToolInstaller@1
    displayName: 'Installing NuGet utility'

  # Generate SemVer version
  - task: DotNetCoreCLI@2
    displayName: 'Install gitversion'
    inputs:
      command: 'custom'
      custom: 'tool'
      arguments: 'install -g gitversion.tool'

  - task: DotNetCoreCLI@2
    displayName: 'Gitversion setup'
    inputs:
      command: 'custom'
      custom: 'gitversion'
      arguments: '/output buildserver'

  # Restore project dependencies
  - task: NuGetCommand@2
    displayName: 'Restoring dependencies of the package'
    inputs:
      command: 'restore'
      restoreSolution: '${{ parameters.solution }}'
      feedsToUse: 'select'
      vstsFeed: '${{ parameters.feed }}'

  # Build
  - task: VSBuild@1
    displayName: 'Building solution'
    inputs:
      solution: '${{ parameters.solution }}'
      platform: '${{ parameters.buildPlatform }}'
      configuration: '${{ parameters.buildConfiguration }}'

  # Execute unit tests
  - task: VSTest@2
    displayName: 'Executing unit tests'
    inputs:
      platform: '${{ parameters.buildPlatform }}'
      configuration: '${{ parameters.buildConfiguration }}'

  # Package assemblies
  - task: NuGetCommand@2
    displayName: 'Packaging the artifact'
    inputs:
      command: 'pack'
      packagesToPack: '${{ parameters.packagesToPack }}'
      versioningScheme: 'byEnvVar'
      versionEnvVar: GitVersion.NuGetVersion
      includeReferencedProjects: ${{ parameters.packageAddReferences }}
      configuration: '${{ parameters.buildConfiguration }}'

  # Publish assemblies
  - task: NuGetCommand@2
    displayName: 'Publishing the artifact to feed'
    inputs:
      command: 'push'
      packagesToPush: '${{ parameters.packagesToPush }}'
      nuGetFeedType: 'internal'
      publishVstsFeed: '${{ parameters.feed }}'

To reuse this template, a project will create its own azure-pipelines.yml file with the following content:

# File: azure-pipelines.yml
resources:
  repositories:
    - repository: templates
      type: git
      name: TeamProcess

# Template reference
jobs:
- template: Process/Pipelines/net/base-netfull-pipeline.yml@templates

Conclusion

With a few days invested in reading msdn and other literature about the setup of Azure, I managed to achieve the creation of a fully automated NuGet package continuous delivery flow. The automatic versioning of the code is something I never thought of in the past but it is a real game changer. Without it, creating this delivery flow would have been trickier with additional scripting and/or code commits prior to build. The only difficulty I face was related to GitVersion add-on(s) in Azure. Many of them co-exist in the marketplace and it's really confusing. My recommendation is to used DotNetCLI instead which is a robust workaround to the add-on.

Read More

Create NuGet packages for Managed and Native libraries

Managed libraries

Nuspec file

Before jumping into nuspec file creation, one should know that nuget can generate a package directly from a visual studio project:

nuget pack MyPackage.csproj

If you're willing to enter advanced mode, then you need a nuspec file.

First of all, instead of recreating the wheel, check the official documentation here : https://docs.microsoft.com/en-us/nuget/create-packages/creating-a-package

Now, here is a shortcut :
    1. Create a .nuspec file and fill it as follows:

    <?xml version="1.0"?>
    <package >
    <metadata>
        <id>Mypackage</id> <!-- Must be unique to avoid confusion during package resolutions -->
        <version>1.0.0</version> <!-- major.minor.micro[.optional] -->
        <title>My first package</title> <!-- Title of the library as it will appear in nuget manager -->
        <authors>Deadpool</authors> <!-- Author of the component -->
        <owners>Marvel</owners> <!-- Legal owner of the component -->
        <description>
        This is a short description of the component and describes what it is intended for.
        </description>
        <releaseNotes>
        [Optional] If this version has any particularity, it should be listed here
        </releaseNotes>
        <summary>
        Same as description but shorter
        </summary>
        <language>en-US</language>
        <projectUrl>http://mygitserver.com/mypackage</projectUrl>
        <requireLicenseAcceptance>false</requireLicenseAcceptance>
        <licenseUrl>http://opensource.org/licenses/Apache-2.0</licenseUrl> <!-- Or any other -->
        <copyright>Copyright if any</copyright>
        <dependencies> <!-- List any dependency that should be checked out with this package -->
            <group targetFramework="net35"> <!-- check all target frameworks here : https://docs.microsoft.com/en-us/nuget/reference/target-frameworks#supported-frameworks -->
                <dependency id="MyDependency" version="2.0.1" /> <!-- Here we specify to use MyDependency v2.0.1 built for .NET3.5 -->
                <!-- Others listed here -->
            </group>
        </dependencies> 
        <references></references> 
        <tags></tags> <!-- [Optional] any tags that would help your team finding your package here -->
    </metadata>
    <files>
        <!-- This will copy all the files from release build output to a lib subfolder in the package -->
        <!-- All references placed under lib will be automatically added to your project -->
        <!-- lib subfolder is mandatory. Create a folder for each supported frameworks -->
        <file src="bin\Release\**" target="lib\net35" exclude="**.pdb" /> 
        <!-- [Optional] Add documentation to the library -->
        <file src="doc\**" target="doc" />
    </files>
    </package>

Once this is done, creating the package is straightforward:

nuget pack Mypackage.nuspec

Notes that might help :

•  Be careful with dependencies. If your package A depends on a package B 1.0.0.3258 with the latest digit being a build number, ensure that the exact same version of package B is packaged and stored in your nuget server. The build number is re-generated at each build and if it changes, the dependency resolution will fail. Ideally, remove this last digit from version reference.

•  If your managed library relies on a native dll that is platform sepcific, you'll need to add the dlls to your package as follows:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>Phidgets</id>
    <version>2.1.8</version>
    <title>Phidgets</title>
    <authors>Me</authors>
    <owners>Me</owners>
    <description>
      Phidget interfacing library
    </description>
    <releaseNotes></releaseNotes>
    <summary>
      Phidget interfacing library for .NET
    </summary>
    <language>en-US</language>
    <projectUrl>https://mygitserver.com/Phidgets</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>http://opensource.org/licenses/Apache-2.0</licenseUrl>
    <copyright></copyright>
    <dependencies>
    </dependencies>
    <references></references>
    <tags></tags>
  </metadata>
  <files>
        <file src="lib\net20\**" target="lib\net20" exclude="**.pdb;" />
        <!-- Platform specific libraries are usually placed under a runtime subfolder in the package. -->
        <!-- The second level specifies the platform with an RID identifier (see here : https://docs.microsoft.com/en-us/dotnet/core/rid-catalog) -->
        <!-- Those libraries won't be copied to your output directory automatically. For that, either use a .targets file (see further in this article) or add a build step in your main project -->
        <file src="native\x86\**" target="runtimes\win-x86\native" />
        <file src="native\x64\**" target="runtimes\win-x64\native" />
  </files>
</package>

Native libraries

Native packages can be generated with a third party app called CoApp if you're not running under Windows 10. Unfortunately, this tool is not maintained anymore, and Win10 users will have to do that manually.

The good news is: it's not that hard.

If you understood the steps for a managed package, you've almost understood how to create a native package as well. Here is an example of nuspec file for a native library:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>stringlib</id>
    <version>4.1.0</version>
    <title>formatlib</title>
    <authors>Me</authors>
    <owners>Me</owners>
    <description>
      Provides string formatting functions for C/C++
    </description>
    <releaseNotes></releaseNotes>
    <summary>
      String formatting library for C/C++
    </summary>
    <language>en-US</language>
    <projectUrl>https://mygitserver.com/stringlib</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>https://opensource.org/licenses/BSD-3-Clause</licenseUrl>
    <copyright></copyright>
    <dependencies>
    </dependencies>
    <references></references>
    <tags>native</tags> <!-- This is important ! -->
  </metadata>
  <files>
      <!-- Note that now, binaries are placed conventionally under a directory named build -->
      <file src="bin\x86\Release\**" target="build\native\libs" exclude="**.pdb"/>
      <file src="include\**" target="build\native\interface" />
      <file src="doc\**" target="doc" />
  </files>
</package>

AFAIK, there is no clear rule on how one should manage the package structure. The structure I use is:

build
|-native
|-|-libs --> binaries go here
|-|-interface --> public headers go here
doc 

If I want to support several platforms, I can do that:

build
|-native
|-|-libs 
|-|-|-win-x86 --> binaries go here
|-|-|-win-x64 --> binaries go here
|-|-|-linux-x86 --> binaries go here
|-|-interface --> public headers go here
doc 

or

build
|-native
|-|-libs 
|-|-|-v100 --> VS2010 binaries go here
|-|-|-v110 --> VS2012 binaries go here
|-|-interface --> public headers go here
doc 

It is up to you to chose how you'd like them stored. Try to be generic (platform->architecture->toolchain).

Once the nuspec file is done, we have to add build rules that will automatically be merged into our solution. To do that, we can create .props file or .targets file or both !
These files will be imported into our project and Visual Studio will execute them during build.

These files need to be placed under 'build' subfolder in your nuget package and shall be named with your package id:

build
|-native
|-|-libs 
stringbuild.props
stringbuild.targets
|-|-interface --> public headers go here
doc 

Copy and paste the following code into your props file :

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" 
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemDefinitionGroup>
        <ClCompile>
            <AdditionalIncludeDirectories>$(MSBuildThisFileDirectory)native\interface\;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
        </ClCompile>
    </ItemDefinitionGroup>
    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
        <ClCompile>
            <RuntimeLibrary>MultiThreadedDebug</RuntimeLibrary>
        </ClCompile>
        <Link>
            <AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
            <!-- Put your lib name here -->
            <AdditionalDependencies>stringlib.lib;%(AdditionalDependencies)</AdditionalDependencies>
        </Link>
    </ItemDefinitionGroup>
    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
        <ClCompile>
            <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
        </ClCompile>
        <Link>
            <AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
            <!-- Put your lib name here -->
            <AdditionalDependencies>stringlib.lib;%(AdditionalDependencies)</AdditionalDependencies>
        </Link>
    </ItemDefinitionGroup>
</Project>

This file tells Visual studio to add your package directory in include dirs and lib dirs. It also adds a dependency to the project. Say you'd like to automatically copy your dlls to your output directory, in this case, copy and paste the following code into your targets file:

<?xml version="1.0" encoding="utf-8" ?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemGroup>
        <StringLibBinaries Include="$(MSBuildThisFileDirectory)native\libs\*.dll" />
    </ItemGroup>
    <Target Name="StringLibCustomTarget" AfterTargets="Build">
        <Copy SourceFiles="@(StringLibBinaries)" DestinationFolder="$(OutDir)" />
    </Target>
</Project>

We're doing this in a targets file because props files can only add a build step that will overwrite any existing build steps in your project.

Alright, now that you're done, just pack your nuget package

nuget pack stringlib.nuspec

Notes that might help :

•  If you're planning to pack static libraries, you might encounter link issues due to incremental debug level. Unfortunately, the only solution I found here was to pack both Debug and Release libraries and to link the right one against your project depending on your configuration

<AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\$(Configuration)\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

•  Same trick can be used if you have packed libraries for different visual studio toolchains

<AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\$(PlatformToolset)\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

•  [Still to be investigated] Some packages are not installing properly from NuGet manager. The installation starts and stops without the green check mark that confirms the installation. I noticed that this was related to the name (e.g. ID) of the package. Suprisingly, adding numbers to the ID (and hence to the names of props and targets files) resolved the issue. This has been seen with VS2012, using Nuget v2.8.

Other things to know about Nuget

Check which version of nuget you need according to the version of visual studio you are using to avoid confusion !

List remote packages

nuget list -Source <url or server name>

Delete package on remote

nuget delete mypackage 1.0.0 -Source <url or server name>

Push a package to remote

nuget push mypackage.1.0.0.nupkg -Source <url>[/subfolder]

with subfolder typically being native or managed.

Force visual studio to install packages in a local directory

Nuget configuration can be overriden by adding a nuget.config file at the root of your solution.
To change the install directory, add this to the file:

<?xml version="1.0" encoding="utf-8"?>
<settings>
  <repositoryPath>..\packages</repositoryPath>
</settings>

In this example, packages will be placed in an upper folder.
Other options can be overriden as listed here : https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file#example-config-file

Read More