Thursday, October 4, 2018

Create NuGet packages for Managed and Native libraries

Managed libraries


Nuspec file


Before jumping into nuspec file creation, one should know that nuget can generate a package directly from a visual studio project:

nuget pack MyPackage.csproj

If you're willing to enter advanced mode, then you need a nuspec file.

First of all, instead of recreating the wheel, check the official documentation here : https://docs.microsoft.com/en-us/nuget/create-packages/creating-a-package

Now, here is a shortcut :
    1. Create a .nuspec file and fill it as follows:

    <?xml version="1.0"?>
    <package >
    <metadata>
        <id>Mypackage</id> <!-- Must be unique to avoid confusion during package resolutions -->
        <version>1.0.0</version> <!-- major.minor.micro[.optional] -->
        <title>My first package</title> <!-- Title of the library as it will appear in nuget manager -->
        <authors>Deadpool</authors> <!-- Author of the component -->
        <owners>Marvel</owners> <!-- Legal owner of the component -->
        <description>
        This is a short description of the component and describes what it is intended for.
        </description>
        <releaseNotes>
        [Optional] If this version has any particularity, it should be listed here
        </releaseNotes>
        <summary>
        Same as description but shorter
        </summary>
        <language>en-US</language>
        <projectUrl>http://mygitserver.com/mypackage</projectUrl>
        <requireLicenseAcceptance>false</requireLicenseAcceptance>
        <licenseUrl>http://opensource.org/licenses/Apache-2.0</licenseUrl> <!-- Or any other -->
        <copyright>Copyright if any</copyright>
        <dependencies> <!-- List any dependency that should be checked out with this package -->
            <group targetFramework="net35"> <!-- check all target frameworks here : https://docs.microsoft.com/en-us/nuget/reference/target-frameworks#supported-frameworks -->
                <dependency id="MyDependency" version="2.0.1" /> <!-- Here we specify to use MyDependency v2.0.1 built for .NET3.5 -->
                <!-- Others listed here -->
            </group>
        </dependencies> 
        <references></references> 
        <tags></tags> <!-- [Optional] any tags that would help your team finding your package here -->
    </metadata>
    <files>
        <!-- This will copy all the files from release build output to a lib subfolder in the package -->
        <!-- All references placed under lib will be automatically added to your project -->
        <!-- lib subfolder is mandatory. Create a folder for each supported frameworks -->
        <file src="bin\Release\**" target="lib\net35" exclude="**.pdb" /> 
        <!-- [Optional] Add documentation to the library -->
        <file src="doc\**" target="doc" />
    </files>
    </package>

Once this is done, creating the package is straightforward:

nuget pack Mypackage.nuspec

Notes that might help :

  •  Be careful with dependencies. If your package A depends on a package B 1.0.0.3258 with the latest digit being a build number, ensure that the exact same version of package B is packaged and stored in your nuget server. The build number is re-generated at each build and if it changes, the dependency resolution will fail. Ideally, remove this last digit from version reference.
  •  If your managed library relies on a native dll that is platform sepcific, you'll need to add the dlls to your package as follows:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>Phidgets</id>
    <version>2.1.8</version>
    <title>Phidgets</title>
    <authors>Me</authors>
    <owners>Me</owners>
    <description>
      Phidget interfacing library
    </description>
    <releaseNotes></releaseNotes>
    <summary>
      Phidget interfacing library for .NET
    </summary>
    <language>en-US</language>
    <projectUrl>https://mygitserver.com/Phidgets</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>http://opensource.org/licenses/Apache-2.0</licenseUrl>
    <copyright></copyright>
    <dependencies>
    </dependencies>
    <references></references>
    <tags></tags>
  </metadata>
  <files>
        <file src="lib\net20\**" target="lib\net20" exclude="**.pdb;" />
        <!-- Platform specific libraries are usually placed under a runtime subfolder in the package. -->
        <!-- The second level specifies the platform with an RID identifier (see here : https://docs.microsoft.com/en-us/dotnet/core/rid-catalog) -->
        <!-- Those libraries won't be copied to your output directory automatically. For that, either use a .targets file (see further in this article) or add a build step in your main project -->
        <file src="native\x86\**" target="runtimes\win-x86\native" />
        <file src="native\x64\**" target="runtimes\win-x64\native" />
  </files>
</package>

Native libraries


Native packages can be generated with a third party app called CoApp if you're not running under Windows 10. Unfortunately, this tool is not maintained anymore, and Win10 users will have to do that manually.

The good news is: it's not that hard.

If you understood the steps for a managed package, you've almost understood how to create a native package as well. Here is an example of nuspec file for a native library:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>stringlib</id>
    <version>4.1.0</version>
    <title>formatlib</title>
    <authors>Me</authors>
    <owners>Me</owners>
    <description>
      Provides string formatting functions for C/C++
    </description>
    <releaseNotes></releaseNotes>
    <summary>
      String formatting library for C/C++
    </summary>
    <language>en-US</language>
    <projectUrl>https://mygitserver.com/stringlib</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>https://opensource.org/licenses/BSD-3-Clause</licenseUrl>
    <copyright></copyright>
    <dependencies>
    </dependencies>
    <references></references>
    <tags>native</tags> <!-- This is important ! -->
  </metadata>
  <files>
      <!-- Note that now, binaries are placed conventionally under a directory named build -->
      <file src="bin\x86\Release\**" target="build\native\libs" exclude="**.pdb"/>
      <file src="include\**" target="build\native\interface" />
      <file src="doc\**" target="doc" />
  </files>
</package>

AFAIK, there is no clear rule on how one should manage the package structure. The structure I use is:

build
|-native
|-|-libs --> binaries go here
|-|-interface --> public headers go here
doc 

If I want to support several platforms, I can do that:
build
|-native
|-|-libs 
|-|-|-win-x86 --> binaries go here
|-|-|-win-x64 --> binaries go here
|-|-|-linux-x86 --> binaries go here
|-|-interface --> public headers go here
doc 

or

build
|-native
|-|-libs 
|-|-|-v100 --> VS2010 binaries go here
|-|-|-v110 --> VS2012 binaries go here
|-|-interface --> public headers go here
doc 

It is up to you to chose how you'd like them stored. Try to be generic (platform->architecture->toolchain).

Once the nuspec file is done, we have to add build rules that will automatically be merged into our solution. To do that, we can create .props file or .targets file or both !
These files will be imported into our project and Visual Studio will execute them during build.

These files need to be placed under 'build' subfolder in your nuget package and shall be named with your package id:

build
|-native
|-|-libs 
stringbuild.props
stringbuild.targets
|-|-interface --> public headers go here
doc 

Copy and paste the following code into your props file :

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" 
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemDefinitionGroup>
        <ClCompile>
            <AdditionalIncludeDirectories>$(MSBuildThisFileDirectory)native\interface\;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
        </ClCompile>
    </ItemDefinitionGroup>
    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
        <ClCompile>
            <RuntimeLibrary>MultiThreadedDebug</RuntimeLibrary>
        </ClCompile>
        <Link>
            <AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
            <!-- Put your lib name here -->
            <AdditionalDependencies>stringlib.lib;%(AdditionalDependencies)</AdditionalDependencies>
        </Link>
    </ItemDefinitionGroup>
    <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
        <ClCompile>
            <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
        </ClCompile>
        <Link>
            <AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
            <!-- Put your lib name here -->
            <AdditionalDependencies>stringlib.lib;%(AdditionalDependencies)</AdditionalDependencies>
        </Link>
    </ItemDefinitionGroup>
</Project>
This file tells Visual studio to add your package directory in include dirs and lib dirs. It also adds a dependency to the project. Say you'd like to automatically copy your dlls to your output directory, in this case, copy and paste the following code into your targets file:
<?xml version="1.0" encoding="utf-8" ?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemGroup>
        <StringLibBinaries Include="$(MSBuildThisFileDirectory)native\libs\*.dll" />
    </ItemGroup>
    <Target Name="StringLibCustomTarget" AfterTargets="Build">
        <Copy SourceFiles="@(StringLibBinaries)" DestinationFolder="$(OutDir)" />
    </Target>
</Project>

We're doing this in a targets file because props files can only add a build step that will overwrite any existing build steps in your project.

Alright, now that you're done, just pack your nuget package

nuget pack stringlib.nuspec

Notes that might help :

  •  If you're planning to pack static libraries, you might encounter link issues due to incremental debug level. Unfortunately, the only solution I found here was to pack both Debug and Release libraries and to link the right one against your project depending on your configuration

<AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\$(Configuration)\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

  •  Same trick can be used if you have packed libraries for different visual studio toolchains

<AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)native\libs\$(PlatformToolset)\;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>

  •  [Still to be investigated] Some packages are not installing properly from NuGet manager. The installation starts and stops without the green check mark that confirms the installation. I noticed that this was related to the name (e.g. ID) of the package. Suprisingly, adding numbers to the ID (and hence to the names of props and targets files) resolved the issue. This has been seen with VS2012, using Nuget v2.8.

Other things to know about Nuget

Check which version of nuget you need according to the version of visual studio you are using to avoid confusion !

List remote packages

nuget list -Source <url or server name>

Delete package on remote

nuget delete mypackage 1.0.0 -Source <url or server name>

Push a package to remote

nuget push mypackage.1.0.0.nupkg -Source <url>[/subfolder]

with subfolder typically being native or managed.

Force visual studio to install packages in a local directory

Nuget configuration can be overriden by adding a nuget.config file at the root of your solution.
To change the install directory, add this to the file:

<?xml version="1.0" encoding="utf-8"?>
<settings>
  <repositoryPath>..\packages</repositoryPath>
</settings>

In this example, packages will be placed in an upper folder.
Other options can be overriden as listed here : https://docs.microsoft.com/en-us/nuget/reference/nuget-config-file#example-config-file

Monday, August 13, 2018

Migration from SVN to GIT

Step 0 : Define migration strategy



Option 1 [recommended] : If possible, do not bother to migrate the SVN history to git but rather keep the old repositories archived and start over in git from the tip of the trunk.

Option 2 : Migrate trunk and tags only.

Option 3 [can be complex] : Migrate the entire repository.

I picked option 3 because my colleagues wanted to keep all of the history. IMO this is not necessary as long as the old repositories are still accessible. Plus, the migrated history is not perfectly identical to SVN and can be more confusing than helpful.


Step 1 : Create a file with authors


We should convert authors' svn identifiers to git format (Firstname Name <mail>).
To do this, run the following command from your SVN repo

svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors.txt


Now edit the file and fill the missing information to finally have something like that:

MMO = Mickey Mouse <mickey.mouse@disney.com>
DDU = Donald Duck <donald.duck@disney.com>

Step 2 : Migrate dependencies first


If your main project is using externals, those will need to be converted to submodules (other options are possible like subtree or third party git externals but in most cases, submodule will do just fine).

To do that, repeat the following commands for each dependency repository:

git svn clone --trunk=<trunk_dir> --branches=<branches_subdir> --tags=<tags_subdir> --no-metadata --authors-file=authors.txt <svn_repo_url> .


Note: if the svn repository follows the standard 'trunk,branches,tags' structure, replace '--trunk=<trunk_dir> --branches=<branches_subdir> --tags=<tags_subdir>' with --stdlayout

Now export the branch list to a file.

git branch -r > branches.txt


Filter the list of branches in branches.txt and keep only those you want to migrate.
Copy tags to a separate file for now (tags.txt).

Now for each entry in branches.txt, do:

git checkout -b <local_branch_name> <remote_branch_name>


Use the branch name in branches.txt for <remote_branch_name>.

For each tag, do the following:

git checkout -b <local_branch_name> <remote_branch_name>
git tag <local_branch_name>
git checkout master
git branch -D <local_branch_name> 

Tags are exported as branches by git-svn so we are forcing a checkout and we manually add the tag in git repository.

Now run the following command to clean history from empty commits (migration side-effect):

git filter-branch --prune-empty -f -- --all


Delete trunk branch

git branch -D trunk


Configure remote git repository

Here I suppose that you already have created a remote git repository to host your code.

git remote add origin <git_remote_repo_url>


Finally push the repo

git push origin --mirror


Step 3 : Migrate the main project repository


Repeat steps from Step 2 until trunk deletion.
From there, we will need to resolve externals.

We will start with master branch, but these steps need to be repeated for each branch.

git checkout master


Run the following command to export the list of externals to a file.

git svn show-externals --id=origin/trunk > externals.txt


Note: In my case, this command wouldn't work for other branches and it is particularily slow. I managed to achieve the same thing from svn command line in the old repository, for each branch.

svn propget svn:externals -R <externals_dir> > externals.txt


Suppose our svn:externals were in a folder named Externals, here is what should be done for each submodule:

git submodule add --force <dependency_git_remote_repo_url> ./Externals/<submodule_dir>


Then we need to fix the submodule to the commit that matches the version used in svn.

cd ./Externals/<submodule_dir>


For tags:
git checkout <commit_id_or_tag_label>


For branches:
git checkout -b <local_name> <remote_git_branch_name>


cd ../..


Repeat these steps for all submodules. Once done, commit:

git commit -m "Migrated svn:externals properties to .gitmodules"


Repeat these steps for all branches. Before starting, for each branch, do cleanup to avoid confusing error messages:

rm -rf ./Externals


Finally, when all branches have been processed, push the repository to remote server:

git remote add origin <git_remote_repo_url>

git push origin --mirror

Tuesday, November 21, 2017

Host a django app in Apache [Windows]

This article explains how to host a django application in Apache server. The following steps are valid for both 32 and 64 bits machine but be sure to install all of the tools for the same architecture.

For the following steps, we imagine that our project is located in C:/tools/SupervisionTool and is structured as shown below:

SupervisionTool/
  Analyzers/
    migrations/
    static/
    templates/
    ...
  SupervisionTool
    settings.py
    urls.py, wsgi.py
  manage.py


Before going further, be sure that you have the Administrator rights on the machine.

1. Install Python 3.6 and ensure that it is added to PATH
2. Install the last version of WAMP
3. Install VC15 redistributable tools
4. Create an environment variable named MOD_WSGI_APACHE_ROOTDIR and set it to apache install directory (c:/<wamp install dir>/bin/apache/apache<version>)
5. If you haven't created a virtual environment for your django application, start with

pip install virtualenvwrapper-win
mkvirtualenv SupervisionTool

<install all the packages necessary for your application (django,pypiwin32,...)>

Then

workon SupervisionTool
pip install mod_wsgi
mod_wsgi-express module-config

The last command will print the lines that will be necessary to configure Apache. You will see something like:


LoadFile "c:/python36/python36.dll"
LoadModule wsgi_module "c:/users/administrateur/envs/supervisiontool/lib/site-packages/mod_wsgi/server/mod_wsgi.cp36-win_amd64.pyd"

Copy these lines and add them to c:/<wamp install dir>/bin/apache/apache<version>/conf/httpd.conf

6. Open c:/<wamp install dir>/bin/apache/apache<version>/conf/extra/httpd-chosts.conf and replace the content with the following lines:

# virtual SupervisionTool
<VirtualHost *:80>
    ServerName localhost
    ServerAlias supervisiontool@lni-swissgas.lni.ads
    ErrorLog "logs/supervisiontool.error.log"
    CustomLog "logs/supervisiontool.access.log" combined
    WSGIScriptAlias /  "C:/tools/SupervisionTool/SupervisionTool/wsgi.py"
    <Directory "C:/tools/SupervisionTool/SupervisionTool">
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>

    Alias /Analyzers/static "C:/tools/SupervisionTool/static"
    <Directory "C:/tools/SupervisionTool/static">
        Require all granted
    </Directory>  
</VirtualHost>
# end virtual SupervisionTool


Note: Replace C:/tools/SupervisionTool with your install directory.

7. Run the following command from the project root dir to serve all of the static file (admin included) from a unique directory (defined by STATIC_ROOT):

python manage.py collectstatic

8. Set ALLOWED_HOSTS in SupervisionTool/settings.py to ['*'] (or list the IP adresses or symbols that will give access to the app)
9. Open SupervisionTool/wsgi.py and replace its content with :

activate_this = 'C:/Users/administrateur/Envs/SupervisionTool/Scripts/activate_this.py'
# execfile(activate_this, dict(__file__=activate_this))
exec(open(activate_this).read(),dict(__file__=activate_this))

import os
import sys
import site

# Add the site-packages of the chosen virtualenv to work with
site.addsitedir('C:/Users/administrateur/Envs/SupervisionTool/Lib/site-packages')

# Add the app's directory to the PYTHONPATH
sys.path.append('C:/tools/SupervisionTool')
sys.path.append('C:/tools/SupervisionTool/SupervisionTool')

os.environ['DJANGO_SETTINGS_MODULE'] = 'SupervisionTool.settings'
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "SupervisionTool.settings")

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()


9. Open Services in Windows and fin Apache (here wampapache) and set the start mode to Automatic. Finally, start apache service.

10. Type localhost in your web browser and verify that your app is accessible.


Troubleshooting



  • Compilation error with mod_wsgi-express module-config : The most probable reason is that you mixed 32/64 bits architectures when installing the tools. Try a cleanup and reinstall.
  • Apache fails to start in Services : Potentially a syntax error in configuration files. To get more information about this, right click on Wamp tools in tray icon menu, select Tools->Show Apache loaded Modules
  • Still not working ? Ensure that you configured PYTHONPATH correctly on host (here, it should be set to C:/tools/SupervisionTool/Analyzers)


Thursday, October 26, 2017

WCE7 : Add chinese characters support

This article explains how chinese characters can be added to WinCE7 OSDesign. Without this feature, the views would display empty square shapes instead of chinese text, meaning that the decoding of input strings failed.

Project properties

Right-click on your OSDesign project and select Locale in the left panel.


On the right side, fill all parameters with languages that you want to include in the image and save.

The NLS Table

The NLS table is a list of locale identifiers that is necessary to .NET Compact framework. The local identifiers of the languages that you want to include in your image must be listed in this table.

To do this, copy the file named nlscfg.inf from %_WINCEROOT%\public\COMMON\oak\files and paste it to %_WINCEROOT%\PLATFORM\<yourBSP>\FILES. Edit this file and overwrite its content with the following:

007f 0403 0406 0407 0807 0c07 1007 1407 0409 0809 0c09 1009 1409 1809 1c09 2009
082C 042C 0423 0402 0845 0445 0405 0465 0408 540A 0425 0429 0464 0447 040D 0439
101A 041A 040E 042B 0411 0437 043F 044B 0412 0457 0440 0427 0426 042F 0450 044E 
0446 0415 0418 0419 044F 041B 0424 041C 1C1A 0C1A 281A 181A 081A 045A 0449 044A 
041E 041F 0444 0422 0420 0843 0443 042A 0804 0C04 1404 1004 0404 2409 2809 2c09 
3009 3409 040a 080a 0c0a 100a 140a 180a 1c0a 200a 240a 280a 2c0a 300a 340a 380a 
3c0a 400a 440a 480a 4c0a 500a 040b 040c 080c 0c0c 100c 140c 180c 040f 0410 0810 
0413 0813 0414 0814 0416 0816 041d 081d 0421 042d 0436 0438 043e 083e 0441 0456
045e 3801 3C01 1401 0C01 0801 2C01 3401 3001 1001 1801 2001 4001 0401 2801 1C01
2401
LOC_INCLUDELOCALES


This table lists all identifiers supported by WCE7. As Erwin pointed out in his blog, some locale identifiers are not supported and will generate exceptions if used in your application.

Pick the right items from Catalog

Look for the following sysgen variables and ensure that they are checked :

SYSGEN_DOTNETV35_SR_CHS
SYSGEN_DOTNETV35_SR_CHT
SYSGEN_UNISCRIBE
SYSGEN_ FONTS_MSYH
SYSGEN_ FONTS_MSYHBD
SYSGEN_ FONTS_MSJH
SYSGEN_ FONTS_MSJHBD
SYSGEN_NLS_ZH
SYSGEN_NLS_ZH_* (all chinese locales)


Now, ensure that the following variables are NOT checked:

SYSGEN_GB18030
SYSGEN_AGFA_FONT

Finally rebuild the entire image. You should see a big difference in the image size (mine went from 37M to 111M).

Thursday, October 5, 2017

Application logs

Logging is a fundamental mechanism for the application. It brings valuable information to support teams when something goes wrong and saves a lot of time in bug investigation. But this utility can easily turn into a "spam engine" if the developers do not carefully select what/when/how much to log.

What do we expect from logs

A support team will look for relevant events or user actions that can explain a troublesome situation. Here is an example:

09:50:05 [INFO] [AppOperation] Mix configured with CMP=Nitrogen, CST=SynthAir, C=245,8 ppm
09:50:05 [INFO] [Solver] Computed flows for (C=2445,8 pp) : MFC1=1200 mlmin, MFC2=2450 mlmin
09:50:06 [INFO] [AppCalibration] Compensation of MFC1=1200 mlmin to MFC1=1202 mlmin
09:50:06 [INFO] [AppCalibration] Compensation of MFC2=2450 mlmin to MFC1=2449 mlmin
09:50:06 [INFO] [Peripherals] Setting MFC1=1200 mlmin
09:50:06 [INFO] [Peripherals] Setting MFC2=2449 mlmin
09:50:06 [INFO] [Peripherals] Setting Hardware state to Mix2Levels
09:50:06 [INFO] [AppOperation] Mix started
09:52:08 [ERROR] [AppAlarm] (Stability Alarm ON) MFC1 flow is not stable (Measured=1015 mlmin)

In this scenario, the user complains about poor results on his device. We suppose that the Alarm was not signaled on a visual indicator to inform the user. With this bunch of logs, developers can immediately:
  • Point the origin of the issue 
  • Get the conditions of the operation that lead to the issue and try to run the same experiment at his office
Obviously, none of these actions will solve the problem but the information that logs gave to the developer will highlight which functional part went wrong, and help him to follow the right track to resolve the client issue.
 It is essential for developers to have the following information in logs:
  • Timestamp 
  • Log level : developers will filter the errors or warnings first to find an obvious reason for the malfunction 
  • Synthetic information : Long phrases are useless: a log must go the point, there is no litterature trophy for well-written logs. Most of the time, use action verbs and parameters. 
  • Workable data : the parameters listed in our logs specify numerical values with a unit. It seems obvious to say that without the unit, the developer cannot investigate. Plus, the location where the error occured is necessary. In our example, the name of classes are displayed. Either do this or use functional area names logged (ex: Config, Security, Alarms, ...). 

Other things to have in mind:
  • Keep files small : do not go over a few MB for text files (I do not include DB here). Opening and parsing a heavy file is long and inefficient. Use several small rolling files instead. 
  • Keep short history : there is no need to keep traces from last week if nothing happened there. This is closely related to file size for text logs.

What to log

  • Application events: notifications, alarms, state changes, ... 
  • Operation-related information : settings update, user actions, ...

What NOT to log

  • Periodic data : if live values need to be logged regularly, put them in a separate log file and preferably go for a DB that can handle large amount of entries better. Developers can then refer to timestamps for investigations. 
  • GUI input errors : if a user enters invalid value, it is not an error for the application, it is an error for the form. As long as the frontend does not interact with the application, whatever happens between the user and the form does not have an impact on the application.

Where to log

A software architecture is built with modules interacting with each others. The business logic of the application is dispatched between these modules. So basically, each functional area of the application shall manage its own logs. To do things neatly, centralize the logging features in a service shared by all components.
Do not repeat information. Take the following example: A calls B, B fails. This is the type of log that we should expect from it:

10:00:01 [INFO] [A] Action A1 started
10:00:01 [INFO] [B] Setting xxx=1234
10:00:01 [ERROR] [A] Action A1 failed (Code=Invalid Setting Error)

and not something like :

10:00:01 [INFO] [A] Action A1 started
10:00:01 [INFO] [B] Setting xxx=1234
10:00:01 [ERROR] [B] Given value is invalid
10:00:01 [ERROR] [A] Action A1 failed (Code=Invalid Setting Error)

The error occured at a known place, so either log it from there or from the client (recommended).

Debugging information

Take the following example of code:

void ExecuteUserRequest()
{
   env = GetEnvironmentValue();
   if(env < THRESHOLD)
   {
      doA();
   }
   else
   {
      doB();
   }
}
Here, myFunc behaves depending on an external factor. Here, we can chose to log the action (doA or doB) with an INFO level and add debugging information to the user to help him understand why this precise action occured :

void ExecuteUserRequest()
{
   env = GetEnvironmentValue();
   LogDebug("env = %d", env);
   if(env < THRESHOLD)
   {
      LogInfo("Executing A");
      doA();
   }
   else if(env >= THRESHOLD)
   {
      LogInfo("Executing B");
      doB();
   }
}
Generally speaking, Debug level gives execution details and exposes implementation data to the developer.

Memo

Here are some basic rules that I use to write my application logs.

Code type Log level
user input error (GUI) No log.
Frequently accessed functions (ex: sensor value read) Debug or no log if we log in a text file. Use a DB if possible.
Implementation detail (subroutines, intermediate values,...) Debug
User action or application event Info
Alarm or abnormal situation that does not result in an error immediately Warn
Operation failure on user action Warn if the error is normal given the context (the product acts as specified). Error otherwise.
Application error (internal non-critical errors or errors managed by the application) Error
Critical non recoverable error (unmanaged exception, errors that lead to a non-functioning application) Fatal

Friday, July 21, 2017

Build U-Boot for BeagleCore

BeagleCore is a variant of the BeagleBone Black. It consists of 2 modules : BCM1 and BCS2:
* BCM1 is the System-On-Module that contains most of the BeagleBone Black design
* BCS2 is a mainboard with the form factor of the original BeagleBone Black. The BCM1 is soldered on BCS2 and the pair is an equivalence of the BBB.

Recently, I needed to run U-Boot on a custom board built with a BCM1. I followed some tutorials on the internet and the result was that U-Boot never started. After some investigations, I found out that the EEPROM that contains BBB identification is not present on BCM1, but is on BCS2 instead. The consequence is that U-Boot cannot identify the board and the initialization procedure fails.
In this post, I will detail every step to make U-Boot run on BCM1.

Note: All of these steps were performed on Windows 10 with bash (ubuntu terminal).

Install tools

sudo apt-get install gcc-arm-linux-gnueab

Create an alias to simply following commands


alias armmake='make -j8 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- '

Grab U-Boot

git clone
git://git.denx.de/u-boot.git u-boot
cd u-boot

Edit the code


Usually, we do a patch for these type of things. I'll try to post it later.
We will modify some parts of the code to stub the EEPROM so that U-Boot behaves exactly like it was a BBB board.

Open board_detect.c and add the following right after the #include statements:
static struct ti_common_eeprom stub_eeprom = 
{
    TI_EEPROM_HEADER_MAGIC,
    "A335BNLT",
    "00C0",
    "1716BBBK2450",
    "",
    {{0x11,0x22,0x33,0x44,0x55}, // Dummy Mac addresses
     {0x11,0x22,0x33,0x44,0x55},
     {0x11,0x22,0x33,0x44,0x55}},
    11749353662462671858,
    12656917672993063804
};

Replace the content of the function 'eeprom_am_set' with
{
    return 0;
}

Do the same with 'eeprom_am_get'.

Open board_detect.h.

Replace TI_EEPROM_DATA definition with:
 #define TI_EEPROM_DATA ((struct ti_common_eeprom *)\
                &stub_eeprom)

Config & Compile

armmake  distclean
armmake am335x_boneblack_defconfig
armmake

Copy to an SD Card


Format an SD card to FAT32 (boot flag ON).
Go to your U-Boot directory and copy MLO to SD Card first.
Then copy u-boot.img to SD-Card.

And there you go ! U-Boot shall start normally.
Here is an example of my uEnv.txt :
bootpart=0:1
bootdir=
bootfile=uImage
fdtfile=am335x-boneblack.dtb
fdtaddr=0x80F80000
loadaddr=0x80200000
optargs=quiet
mmcdev=0
mmcroot=/dev/mmcblk0p2 ro
mmcrootfstype=ext4 rootwait
loadbootenv=load mmc ${mmcdev} ${loadaddr} ${bootenv}
loadfdt=load mmc ${bootpart} ${fdtaddr} ${bootdir}/${fdtfile}
uenvcmd=load mmc 0 ${loadaddr} ${bootfile};run loadfdt;setenv bootargs console=${console} ${optargs};bootm ${loadaddr} - ${fdtaddr}


Tuesday, June 20, 2017

Machine learning basics

There are several types of machine learning but we will focus on the followings in this article:

  • Supervised learning
  • Unsupervised learning
  • Reinforcement learning

Supervised learning


Basically used to make predictions about future data from labeled/categorized training dataset.


A dataset is a table where:

  • each row is a sample
  • each column is a feature
  • each row is labeled with a class label

 Classification


A supervised learning task with discrete class labels is called a classification task.
Classification is a subcategory of supervised learning where the goal is to predict the categorical class labels of new instances based on past observations.

Example:

Dataset with  [data; label]
Data 0 : [ I like sport; Present]
Data 1 : [ I love shopping, Present]
Data 2 : [ I was in Amsterdam, Past]
Data 3 : [ I did something wrong, Past]
Data 4 : [ I do exercise every day, Present]

New data:

I did not know --> Past
I love running --> Present

With 2 possible class labels, the task is a binary classification task.
With more, it is a mutli-class classification task.

Example of multi-class dataset:

[Picture of cat; cat]
[Picture of dog; dog]
[Picture of mouse; mouse]

Here the machine learning system would be able to recognize a dog, a cat or a mouse but wouldn't succeed with any other animal because it is not part of our dataset.

Typical example of two-dimensionnal dataset for a binary classification task:

Data 0 : [  [0;0] ; Orange]
Data 1 : [  [1;1.5] ; Orange]
Data 2 : [  [1;2] ; Orange]
Data 3 : [  [1;2.8] ; Orange]
Data 4 : [  [2;1.5] ; Orange]
Data 5 : [  [2;2.5] ; Orange]
Data 6 : [  [3;0] ; Blue]
Data 7 : [  [3;1.5] ;Blue]
Data 8 : [  [3;2] ; Blue]
Data 9 : [  [4;2.8] ; Blue]
Data 10 : [  [4;1.5] ; Blue]
Data 11 : [  [4;2.5] ; Blue]
Data 12 : [  [5;3] ; Blue]

It is two-dimensionnal because each sample of the dataset has 2 values (usually named x1,x2). If we represent these samples on a 2-dimensionnal graph, we would see this:


The prediction would be based on the distribution of the sample. A point with x1 > 3 would be predicted as Blue and a point with x1 < 2 would potentially be red.

Regression


Regression is also called prediction of continuous outcomes. In regression analysis we give a serie of numbers (x or predictor) and response variables (y or outcome) and we try to find a relationship between them to predict a future outcome.

Ex:

with [x;y]
Data 0 : [ 0 ; 0 ]
Data 1 : [ 1 ; 1.5 ]
Data 2 : [ 1.5 ; 1 ]
Data 3 : [ 2 ; 2 ]
Data 4 : [ 2.5 ; 2.6 ]
Data 5 : [ 3 ; 3.2 ]
Data 5 : [ 4 ; 3.9 ]

Several types of algorithm can be selected to process input data. The following figure illustrates the concept of linear regression:


The computed curve will be used to predict the outcome of new data.

Reinforcement learning


Here the goal is to develop a system (agent) that improves its performances based on interactions with environment. The system will receive a feedback (reward) for every one of its actions. Each reward informs him of the quality of his action.
The agent will learn a series of actions that maximizes this reward via an empirical try-and-error approach.

A typical example is Google's Deepmind which beat the best Go players.

Unsupervised learning


In supervised learning, we include the right answer (labels) into the dataset. Here, we don't know the right answer beforehand. We are dealing with uncategorized data with an unknown structure.
With unsupervised learning, we can explore the structure of our data to extract meaningful information without an outcome or a reward.

Clustering


Clustering is an exploratory data analysis technique which groups data together by similarity (unnsupervised classification).

Dimensionality reduction


Dimensionality reduction is another unsupervised learning field. To prevent against the computation of huge amounts of data which results in performance and storage issues, unsupervised dimensionality reduction preprocesses data to remove noise and retain relevant information.


 
biz.