Rick Minerich's Development Wonderland : C++

How to determine which language(s) were used to build a .NET assembly

RickM — Thu, 26 Feb 2009 11:00:00 GMT

While in most cases there is no explicit information in an assembly as to which languages it was compiled from, it is possible to make an educated guess as to which languages were used. This is due to the fact that each different .NET compiler leaves it’s own unique type of fingerprint. In this article I discuss both my methodology for finding these fingerprints and which were unique to each language I used.

Methodology

For each language I made a new class library project. I then reflected and compared each assembly to determine which unique characteristics it had. It turned out that, at least for C#, F#, VB and C++, each was uniquely identifiable by the existence, or lack thereof, of certain features.

So to break it down a bit.

In each project I added one class and one public method in each of those classes:

  1: public class CSharpClass
  2: {
  3:     public void LocalMethod() {}
  4: }

After compiling each of these projects into it’s own assembly, I referenced them from another testing project. To grab a set of features for each language, I used the following three reflection calls:

Assembly.GetTypes()
Assembly.GetCustomAttirbutes()
Module.GetFields( BindingFlags.NonPublic | BindingFlags.Static )

Then, with a simple program, I found which of these features were unique for each language. This set of unique features ultimately represents a map of the imprint each compiler leaves.

F#

A compiled F# library will only have one attribute by default:

Microsoft.FSharp.Core.FSharpInterfaceDataVersionAttribute

This made it the easiest to differentiate of all the languages I tested. Even more interesting, this attribute contains three fields which specify the specific version of the F# compiler used to generate the assembly:

Major    1    int
Minor    9    int
Release    6    int

I’m always impressed with how the F# team consistently goes above and beyond when it comes to the small details.

Visual Basic

The Visual Basic assembly I generated was also easily identifiable via extra types which were automatically added:

My.MyApplication
My.MyComputer
My.MyProject
My.MyProject+MyWebServices
My.MyProject+ThreadSafeObjectProvider`1
My.Resources.Resource
My.MySettings
My.MySettingsProperty

As you can see from this list, the existence of these types in the “My” namespace is a fairly safe indicator that the Visual Basic language was used.

C++ CLI / Managed C++

C++/CLI and Managed C++ are considered to be the same language with slightly different syntax as they share the same compiler. However, there are four different compilation modes for C++ and each has somewhat different results.

/CLR – Common Language Runtime Support
/CLR:pure – Pure Common Language Runtime Support
/CLR:safe – Safe Common Language Runtime Support
/CLR:OldSyntax – Managed C++ Syntax

The /CLR, /CLR:pure and /CLR:OldSyntax settings provide easy to classify assemblies, as they all inject an enormous number of types (70+) into the assembly. I verified that contained two types from the vc_attributes namespace:

vc_attributes.YesNoMaybe
vc_attributes.AccessType

However, /CLR:Safe is much different in that it injects no types and adds no assembly attributes by default. The generated assembly was almost completely clean. I was forced to use Reflector to determine how to differentiate this from C#.

C#

C# was one the most difficult to identify assembly type. This is due to the fact that it has no unique types and only one unique attribute:

System.Reflection.AssemblyConfigurationAttribute

Unfortunately, this attribute is defined in the AssemblyInfo.cs file and so we can’t depend on it. Up to this point it was only necessary to use two reflection calls:

Assembly.GetTypes()
Assembly.GetCustomAttirbutes()

I was hoping to keep things very simple. However, to differentance these two languages it’s necessary to go a bit further. It turns out that C++ always injects an module level field into the assembly while C# does not. And so by using:

Module.GetFields(BindingFlags.NonPublic | BindingFlags.Static)

We can check for the existence of this kind of field and so differentiate these two types.

After some investigation with reflector, I was able to find one particular feature unique to C#. Unfortunately, it requires disassembling functions and looking at the resulting IL. It seems as though a function definition never has a .maxstack of less than 8. In all other languages I observed .maxstack had been set to values as low as 0 when defined in an empty function.

However, as I am only currently concerned with these four languages, my testing on this matter has been very shallow and so pleae take it with a grain of salt.

Conclusion

I admit that my sample Assembly set was very small and my feature set very large. However, while this type of classification may not be robust enough to be applicable to a system which depended on these results being absolutely true, I’ve shown that it is in fact entirely possible to make reasonably confident guesses as to the language used to generate a .NET assembly while using only simple reflection. It would be interesting to see how well this holds for obfuscated assemblies as well as other “bare minimum” compilations generated via different combinations of compiler settings.

The next obvious step would be to extend what I have already written into a full Bayesian classifier. Would be much better than a hardcoded hierarchy which would be fragile and possibly completely and repeatedly incorrect for some cases. Another big advantage of using machine learning here, is that it would be easy to add new features and classification categories.

One of these languages is not like the others - Part 1 - Enforcement of Abstract Implementation

RickM — Tue, 10 Feb 2009 15:47:00 GMT

This is the first in a series of posts on the topic of interaction between different .NET languages. I will cover all of the major Microsoft languages: C#, Visual Basic, F# and C++/CLI.

In this first post in the series I will build a four language project in Visual Studio 2008 and begin to explore inter-language inheritance. One of the languages behaves in a significantly different way when compared with the others, can you guess which one it is?

Enforcement of Abstract Implementation

First things first, let’s define our abstract base class. C# is considered the standard language for library architecting and so it is what we will be using.

  1: public abstract class CSharpBaseClass
  2: {
  3:   public abstract void PublicAbstractMethod();      
  4:   public abstract void PublicAbstractMethod(int inint);
  5:   protected abstract void ProtectedAbstractMethod();
  6:   protected abstract void ProtectedAbstractMethod(int inint);
  7: }

In each of our child projects we will be inheriting from this class. Let’s start with C#.

  1: public class CSharpChildClass : CSharpBaseClass
  2: {
  3: }

Now anyone with even a bit of experience in the C# language knows that this won’t compile. For each of the abstract methods in the base class the inheriting class is expected to provide an implementation.

… error CS0534: 'CSharpChild.CSharpChildClass' does not implement inherited abstract member …

This is similarly true for Visual Basic.

  1: Public Class VBChildClass

  2:   Inherits CSharpBase.CSharpBaseClass

  3: End Class

… error BC30610: Class 'VBChildClass' must either be declared 'MustInherit' or override the following inherited 'MustOverride' member(s):

F# shares this behavior as well.

  1: type FSharpChildClass() =
  2:   inherit CSharpBase.CSharpBaseClass()

… error FS0191: No implementation was given for 'CSharpBaseClass.PublicAbstractMethod() : unit'.

Kudos to the F# compiler designers for the verbose compile time errors.

error FS0054: This type is 'abstract' since some abstract members have not been given an implementation. If this is intentional then add the '[<AbstractClass>]' attribute to your type.

However, C++/CLI does not. It’s perfectly acceptable to inherit from an abstract class and provide no implementation for it’s abstract members.

  1: public ref class CppChildClass : public CSharpBaseClass
  2: {
  3: };

While a warning is issued, it seems insufficient for what has just taken place:

… warning C4570: 'CppChild::CppChildClass' : is not explicitly declared as abstract but has abstract functions

The result is that the class in question is defaulted to abstract. If someone where to attempt to instantiate it downstream, it would cause a compiler error:

  1: CppChildClass cpp = new CppChildClass();
  2: cpp.LocalMethod();

error CS0144: Cannot create an instance of the abstract class or interface 'ManagedCppChild.ManagedCppChildClass'

This is bad because, if you were an API developer and so did not consume all of your objects, it would be possible to ship a version of your product with abstract suddenly toggled on for one or more of your classes. All it would take would be for a developer to add a new inherited class or member to an existing abstract base class without writing a corresponding test for every abstract member or inheriting class.

Conclusion

So, what can we learn from this? It seem that when using C++/CLI (or managed C++) it is not sufficient to depend on abstract classes to enforce that members are implemented. Testing must also be strictly enforced.

Avoiding the Dangers of Ambiguously Defined Data-Types

RickM — Fri, 03 Oct 2008 15:10:00 GMT

When you are handed a string, integer, or any value type, can you know what it really represents? Can you define the range of appropriate behaviors for that data? Can you tell if it's formatted correctly? The problem is, in all of these cases, you can't. You can't be sure of it's meaning, it's format or even how to treat it. This is why ambiguous types break the object-oriented programming paradigm and should be avoided whenever possible.

Forward

It's laughable to think that a Physical Engineer or Physicist would work without the aid of units. This is because, when you have undefined units, you can't verify that the thing you get at the end of a computation is defined in a consistent way.

In computing, the counterpart to the Physical Engineer is the Software Engineer. Why is the Software Engineer not held to the same type of design integrity? The use of ambiguously defined variables is one of the largest sources of bugs in modern software development and yet we ignore it. In fact, with many dynamically typed languages we are moving in the direction of more and more ambiguity.

In C# and Java, strings and value types are defined only in terms of their lexical and mathematical operations. Except from the context in which they are used, they carry no additional information about the meaning of their content. In order to perform any non-trivial operation on that data, you have to make assumptions about it's meaning, which in turn leads to bugs.

Is what you are writing really object oriented?

This may seem like a silly question to ask in the modern programming era. Almost everyone is using an object oriented language these days and its mostly taken for granted. However, as any beginning programmer knows, it is very easy to program in an object oriented programming language while completely ignoring the underlying paradigm.

We don't use OO because it's an agreed standard for implementing a programming language, we use it to solve a specific set of problems. These problems are directly related to the modeling of data and data manipulation. Groups of data subtypes are classified into a larger object and operations on that data are defined in such a way as to model behavior in terms of that classification. If the paradigm is ignored these issues become manifest as ambiguity and disorganization.

It follow that using ambiguous data-types in a public API goes directly against not only the object oriented paradigm but also the broader goal of data classification and program integrity.

Why not pass strings and value types as arguments to public methods?

Using an ambiguously defined data type instead of a well defined object means that the domain and formatting of that data is left open to question. It also means that if a user of that data formats it inappropriately it may find it's way into an operation for another type of data and cause havoc.

For a programmer that is familiar with object oriented programming it often comes down to a choice of encapsulation versus convenience. I know as well as anyone that having to define an object to carry your data when you could just pass in a string and a couple of integers can feel tedious. However, by taking the easy way out, you are setting things up to fail down the road.

Consider, is it possible that sometime in the future someone will be using the API you are defining? Will the person looking at that code know the specific format you chose for that string? What about the range of valid values for those integers? What kinds of assumptions about that data will be made that you may have not considered? If they mess up that format or get the range wrong, at what point will it become obvious?

There are a lot of advantages to using objects instead: An object which was designed to carry that same data could verify that it is well defined when it is constructed. It can hand back that data formatted or manipulated in many different ways. You can well defined questions which can be asked about that data which makes program control flow more obvious. Best of all, the code for all of these things is centralized and up to the discretion of the designer of said object. Using a well defined object, two people with different assumptions about that data but asking the same question will get the same result.

Of course, at the lowest level a CPU will be operating on basic value types and so our code will always reflect that to some degree. However, because ambiguous types are extremely dangerous, they should almost always be encapsulated and well defined.

What tools are available to me in .NET?

If you are a .NET programmer the best thing you can do right now is to become familiar with the already existing classes available for making your strings more well defined such as System.Uri and System.IO.FileInfo. I've put up a question on stackoverflow in order to try to build up a list of available container classes.

Beyond the use of predefined classes, it's best to make your own encapsulation objects with heavy up front validation. You can then use extension methods to make native .NET classes take your new validated type. It would be worth putting together a library of free encapsulation classes, structs and extension methods to allow for easy interoperability.

Beware, the .NET Framework encourages ambiguous types.

Unfortunately, as well designed as it is for many things, the .NET Framework is pretty bad about ambiguity. Based on classes such as System.IO.Path and System.IO.FileStream which for some reason take paths represented as strings, you might even say that ambiguity is encouraged. Consider the vast number of methods in .NET classes which take unencapsulated strings and scalar types.

The most unfortunate side effect of this design is that users of the .NET API may come to believe that this is the way things should be in a proper object oriented API. Also, as a consequence, if you want your own product's API to be easily understood by a .NET user you have to follow the same destructive conventions.

F# helps to solve this problem with Units of Measure.

In most object oriented languages, if you were to encapsulate every single scalar value that was passed into a method, it would be quite a lot of extra coding. Microsoft's newest programming language, F#, has a feature called Units of Measure which allows a programmer to optionally specify both meaning and behavior for classes of scalar types.

A scalar with a unit of measure is a real type which is enforced by the compiler. When an operation is performed the resulting type is that of the combined units used for the calculation, just as they would be in physics or engineering. This is because F# is designed in part to be used by engineers and scientists. As a side effect we as programmers get to reap the same benefit.

This type of scalar type classification has long been missing from object oriented languages. It's a huge step forward and I hope other programming languages move to adopt it quickly.

Addendum

@Mark and Jon:

I think I may not have fully expressed what I meant by "handed a string or integer". The context am talking about is post-compilation when variable names no longer have real meaning. In this context (discounting reflection on variable name) all you can tell about an integer is where it is coming from and it's value. An executing program does not have access to documentation.

It's also important to consider reflective programming. It is generally agreed that reflective programming is where the object oriented world is headed. However, when you have a scalar value, you can tell very little about it through reflection. While it is possible to retrieve the variable's name using reflection, actually using the name of said variable to carry type information is a methodology which is extremely prone to errors from mistyping. Errors which will not be caught at compile time and which may lead your program to incorrect paths of execution.

Also, I want to note that while a string is not a scalar value it does suffer from the same type of ambiguity. For this reason it, and other "base" types, should be handled similarly.

Changing Your Garbage Collector Settings on the Fly (.NET Memory Management: Part 5)

RickM — Wed, 20 Aug 2008 20:39:00 GMT

It has come to my attention via a recent DZone article that .NET 3.5 and 2.0 SP1 jointly included a new feature which lets you manipulate the way your garbage collector acts programmatically. This can be done through changing the value of a new property of the System.Runtime.GCSettings class named LatencyMode. In this article I will walk you through this new property and the different effects of each of it's possible settings.

Other Articles in This Series

Part 1 – Basic Housekeeping
Part 2 – Improving Performance Through Stack Allocation
Part 3 – Increasing the Size of your Stack
Part 4 – Choosing the Right Garbage Collector Settings
Part 5 – Changing Your Garbage Collector Settings on the Fly

LatencyMode

The new LatencyMode property accepts three different enumerated values. The first two of which are Batch and Interactive which correspond to turning <gcConcurrent> on and off inside of your application configuration. However, the effects of the third value, which is named LowLatency, were previously unavailable by any means. By setting this the LatencyMode to LowLatency you can now put the Garbage Collector in an ultra-conservative mode which will insure the application will be interrupted as infrequently as possible.

There are a couple of things you should take note of before changing your Garbage Collector's LatencyMode:

LatencyMode settings are process-wide and so all of the threads in your application will be effected.
Changing LatencyMode will have no effect if your application has <gcServer> mode set in its configuration file.

Now, let’s explore each of the possible values of LatencyMode in detail:

Batch

When it’s useful:
It’s useful in applications without a UI and in Server Based Software. It may also be useful to switch your garbage collector into Batch latency mode if your application has a data crunching mode which occurs after a user interface.

What it does:
This is the most intrusive setting. Garbage Collection is done non-concurrently in one big batch call. Your program will be suspended while garbage collection takes place.

Defaults:
This is the default value when <gcConcurrent> is disabled.

Notes:
Even though this will be the enumeration which will be set if <gcServer> is enabled, the garbage collector will act as described in my previous article. In fact, if you have your application defined to be in <gcServer> mode the LatencyMode property cannot be changed from its default of “Batch”.

Interactive

When it’s useful:
It’s the best mode for most any UI-based application.

What it does:
This is the most balanced setting. Garbage Collection happens concurrently with your application. Most of the work is done in a separate thread and although your application will be suspended it will not happen very often and for only very short periods of time.

Defaults:
This is the default value for garbage collection on workstations. If <gcConcurrent> is left at its default value, this is the type of garbage collection that will occur.

LowLatency

When it’s useful:
It’s useful for time sensitive applications such as 3d rendering or data acquisition.

What it does:
This is the least intrusive mode the Garbage Collector can be set to. As in Interactive, garbage collections happen concurrently with your application. However, collection of older objects will only happen when memory pressure becomes high. Collection of generation 2 objects is kept to a bare minimum.

Defaults:
This is never the default setting.

Notes:
LowLatency mode was designed for short time use only during time-critical sections of your application. Leaving an application in LowLatency mode for an extended period of time will cause unused objects accumulate. Before using low latency mode, you should take a look at the guidelines laid out at the bottom of the MSDN Latency Modes page.

References
MSDN Visual Studio 2008 Developer Center – Latency Modes
MSDN .NET Framework Developer Center – GCLatencyMode Enumeration
DZone - Low-Latency GC in .NET 3.5

A .NET Assembly for Cloning Objects with Arbitrary Field Value Changes: IcManipluator

RickM — Fri, 11 Jul 2008 18:45:00 GMT

After my last post, A Safe and Asynchronous One to Many Stream Copy Through IL and Inheritance ”, I ordered a few books and spent some time playing with generating IL. Along the way I’ve developed a library which allows you to make a franken-clone of...(read more)

Choosing the Right Garbage Collector Settings for Your Application (.NET Memory Management: Part 4)

RickM — Wed, 14 May 2008 18:11:00 GMT

Tuning the garbage collector to the specific context of the particular application can significantly improve the performance of both non-threaded and multi-threaded applications. In this post I discuss the gcConcurrent and gcServer settings which allow...(read more)

Basic Memory Housekeeping (.NET Memory Management: Part 1)

RickM — Thu, 03 Apr 2008 21:25:00 GMT

This is the first in a series of posts I will be writing about managing memory in .NET. Before I move on to more complex techniques, I thought it would be good to cover the basics. Articles in This Series Part 1 - Basic Housekeeping Part 2 - Improving...(read more)

Counting Processors in .NET: The Pros and Cons of Five Different Methods

RickM — Tue, 01 Apr 2008 18:54:00 GMT

There are a great number of different ways to count the number of processors available to the .NET developer. In this post I will go over some of the more common methods and their pros and cons. The Envirionment.ProcessorCount Way Code: Environment .ProcessorCount;...(read more)