Rick Minerich's Development Wonderland : reflection

F# Discoveries This Week 08/27/2010

RickM — Sat, 28 Aug 2010 00:04:06 GMT

I’m back again with another edition of F# Discoveries this Week. The most exciting announcement this week is the new F# August 2010 CTP which brings with it the ability to build your own VS2010 F# Express Edition with Visual Studio Shell and the prospect of building F# applications on Windows Phone 7. Be sure to check out Daniel Mohl’s Windows Phone 7 template before getting started.

News and Upcoming Talks

F# (August 2010 CTP)

Announcing the F# 2.0 Standalone Tools Update

A standing invitation to F# users to "get in touch"

Don Syme lecturing at the F#unctional Londoners Meetup (9/9)

Tools and Libraries

WekaSharp: An F# wrapper for Weka (Yin Zhu)

F# and C# Win Phone List Template (Silverlight) (Daniel Mohl)

fsai Release FSharp AI library 1.10.0 (Lukáš Kouřil)

RockScroll: my new favourite Visual Studio Add-In (Kean Walmsley)

Video, Audio and Slides

Slides and samples from F# talk in Manchester (Oliver Sturm)

F# Intro Talk at NextGenUG in So’ton (Phillip Trelford)

Slides from Combinators, DSLs, HTML, and F# (Robert Pickering)

Busy .NET Developer’s Guide to F#: Basics (Ted Neward)

Technical Articles

iTunes via F# Interactive (Cameron Taggart)

F# for puzzles (Morse code decoder) (Brian McNamara)

Extracting elements from Win32 resource files (Luis Diego Fallas)

A variation on Matt Moloney's "Memento" pattern (Don Syme)

Bound Variables in F#: What Reflection Reveals (Neil Carrier)

Getting F# Running on a Continuous Integration Server (Ade Miller)

How F#'s async really works? (George Cullen)

Segment Tree in F# (Neil Carrier)

Bound Variables in F#: What Reflection Reveals (Neil Carrier)

Fuzzy Logic Experiments in F# (Neil Carrier)

Fuzzy Logic in F#, Example 1 (Neil Carrier)

Fuzzy Logic in F#, Now More Functional! (Neil Carrier)

F# “fun” with Windows Phone 7 (Fahad)

Getting started with Windows Phone in F# (Senthil Kumar)

Euler Problem 10 (Mark Pearl)

Euler Problem 16 (Mark Pearl)

Project Euler Problem 9 in F# (Stefano Ricciardi)

Project Euler Problem 8 in F# (Stefano Ricciardi)

Project Euler Problem 6 and 7 in F# (Stefano Ricciardi)

Enumerable.Skip vs Seq.skip (Mauricio Scheffer)

Getting Started and Evangelism Articles

Putting the fun in functional programming (Daniel Markham)

Back to school : Getting to know F# (Mehfuz Hossain)

Learning F# by writing C# (Alan Hemmings)

F# - sharper than C# (Josh Reuben)

Learning F# for C# developers (Bill Morrissey)

F# Community

Ars Technica - What do you think of F# ?

SO - How to merge sorted sequences in F#.

SO - Which to use, C# or F#? In this "real world" case

hubFS: F# Databse Application Samples & Articles

hubFS: MailboxProcessor.TryScan paradigm

Discoveries This Week 05/01/2009

RickM — Sun, 03 May 2009 22:58:00 GMT

A few weeks off left me with renewed vigor. This week we have some very interesting posts on reflective and reactive programming, enhancing F# by adding additional operators, IL analysis via simulation and finally, observations on the productization of F#.

Luke Hoban’s Lang.NET presentation on the productization of F#

A set of interesting observations from our experience taking F# over the last 18 months or so […] from a relatively mature research project […] into something that’s going to be a part of the next version of visual studio.

I’ve just begun to watch the Lang.NET videos. While I haven’t had a chance to watch them yet, I’ve heard from others in the community that Amanda Laucher, Jason Olson and Tomas Petricek all had fantastic presentations.

Alex P’s Thoughts on reflective programming with F#

Let's take a different approach. I'm going to bring back a piece of functionality i talked about in earlier articles - quotations, and using quotations to determine function signatures.

The most interesting idea from this post was that of stacking discriminated unions to attach meta information to parameters. This meta information then can be used when reflecting to determine the scope of parameter inputs.

Jason Kikel on Adding Ruby and C# Operators to F#

Ruby (and in many ways Perl) has a bunch of really succinct, elegant operators that contribute to the satisfaction of using the language. […] Since F# doesn’t implement many of my favorite Ruby operators, we’ll put together some implementations here.

Quite a few useful set operators are defined in this post. Many I feel should be included in the language.

Ian Voyce on IL analysis using F#

I recently needed to determine which functions were called by some of our F# code. Naively, you can use existing tools like ildasm, to disassemble a .NET DLL and then search the resulting IL source code for references. The obvious problem here though, is that you’re going to include all references whether or not they’re actually called.

I found Ian’s method of using discriminated unions to represent IL very clever. Also, the possibilities in directly using the AbstractIL API seem interesting. I may have to explore this myself.

J-C on Functional Reactive Programming in F# Parts 1, 2 and 3

A functional, reactive program is a program written in a functional language (where side effects are inexistent (sic) or at least limited to a minimum) and producing results that depend on time, in a continuous manner on one hand and on discrete events on the other hand.

I had not heard of Functional Reactive Programming previously. It seems particularly useful for time sensitive domains such as robotics or game AI.

I apologize for the fact that I’ve disabled anonymous comments. Given the amount of spam I was receiving I had no choice. If you would like to leave a comment, or think you might in the future, I encourage you to take the one time step of creating an account.

How to determine which language(s) were used to build a .NET assembly

RickM — Thu, 26 Feb 2009 11:00:00 GMT

While in most cases there is no explicit information in an assembly as to which languages it was compiled from, it is possible to make an educated guess as to which languages were used. This is due to the fact that each different .NET compiler leaves it’s own unique type of fingerprint. In this article I discuss both my methodology for finding these fingerprints and which were unique to each language I used.

Methodology

For each language I made a new class library project. I then reflected and compared each assembly to determine which unique characteristics it had. It turned out that, at least for C#, F#, VB and C++, each was uniquely identifiable by the existence, or lack thereof, of certain features.

So to break it down a bit.

In each project I added one class and one public method in each of those classes:

  1: public class CSharpClass
  2: {
  3:     public void LocalMethod() {}
  4: }

After compiling each of these projects into it’s own assembly, I referenced them from another testing project. To grab a set of features for each language, I used the following three reflection calls:

Assembly.GetTypes()
Assembly.GetCustomAttirbutes()
Module.GetFields( BindingFlags.NonPublic | BindingFlags.Static )

Then, with a simple program, I found which of these features were unique for each language. This set of unique features ultimately represents a map of the imprint each compiler leaves.

F#

A compiled F# library will only have one attribute by default:

Microsoft.FSharp.Core.FSharpInterfaceDataVersionAttribute

This made it the easiest to differentiate of all the languages I tested. Even more interesting, this attribute contains three fields which specify the specific version of the F# compiler used to generate the assembly:

Major    1    int
Minor    9    int
Release    6    int

I’m always impressed with how the F# team consistently goes above and beyond when it comes to the small details.

Visual Basic

The Visual Basic assembly I generated was also easily identifiable via extra types which were automatically added:

My.MyApplication
My.MyComputer
My.MyProject
My.MyProject+MyWebServices
My.MyProject+ThreadSafeObjectProvider`1
My.Resources.Resource
My.MySettings
My.MySettingsProperty

As you can see from this list, the existence of these types in the “My” namespace is a fairly safe indicator that the Visual Basic language was used.

C++ CLI / Managed C++

C++/CLI and Managed C++ are considered to be the same language with slightly different syntax as they share the same compiler. However, there are four different compilation modes for C++ and each has somewhat different results.

/CLR – Common Language Runtime Support
/CLR:pure – Pure Common Language Runtime Support
/CLR:safe – Safe Common Language Runtime Support
/CLR:OldSyntax – Managed C++ Syntax

The /CLR, /CLR:pure and /CLR:OldSyntax settings provide easy to classify assemblies, as they all inject an enormous number of types (70+) into the assembly. I verified that contained two types from the vc_attributes namespace:

vc_attributes.YesNoMaybe
vc_attributes.AccessType

However, /CLR:Safe is much different in that it injects no types and adds no assembly attributes by default. The generated assembly was almost completely clean. I was forced to use Reflector to determine how to differentiate this from C#.

C#

C# was one the most difficult to identify assembly type. This is due to the fact that it has no unique types and only one unique attribute:

System.Reflection.AssemblyConfigurationAttribute

Unfortunately, this attribute is defined in the AssemblyInfo.cs file and so we can’t depend on it. Up to this point it was only necessary to use two reflection calls:

Assembly.GetTypes()
Assembly.GetCustomAttirbutes()

I was hoping to keep things very simple. However, to differentance these two languages it’s necessary to go a bit further. It turns out that C++ always injects an module level field into the assembly while C# does not. And so by using:

Module.GetFields(BindingFlags.NonPublic | BindingFlags.Static)

We can check for the existence of this kind of field and so differentiate these two types.

After some investigation with reflector, I was able to find one particular feature unique to C#. Unfortunately, it requires disassembling functions and looking at the resulting IL. It seems as though a function definition never has a .maxstack of less than 8. In all other languages I observed .maxstack had been set to values as low as 0 when defined in an empty function.

However, as I am only currently concerned with these four languages, my testing on this matter has been very shallow and so pleae take it with a grain of salt.

Conclusion

I admit that my sample Assembly set was very small and my feature set very large. However, while this type of classification may not be robust enough to be applicable to a system which depended on these results being absolutely true, I’ve shown that it is in fact entirely possible to make reasonably confident guesses as to the language used to generate a .NET assembly while using only simple reflection. It would be interesting to see how well this holds for obfuscated assemblies as well as other “bare minimum” compilations generated via different combinations of compiler settings.

The next obvious step would be to extend what I have already written into a full Bayesian classifier. Would be much better than a hardcoded hierarchy which would be fragile and possibly completely and repeatedly incorrect for some cases. Another big advantage of using machine learning here, is that it would be easy to add new features and classification categories.

Discoveries This Week 01/09/2008

RickM — Fri, 09 Jan 2009 16:37:00 GMT

It’s been a very exciting week. I actually had more things to post than time would allow me to write about. I’ll have to save them for next time. Blog: Daniel Spiewak’s What is Hindley-Milner? (and why is it cool?) Hindley-Milner is the algorithm all...(read more)

The Future of Programming Languages

RickM — Wed, 15 Oct 2008 14:18:00 GMT

Yesterday, my coworker Lou Franco pointed me to a fantastic talk by Anders Hejlsberg on the future of programming languages. In this talk Anders argues that the future of programming language development will be focused on three events: the explosion in the use of Dynamic Languages, the creation of many new Domain Specific Languages and the rising need for Concurrency.

In this article I'm going to focus on Dynamic Typing and Metaprogramming in particular. It's interesting to consider what the impact on the maintainability of our code from their increased prominence might be.

Dynamic vs Static Typing

Over the last decade the use of dynamic programming languages has exploded. Working at the higher level of abstraction that dynamic languages provide allows the programmer to spend much more time building things and much less time worrying about the details. For sure, when prototyping or experimenting the speed of dynamic languages is a huge boon. However, without type checking at compile time the potential for really difficult to track down bugs is huge.

Just think about the kinds of strange bugs that can happen when any type can be passed into any argument of any method. It gets even worse when you consider that these types may be silently converting themselves. Without great care, the number of potential execution paths in your program could skyrocket.

This can be offset by really strict unit testing and extremely high code coverage. However, I have yet to see a large real world project that had this level of testing.

For these reasons I find the new C# "dynamic" keyword rather worrying. It allows the programmer to declare variables which are only type checked at runtime instead of at compile time. I would hate to have to maintain a code base where it was liberally used.

Anders also mentions F#'s static but implied typing system. He talks a bit about the two worlds of static and dynamic meeting in the middle in this kind of way. I agree, this is really the best of both worlds. You gain the "dynamic feel" but get almost none of the downsides. The security of static types without all the extra typing.

Metaprogramming

I agree that Metaprogramming is a powerful tool, especially for testing. In a ecosystem designed around objects having meaning, it only makes sense to be able to ask questions about those objects and be able to dynamically manipulate how they operate. However, I am worried that it may in some cases become a substitute for well designed architecture.

Also, because of it's nature, in many cases Metaprogramming necessitates the use of strings to represent information about the application. Strings whose contents will not be checked at compile time. Because of this it is very easy to make breaking changes to an application which won't be caught when the program is compiled.

Take for example the idea of a class with a single property. In the code for the application in which that class lives, a code branch depends on reflection to check if that property exists. If you were to rename that property that path would no longer be taken and a bug would result. If the creator of that code had used an interface instead of reflection it would have broken at compile time.

Conclusion

I'm not saying that these kinds of powerful tools shouldn't exist. They are extremely useful in many cases. We just need to be very careful in their use. As Steve Hawley likes to say, when you have a shiny new hammer everything starts to look like nails.

Edit: I want to note that I had intended to include a subsection on the direct modification of abstract syntax trees but ran out of time. Also, a good friend of mine also pointed out to me that Lisp style macros actually do let you work with the underlying code in a much more meaningful way than C++ style text manipulation.

A .NET Assembly for Cloning Objects with Arbitrary Field Value Changes: IcManipluator

RickM — Fri, 11 Jul 2008 18:45:00 GMT

After my last post, A Safe and Asynchronous One to Many Stream Copy Through IL and Inheritance ”, I ordered a few books and spent some time playing with generating IL. Along the way I’ve developed a library which allows you to make a franken-clone of...(read more)