Dan Barvitsky

Detecting SearchServer (and other services) in SharePoint farm

Jake and I just got into an interesting problem while working on search functions for VizitSP. My code was using types from Microsoft.Office.Server.Search, which is not available if SearchServer is not installed, so trying to access search functions results in “assembly cannot be loaded” exceptions. So, how do we detect the SearchServer in SharePoint?

SearchServer is part of Shared Services. If you want to make sure it is available you must at least do two things:

1. Detect if SearchServer is installed on the farm;
2. Make sure it is online

Luckily, Microsoft provides access to all farm’s services using SPFarm, SPServiceCollection and SPService classes, so that you can browse farm’s services and check their status. Try following code:

using Microsoft.SharePoint.Administration;
...
SPFarm farm = SPFarm.Local;
foreach (SPService s in farm.Services)
{
    Console.Out.Write(
    String.Format("{0}\t{1}\t{2}({3})\n",
        s.Id,s.Status,s.DisplayName,s.TypeName));
}

This code lists all available services, their IDs and display names. It turns out we are looking for “OSearch”.

SPServiceCollection provides GetValue method, which looks up service by GUID or name. Unfortunately, GUIDs appear to be system-dependent, and lookup by name does not seem to work (we tried Name, TypeName, DisplayName – nothing worked). As result, we had to loop through all SharePoint services and check either Name property or actual service type:

const String TypeName = @"Microsoft.Office.Server.Search.Administration.SearchService";
const String ServiceName = @"OSearch";
...
foreach( SPService s in farm.Services ) {
    if (s.GetType().FullName == TypeName || s.Name == ServiceName ) {
        return s.Status == SPObjectStatus.Online;
    }
}
return false;

We do not use typeof(…) because that will prevent our assembly from loading if SearchServer is not installed. We also do not rely on service name alone or type name alone, since either might change in future. For other services, like Excel Calculation Services or Forms Service, the name may actually be empty, so you will have to go by type alone.

Steve’s wisdom

Just got off the 5-why meeting following the VizitSP 1.6 release. Besides usual 5-why stuff dotImage’s senior architect Steve Hawley shared some interesting thoughts on estimates and the arguing in general. I sort of instinctively wrote that down. So here it is. Smart things belong to Steve everything else is my comment.

Top problems people should consider while doing estimates

Suppose you are estimating a feature and came up with say 20 hours. Now it is time to multiply it by some magical constant. Some people multiply by PI, some multiply by e, some multiply by (PI+e)/2, some come up with some formulae. Here are some driving factors which should increase your estimate:

1. Unusual hardware or complex environment (what you build for): the environment where the product is going to be deployed is complex, non-standardized or volatile. This will increase number for test cases you will have to carry out, which results in more time. I’d probably go by number of typical configurations you are going to test.
2. Novel problems (things nobody did before): you are taking a step into unknown, you may face unpredicted obstacles, maybe even Cthulhu himself.
3. Other people's code (things you don't have control over): could actually be worse than Chtulhu.
4. Complex infrastructure (things you use to build your product): you need to bring your test rigs and build infrastructure up-to-speed with your codebase, if you mess them up, you are dead in the water till you fix them.
5. Poor communication (how you talk to each other): if you communicate everything to anybody this is considerable overhead. If you limit informational wave to certain audience some people may fell off the transmission. Either way it is good to have it written somewhere like wiki.
6. Many assumptions (e.g. of successful integration testing or undocumented features): you should question-and-proof most assumptions you make, which adds time

Five ways how people argue

Estimations usually have some sort of negotiations/arguing involved. In general there are only five ways people can argue

1. Logic (because a or not a is true)
2. Experience (I've done this before, I know)
3. Emotion (because I will stab you)
4. Evidence (look here, see how this works?)
5. Authority (because customer/boss/book says so)

In engineering I’d probably consider (1), (2) and (4) to be positive and (3),(5) to be negative. When in argument, both parties should be in the same mode to be able to resolve an issue. You take on logic with logic, on experience with experience and on armor with armor. Therefore, when you counterpart picks a mode you should either switch to it or drag them into the mode you want. Remember that the goal is not to win the debate, but to come up with a solution. It is always wise to keep conversation creative and friendly, or you may spiral into (3).

For example, suppose there is a negotiation around “the big three options” AKA “the holy trinity of software planning”: we sprint is overstuffed, we can either (a) lose a feature, (b) accept lower quality or (c) extend deadline (which is equivalent of killing sprint and starting over in scrum). X, the product owner, wants (b) and Y, the architect, wants (c). The wrong way would be for Y to say “lower quality means more problems in future, I do not recommend that”. It is not friendly, since Y is engaging in direct fight with X by using (2) and (3). The right way would be “suppose we accept lower quality, how should we prepare tech support for that?”. This makes X think of unpleasant consequences of their decision, which might be acceptable workaround in certain situations. Y shoud try to ask “if (a), (b) or (c) acceptable in current situation” rather than forcing their opinion on opponents.

Upd: I totally messed up the numbers and references. Thanks Elaine!

A word of praise for relevant error messages

What do you think about the following code:

            byte[] privateParams, publicParams;
            using (var rsa = new System.Security.Cryptography.RSACryptoServiceProvider())
            {
                privateParams = rsa.ExportCspBlob(true);
                publicParams = rsa.ExportCspBlob(false);
            }
            using (var rsa = new System.Security.Cryptography.RSACryptoServiceProvider())
            {
                rsa.ImportCspBlob(privateParams);
            }

Should work, right? Sure. Copy-paste, make a unit test and check for yourself – this works as a charm on your local machine. However, shall you run it under SharePoint it does this:

Access is denied.
   at System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr)
   at System.Security.Cryptography.Utils._CreateCSP(CspParameters param, Boolean randomKeyContainer, SafeProvHandle& hProv)
   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.ImportCspBlobHelper(CspAlgorithmType keyType, Byte[] keyBlob, Boolean publicOnly, CspParameters& parameters, Boolean randomKeyContainer, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.ImportCspBlob(Byte[] keyBlob)
...

After some noodling in Google I found this post about .NET implementation adding some bytes to the CSP blob, as well as this CodeProject post, which does the same trick to initialize CSP from blob.

What happened? A good question. First, I thought that .NET code propagated some low-level exception, which, may I suggest, has been constructed from a result code, which, perhaps, was  not too relevant to the problem itself. I applied the “rip those 12 bytes off” fix and it did not work. So I went to MSDN and finally noticed that ImportCspBlob requires KeyContainerPermissionAccessEntryCollection permission. Pardon me? Digging more in MSDN reveals the following details:

1. I am loading a public key. .NET code for RSACryptoServiceProvider wraps CSP implementation of RSA.
2. CSP wants keys in server’s key container to work.
3. When you “import” the public key from Blob, .NET class attempts to drop it in the server’s key container.
4. In order to access server’s key container assembly needs the permission I mentioned above.
5. Local code (unit tests) have that permissions, so everything runs just fine.
6. SharePoint code does not have that permission, so this is where “Access is Denied” is coming from.

Now it makes sense. Apparently this problem is rather popular. I could save myself some 5 hours if the error message was something along the lines “Cannot add key to server key container, because this assembly does not have … permission to access server key store”.

Note to self: when dealing with exceptions, don’t hesitate to create individual exception classes for particular types of exceptions. Create a generic exception class, make it abstract and throw concrete implementations. If you are dealing with and underlying layer (native code, COM, web service), which does not provide informative exceptions, don’t just wrap-and-throw. Try using smart exceptions (pattern?) instead. First, create a base exception class (normally one per library or subsystem):

        public abstract class MyLibException : Exception
        {
            ...
        }

Now if something bad happens you cannot just throw MyLibException, it is abstract. You will have to create a concrete exception class for that particular type of problem:

        public class ConcreteException: MyLibException
        {
            public ConcreteException( Exception cause ) 
            {
            }
        }

Note that I do not propagate “message” parameter – concrete exceptions should come up with their own descriptions as necessary. Well, unless there is an external information, such as message, error code, parameter reference, etc, which has to be included in the exception message. This is an indication of the fact that you need to refactor your concrete exception into “smart” exception:

        public class SmartException: MyLibException
        {
            public SmartException( int resultCode, String responseText, Exception cause ) {
                // Come up with reasonable diagnostics message here
            }
        }

Now your smart exception can come up with an informative message on its own. With a bit of effort, you can make smart exceptions localized  and still be able to identify exceptions by stack trace (because of the class names).

Strengths of smart exception pattern:

1. If you do it right (i.e. thinking of support engineers, third party developers, consultants, IT professionals, etc. who will have to figure out the cause of problem by your exception), you are making your product easier to troubleshoot.
2. Exceptions can be localized and still be identified by class names.
3. Documentation. Long ago, when trees were short and mountains were young, software engineers would have to document error codes, because it was the only way to troubleshoot applications besides of reading logs.  Now we have comment annotations for “throws”, but, let us face it, how many people really provide detailed error messages? And even if they do it all stays in the code documentation. Now you can write code documentation for your concrete exception class, which can be added to the troubleshooting guide. Because, unlike method documentation, it does not expose your code structure. Automatically.
4. Your diagnostics logic can be broken apart from your “main” code.
5. Easier to write unite tests.

Weaknesses of smart exception pattern:

1. More code to write
2. Not too clear when exception classes can be re-used.

ILMerge and ConfigurationSection

Note to self: dearest Dan! If you happen to use ILMerge with an exe file, please make sure you check the app.config file. Why?  Well, suppose you have a custom settings section in your app.config, just like this:

  <configSections>
    <sectionGroup name="Atalasoft">
		<!-- Converter service settings section: -->
		<section 
			name="TransformServer" 
			type="Atalasoft.OO.Service.Settings, Atalasoft.OO.Service" 
			allowLocation="true" 
			allowDefinition="Everywhere"/>
    </sectionGroup>
  </configSections>

As you might notice, the “type” parameter references Atalasoft.OO.Service.Settings class, which extends System.Configuration.ConfigurationSection by introducing custom configuration properties for your WCF service. The “type” parameter consists of two parts – the type name and assembly name. After you ILMerge your service into a solid executable, guess what happens with Atalasoft.OO.Service assembly? Yes, Dan, it is gone! Which means that:

1. all your unit tests will pass as usual,
2. your debug (non-merged) service will run just fine,
3. but your newly installed windows service will start up to the point where it needs to read configuration file and then will fail with a random exception.

Which is such a bucket of joy to debug, as you just have experienced.

To work this around you need to update the app.config and change the name of assembly:

  <configSections>
    <sectionGroup name="Atalasoft">
		<!-- Converter service settings section: -->
		<section 
			name="TransformServer" 
			type="Atalasoft.OO.Service.Settings, Atalasoft.Transform.SelfHostedService" 			
			allowDefinition="Everywhere"/>
    </sectionGroup>
  </configSections>

OK, now you wonder where did Atalasoft.Transform.SelfHostedService come from? This is the name of the primary assembly you used for ILMerge. No “exe” at the end, please.

And don’t forget the obfuscation.

Which formats does OpenOffice support?

What can you extract from the following dialogue between Prospect and Employee?

P: Which formats does <application name> support?
E: Which one are you interested in?
P: Err… Give me all you have!
E: We support many formats, including… which one you were interested in again?

One can, with certain level of confidence, say that:

1. Prospect does not fully know/understand what he/she wants;
2. Employee does not remember from top of his/her head which formats their product supports and does not have a list handy;

If you imagine that employer itself relies on 3rd party components, you may very well end up with #3: employer is not 100% sure which formats are supported by components they use.

It is not a huge secret in ECM world that OpenOffice supports huge variety of formats and has more-or-less appealing API for converting documents; so some people use it to “convert the unconvertible” under the hood of their applications.

Since we are looking into OpenOffice support as well, here is a big question: which formats does the OpenOffice support?

OpenOffice has extendible architecture, so it is hard to say which formats are there. You sure can use the “Open” file command, but, here is a dirty secret, not all stuff OpenOffice can support really is there. Since googling did not give a good answer, so I decided to ask the source. The source code.

Luckily, OpenOffice.org has a common mechanism for registering filters. Namely there is a bunch of XCU files in filter/source/config/fragments/filters. I figured I could use the filter definitions to extract the list of formats I need. So I wrote a simple extraction tool and here is the list I’ve got:

The tool and result files (TXT,OpenOffice and Excel) can be downloaded here.

Needless to say, this list will change from release to release. In order to generate it you will need to download the source code from OpenOffice.org source code download page or directly from their SVN (browse). You will minimally need the filter/source/config/fragments/filters folder.

Bitten by Kerberos

The Kerberos, or Cerberus in Latin, a multithreaded multi-headed dog with tail of serpent heads, the hellhound, who used to guard the gates of Hades behind the river of Styx to keep the dead ancient Greek people dead in the kingdom of Shadows.

Once Kerberos was captured by Hercules and then brought to Eurystheus, where his DNA has been extracted and printed on an amphora. Somewhere around 1980 the amphora got into the hands of MIT researchers, who deciphered the inscriptions and reverse-engineered them into an authentication protocol. There is a legend, that time-to-time spirit of Cerberus comes to life, raises from the informational space and tortures the developers and architects, who did not pay enough attention to his persona back in the college.

To my greater sorrow, I can hereby confirm the above legend and provide the links for those in brave spirit who dare to repeat my path:

• Kerberos Technical Supplement for Windows – start here
• Configuring Kerberos for SharePoint 2007 – this is how I got into it
• Kerberos authentication and troubleshooting delegation issues – read this next for clarification on different names and short tool reference
• Overview of setspn command – a very useful tool to deal with Kerberos config, part of Windows Server 2003 Resource Kit tools
• EventID.NET to help troubleshoot Kerberos errors in Windows log

Here are some tricks to ease the pain of the Kerberos bite:

• To see what the hell is going on in your Active Directory, use the ldifde -f export.txt command. It will save entries of your AD into a semi-readable file. To narrow down the search you may use various options, such as “–d” to specify the root of directory or “-r” to filter the entries;
• Use “klist tickets” or kerbtray tool to display current Kerberos tickets on your desktop (or server). Note that tool only shows tickets for the desktop it runs on. It is located in C:\Program Files\Windows Resource Kits\Tools directory, if you need to copy it;

Joining SharePoint server to a different domain

Once upon a time I had to join a workgroup-based SharePoint test server to a domain. SharePoint wonderland is like 4th dimension – it is full of unexpected and no task is trivial for naive 3D creature. Jake and Dave referred me to a magical “15-step” process to move SharePoint to different domain, but Google’s links are dead and so are my hopes for quick-and-easy migration. So far I came up with “Restoring a SharePoint Site collection to a new domain” and “Join SharePoint server to the Active Directory domain” posts, which I am trying to follow. Basically it is like if you want to sign your home to your spouse, you both should move everything out, blast the house, have your spouse hire a contractor, re-build it from scratch and move everything back in. I sort of missing old good times when one could just edit bunch of files in /etc/ to get the job done…

My solution for unit testing concurrent code in NUnit

I blogged recently about testing parallel code in NUnit. I think I settled with requirements:

1. I want to use threads;
2. I want to use assertions in threads;
3. I want to control how threads step on each other;

Too bad that solutions I tested did not fit 100%. So I came up with a small (200 lines of code) tool called “Sync” that meets my requirements. My test looks as follows:

        [Test]
        public void MonitorTest()
        {
            using (var sync = new Sync())
            {
                List<String> result = new List<String>();
                sync
                    .Add("A", "one")
                    .Add("B", "one")
                    .Add("B", "two")
                    .Add("A", "two");
                ThreadStart main = delegate()
                {
                    result.Add(Thread.CurrentThread.Name + ": before 'one'");
                    sync.Point("one");
                    result.Add(Thread.CurrentThread.Name + ": after  'one'");
                    result.Add(Thread.CurrentThread.Name + ": before 'two'");
                    sync.Point("two");
                    result.Add(Thread.CurrentThread.Name + ": after  'two'");
                };
                sync.Start("A", main);
                sync.Start("B", main);
                
                sync.AssertEnd();
                int n1 = result.IndexOf("A: after  'one'");
                int n2 = result.IndexOf("B: after  'one'");
                Assert.Greater(n1, 0);
                Assert.Greater(n2, 0);
                Assert.Greater(n2, n1);
                Sync.AssertOrder(result, 
                    "A: after  'one'",
                    "B: after  'two'",
                    "B: after  'one'",
                    "A: after  'two'");
                foreach (String s in result) Console.Out.WriteLine(s);
            }
        }

The tool does not care how you run the code – it wraps delegates and propagates exceptions when you call sync.AssertEnd(). It is entirely up to you how to call the code – separate thread, external queue, asynchronous calls – does not matter.

You can (optionally) provide synchronization points (similar to “barriers” in PUnit). Here is how it is done. You insert the synchronization points (unfortunately this is done in code , I wish I could modify assembly on the fly):

sync.Point("one");

Then you configure the Sync object:

sync.Add("A", "one").Add("B", "one")

It defines in which order threads “A” and “B” (these are actual thread names) pass the synchronization point. If thread “B” gets there first, it will wait till thread “A” gets there. This way you can control how race conditions happen by inserting synchronization point prior to it.

All exceptions are thrown when you either call sync.AssertEnd() or when sync object is disposed (this is why you want to put it into the “using” clause). When you call AssertEnd() or Wait(), sync object waits for all wrapped code to complete, so that test won’t exit before all of it is code is done.

Ironically, I am not 100% sure the synchronization code is flawless, though. My concern is that synchronization points can potentially prevent race conditions on different machines/architecture. The situation where you resume two threads simultaneously near the race condition section is very unpredictable. Will they race? Who will get there first? You can’t tell.

As an exercise, I am going to look into Typemock Racer tool and see if my tests are comparable to what it does.

The Sync source code can be downloaded here.

Techniques for testing concurrent code in NUnit

Last time I blogged about a way to make concurrent unit tests deterministic. This time I’d like to focus on the technical aspects, namely NUnit. Let us start with the following test:

        [Test]
        public void TestThread()
        {
            int n = 0;
            var thread1 = new Thread(() =>
            {
                n = 10;
                Assert.Fail("This should fail!");
                n = 20;
            });
            thread1.Name = "Broken thread";
            thread1.Start();
            thread1.Join();
            Assert.AreEqual(10, n);
        }

Can you tell what will happen when you run it? When you start a thread, it will execute lambda function code,  assign 10 to n, and then… Fail? Nope. Assert.Fail(“This should fail”) will only throw an exception. Who is going to catch it? How NUnit can relate that exception to the unit tests running in another thread or, strictly speaking, not running any longer (in case you forgot to join() to it)? So the correct answer is test will print some warnings and will eventually pass:

Thread Name: Broken thread
NUnit.Framework.AssertionException: This should fail!
   at NUnit.Framework.Assert.Fail(String message, Object[] args)
   at NUnit.Framework.Assert.Fail(String message)
	...
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
1 passed, 0 failed, 0 skipped, took 0.89 seconds (NUnit 2.5).

Any solutions? Sure:

1. Use PNUnit, available as part of NUnit as of 2.5.0.9117. It allows you to run parallel (and even distributed!) tests, while capturing exceptions properly. The downside of this approach is that you cannot really use threads, instead you have to break your test into separate tests and make them synchronize to each other using PNUnit services. I am not a huge fan of this approach, since it introduces dependencies between tests and the whole rig becomes a bit more complex.
2. Use CrossThreadTestRunner by Peter Provost. It simply starts a delegate in a thread, waits for the thread to complete, captures any exceptions thrown by the thread and then uses a neat trick to re-throw an exception in unit test’s primary thread. Here is an example:

           [Test]
           public void TestThread()
           {
               int n = 0;
               CrossThreadTestRunner c = new CrossThreadTestRunner(()=>
               {
                   n = 10;
                   Assert.Fail("This should fail!");
                   n = 20;
               });
               c.Run();
               Assert.AreEqual(10, n);
           }

3. Use Mike Peretz’s solution. Mike suggests using asynchronous delegate calls instead of threads to test parallel code. The example:
   

           [Test]
           public void TestThread()
           {
               int n = 0;
               ThreadStart main = () => 
               {
                   n = 10;
                   Assert.Fail("This should fail!");
                   n = 20;
               };
               var asyncResult = main.BeginInvoke(null, null);
               main.EndInvoke(asyncResult);
               Assert.AreEqual(10, n);
           }

In both cases results are the same – test fails where it should fail, NUnit produces reasonable stack trace. In case of CrossThreadTestRunner the stack is more readable, though.

Back to the subject, CrossThreadTestRunner blocks the test’s execution so I cannot use it for parallel code testing. Mike Peretz’s way looks more appealing for testing parallel processes. Unfortunately it comes with a catch: .NET actually uses the thread pool behind the scenes when invoking delegates asynchronously, so it is not guaranteed to run threads in parallel. Here is a sample code to confirm this concern:

        public delegate void T(int n);
        [Test]
        public void TestMany()
        {
            T main = (n) =>
            {
                Console.Out.WriteLine("Thread " + n + " starting");
                Thread.Sleep(1000);
                Console.Out.WriteLine("Thread " + n + " about to fail");
                Assert.Fail("This should fail!");
                Console.Out.WriteLine("Thread " + n + ": you should never see this");
            };
            IAsyncResult[] results = new IAsyncResult[100];
            for (int i = 0; i < 100; i++)
            {
                results[i] = main.BeginInvoke(i,null, null);
            }
            Thread.Sleep(1000);
            for (int i = 0; i < 100; i++)
            {
                main.EndInvoke(results[i]);
            }
        }

Here I start 100 concurrent delegates, each prints first line, waits 1 second, prints the second line and fails. After starting all delegates I wait for 1 second and end the invocations. Can you tell how many threads will run? If you said 100 you are wrong, here is the test output:

Thread 0 starting
Thread 2 starting
Thread 3 starting
Thread 1 starting
Thread 4 starting
Thread 5 starting
Thread 3 about to fail
Thread 0 about to fail
Thread 2 about to fail
Thread 1 about to fail
Thread 6 starting
Thread 7 starting
Thread 8 starting
Thread 9 starting
TestCase 'TestMany' failed: This should fail!
	
	Server stack trace: 
	...
0 passed, 1 failed, 0 skipped, took 1.91 seconds (NUnit 2.5).

Reading Thread.CurrentThread.IsThreadPoolThread inside the delegate returns true, so I’d suggest we are running 6 threads at a time here. This is not good for parallel code testing, since it can eliminate race conditions. In practice you barely run 3+ threads scenarios, but load tests may actually use many threads. Still, this prevents tests to run in predictable manner.

Another sad side effect of this technique: your test will only fail if you call EndInvoke for all delegates you started. If you forget to do it, you may end up in a situation where NUnit thread already finished executing the test, but some asynchronous delegate calls are still sitting in the queue.

So it looks like I need a mixture of both approaches, but this is a topic for a separate post.

Thinking aloud: concurrent Unit tests

While working on conversion service for VizitSP, I run once again into unit testing the concurrent code. The problem is not new, yet I did not see 100% solution yet. The difficulty is coming from the fact that unit tests are deterministic to the bone (i.e. they rely on the fact, that code behaves the same way every time you run it). Concurrent scenarios, however, can behave differently depending on when particular thread gained access to shared resource.

Suppose you have something like this:

        public class HairyBug
        {
            private List<String> messages = new List<String>();
            public void PostMessage(String message)
            {
                this.messages.Add(String.Format("{0}: {1}",messages.Count,message));
            }
            public String[] GetMessages() { return messages.ToArray<String>(); }
        }

Can you spot a bug? The problem is two threads can read the messages count simultaneously prior to adding messages to the array:

Let us do a unit test:

        [Test]
        public void FindHairyBug()
        {
            var bug = new HairyBug();
            
            Thread[] threads = new Thread[10];
            for (int i = 0; i < threads.Length; i++)
            {
                threads[i] = new System.Threading.Thread(()=>
                {
                    bug.PostMessage("Passed");
                });
                //threads[i].Priority = i % 2 == 0 ? ThreadPriority.AboveNormal : ThreadPriority.BelowNormal;
                threads[i].Name = "Thread "+i.ToString();
                threads[i].Start();
            }
            for (var i = 0; i < threads.Length; i++) threads[i].Join();
            foreach (String n in bug.GetMessages()) Console.Out.WriteLine(n);
            Assert.AreEqual(new String[]{
                "0: Passed", "1: Passed", "2: Passed", "3: Passed", "4: Passed", 
                "5: Passed", "6: Passed", "7: Passed", "8: Passed","9: Passed"},
            bug.GetMessages());
        }

Is it a good test? Well, let us see:

1. Is it deterministic? Yes.
2. Does it depend on other tests? No.
3. Does it check the result? Yes.
4. Does it maintain the state between executions? No.

When I run this test it passes on my laptop. Why? Because it is fairly old and has single CPU. Once I move it to my desktop (TADA) it still passes inconsistently. If I uncomment priority setting line it messes everything up badly. It turns out threading introduces some randomization which makes test non-deterministic. Bad test. We’d better fix it:

        [Test]
        public void FindHairyBug2()
        {
            var bug = new HairyBug();
            Thread[] threads = new Thread[10];
            for (int i = 0; i < threads.Length; i++)
            {
                threads[i] = new System.Threading.Thread(() =>
                {
                    bug.PostMessage("Passed");
                });
                //threads[i].Priority = i % 2 == 0 ? ThreadPriority.AboveNormal : ThreadPriority.BelowNormal;
                threads[i].Name = "Thread " + i.ToString();
            }
            for (var i = 0; i < threads.Length; i++)
            {
                // Make sure threads do not step on each other:
                threads[i].Start();
                threads[i].Join();
            }
            foreach (String n in bug.GetMessages()) Console.Out.WriteLine(n);
            Assert.AreEqual(new String[]{
                "0: Passed", "1: Passed", "2: Passed", "3: Passed", "4: Passed", 
                "5: Passed", "6: Passed", "7: Passed", "8: Passed","9: Passed"},
            bug.GetMessages());
        }

Now Unit test is good (from unit testing prospective), but it does not catch the bug, because the code is, well,  not concurrent anymore. Threads should step on each other, but they must do it in consistent matter. How do you do this exactly? Perhaps one could add some synchronization points in code, like this:

        public class HairyBug
        {
            private List<String> messages = new List<String>();
            public void PostMessage(String message)
            {
                Sync.point("getting count");
                int count = messages.Count;
                Sync.point("adding message");
                this.messages.Add(count,messages.Count,message));
            }
            public String[] GetMessages() { return messages.ToArray<String>(); }
        }

Then you could do something like this in unit tests:

            Sync.Scenario
                .Clear()
                .Add("Thread 1", "getting count")
                .Add("Thread 2", "getting count")
                .Add("Thread 1", "adding message")
                .Add("Thread 2", "adding message");
            FindHairyBug();

So when code enters the Sync.point(“getting count”) method, it checks the current thread name and synchronization point’s name. Then it passes those two parameters to Sync.Scenario object, which decides should the thread proceed or not, waits for notification from Sync.Scenario and returns once notification is received. Now threads step on each other in semi-predictable manner, and you can specify how exactly that should happen before running a unit test.

The payback, however, is horrible: you have to embed instrumentization into your code. Moreover, stuffing your code with dozens of synchronization points requires thousands of tests to cover possible combinations. This is not cool. Any other ideas?

Resources worth looking into:

• TypeMock racer - (thanks, Lou): http://www.typemock.com/learn_about_typemock_racer.php
• ConTest – IMHO correct but very complex approach (Java): http://www.ibm.com/developerworks/java/library/j-contest.html
• Advanced Unit Tests, Part V – two, IMHO, semi-usable patterns for concurrent testing: http://www.codeproject.com/KB/architecture/autp5.aspx

Coming up next: technical difficulties specific to NUnit.

STSADMIN tasks in NAnt

A Unix admin would spend 2 hours to create a script that does in 5 minutes what a user would do in 1 hour.

Recently I ran across Peel me a grape’s wonderful NAnt Macrodef and Remote NAnt posts, and it really moved me to extend my user build scripts with automatic deployment. Indeed, wouldn’t it be neat it my solution could be deployed to the test server automatically after I build it in Visual Studio? So I came up with several NAnt macros for the STSADMIN deployments, namely:

• stsadm command wrapper
• sp-reset, which either re-sets application pool or entire IIS, depending on the arguments
• sp-clean, which removes the solution completely
• sp-deploy, which does the full-blown deployment of Vizit’s WSP
• and my favorite one sp-refresh, which registers new version of assemblies, synchronizes the files and restarts the IIS server/application pool.

I packaged all definitions into stsadm-include.xml, which is copied over to the test server and executed via remote NAnt hack. I’m not 100% happy with this solution, really. I think my remote NAnt implementation could be better. It needs to add pick up properties on caller side instead of using hardcoded values in the script… Anyways, here are some NAnt snippets:

This one is fairly straightforward – a wrapper for STSADM command. I’d rather call STSADM programmatically, but this will work for me too.

	<!-- Invoke STSADM command -->
	<macrodef name="stsadm">
		<attributes>
			<attribute name="action"/>
			<attribute name="failonerror" default="true"/>
			<attribute name="arg1" property="stsadm_arg1" default=""/>
			<attribute name="arg2" property="stsadm_arg2" default=""/>
			<attribute name="arg3" property="stsadm_arg3" default=""/>
			<attribute name="arg4" property="stsadm_arg4" default=""/>
			<attribute name="arg5" property="stsadm_arg5" default=""/>
			<attribute name="arg6" property="stsadm_arg6" default=""/>
		</attributes>
		<sequential>
			<exec program="${Target.Stsadm.Path}" failonerror="${failonerror}">
				<arg value="-o"/>
				<arg value="${action}"/>
				<arg value="${stsadm_arg1}"/>
				<arg value="${stsadm_arg2}"/>
				<arg value="${stsadm_arg3}"/>
				<arg value="${stsadm_arg4}"/>
				<arg value="${stsadm_arg5}"/>
				<arg value="${stsadm_arg6}"/>
			</exec>
		</sequential>
	</macrodef>

This one is a bit trickier. It resets either application pool or whole IIS:

	<!-- Resets IIS application pool or entire IIS server -->
	<macrodef name="sp-reset">
		<attributes>
			<attribute name="application-pool" property="application-pool"/>
		</attributes>
		<sequential>
				<if test="${not property::exists('application-pool')}">
					<log msg="Recycling IIS..."/>
					<exec program="iisreset"/>
				</if>
				<if test="${property::exists('application-pool')}">
					<log msg="Recycling application pool ${application-pool}..."/>
					<exec program="cscript.exe">
						<arg value="//B"/>
						<arg value="C:\Windows\System32\iisapp.vbs"/>
						<arg value="/a"/>
						<arg value="${application-pool}"/>
						<arg value="/r"/>
					</exec>
				</if>
		</sequential>
	</macrodef>

These two are a bit more complex. They do deployment and clean up for given solution file.

	<!-- Clean up SharePoint -->
	<macrodef name="sp-clean">
		<attributes>
			<attribute name="url"/>
			<attribute name="admin-url"/>
			<attribute name="solution" />
			<attribute name="feature" />
		</attributes>
		<sequential>
			<if test="${property::exists('feature') and property::exists('url')}">
				<log msg="Deactivating feature ${feature} on ${url}"/>
				<stsadm action="deactivatefeature"
					arg1="-name"
					arg2="${feature}"
					arg3="-url"
					arg4="${url}"
					arg5="-force"
					failonerror="false"/>
			</if>
			<if test="${property::exists('feature') and property::exists('admin-url')}">
				<log msg="Deactivating feature ${feature} on ${admin-url}"/>
				<stsadm action="deactivatefeature"
					arg1="-name"
					arg2="${feature}"
					arg3="-url"
					arg4="${admin-url}"
					arg5="-force"
					failonerror="false"/>
			</if>
			<if test="${property::exists('solution') and property::exists('url')}">
				<log msg="Retracting solution ${solution} from ${url}"/>
				<stsadm action="retractsolution"
					arg1="-name"
					arg2="${solution}"
					arg3="-url"
					arg4="${url}"
					arg5="-immediate"
					failonerror="false"/>
					<log msg="Executing service admin jobs"/>
					<stsadm action="execadmsvcjobs" failonerror="false"/>
			</if>
			<if test="${property::exists('solution') and property::exists('admin-url')}">
				<log msg="Retracting solution ${solution} from ${admin-url}"/>
				<stsadm action="retractsolution"
					arg1="-name"
					arg2="${solution}"
					arg3="-url"
					arg4="${admin-url}"
					arg5="-immediate"
					failonerror="false"/>			
					<log msg="Executing service admin jobs"/>
					<stsadm action="execadmsvcjobs" failonerror="false"/>
			</if>			
			<if test="${property::exists('solution')}">
				<log msg="Deleting solution ${solution}"/>
				<stsadm action="deletesolution"
					arg1="-name"
					arg2="${solution}"
					arg3="-override"
					failonerror="false"/>
			</if>
			<log msg="Cleanup completed. List of solutions:"/>			
			<stsadm action="enumsolutions" failonerror="false"/>
		</sequential>
	</macrodef>
	<!-- Deploy solution to given URLs -->
	<macrodef name="sp-deploy">
		<attributes>
			<attribute name="solution-file"/>
			<attribute name="url"/>
			<attribute name="admin-url"/>
			<attribute name="feature"/>
		</attributes>
		<sequential>
			<fail message="Build solution file ${solution-file} does not exist"
				if="${not file::exists(solution-file)}"/>		
			<property name="solution" value="${path::get-file-name(solution-file)}"/>
			<log msg="Adding solution ${solution} to server"/>
			<stsadm action="addsolution" arg1="-filename" arg2="${solution-file}"/>
			<if test="${property::exists('url')}">
				<log msg="Deploying solution ${solution} to ${url}"/>
				<stsadm		
					action="deploysolution"
					arg1="-name" arg2="${solution}" arg3="-immediate" arg4="-allowGacDeployment" 
					arg5="-url" arg6="${url}"/>
				<log msg="Executing administrative jobs..."/>
				<stsadm action="execadmsvcjobs"/>
				<log msg="Activating Vizit feature on ${url}"/>
				<stsadm action="activatefeature" arg1="-name" arg2="${feature}" arg3="-url" arg4="${url}"/>
			</if>
			<if test="${property::exists('admin-url')}">
				<log msg="Deploying solution ${solution} to ${admin-url}"/>
				<stsadm		
					action="deploysolution"
					arg1="-name" arg2="${solution}" arg3="-immediate" arg4="-allowGacDeployment" 
					arg5="-url" arg6="${admin-url}"/>
				<log msg="Executing administrative jobs..."/>
				<stsadm action="execadmsvcjobs"/>
				<log msg="Activating Vizit feature on ${admin-url}"/>
				<stsadm action="activatefeature" arg1="-name" arg2="${feature}" arg3="-url" arg4="${admin-url}"/>
			</if>
			<log msg="Deployment completed:"/>
			<stsadm action="enumsolutions"/>		
		</sequential>	
	</macrodef>

I wonder if anybody did a fully functional STSADM wrapper in NAnt… In particular some post-install checks, such as enumerating solutions, would be very nice. In case someone is interested in the build scripts, please drop me an e-mail.

Deploying debug assemblies in the GAC

Don’t ask me why, but at some point I ended up with the necessity of having debug assemblies in the GAC. OK, if you still ask, it is about SharePoint remote debugging. After dragging-and-dropping PDBs and DLLs for 651th time I almost started feeling an imprint in my desk surface, which mouse carved out while travelling back and forth. Sounds like it is time for another tool in my toolbox.

Technically C:\Windows\Assembly is just a folder:

C:\WINDOWS\assembly>dir
 Volume in drive C has no label.
 Volume Serial Number is 4C18-31DE
 Directory of C:\WINDOWS\assembly
02/06/2009  10:32 AM    <DIR>          GAC
03/30/2009  10:32 AM    <DIR>          GAC_32
03/30/2009  10:32 AM    <DIR>          GAC_MSIL
01/05/2009  05:43 PM    <DIR>          NativeImages1_v1.1.4322
03/19/2009  03:28 AM    <DIR>          NativeImages_v2.0.50727_32
03/30/2009  10:32 AM    <DIR>          temp
03/30/2009  06:16 PM    <DIR>          tmp
               0 File(s)              0 bytes
               7 Dir(s)  34,519,719,936 bytes free

Assembly DLLs are located in folders. Each assembly has a separate folder for every version:

 Directory of C:\WINDOWS\assembly\GAC_MSIL\System.Xml\2.0.0.0__b77a5c561934e089
01/06/2009  01:01 PM    <DIR>          .
01/06/2009  01:01 PM    <DIR>          ..
01/06/2009  01:01 PM         2,068,480 System.XML.dll

Apparently, dropping a PDB file next to the assembly DLL picks up the debugging symbols just fine! Even better, there is a very simple way to deploy debug assemblies in GAC programmatically. Here is how you deploy an assembly:

string dll_path = ...;
var publish = new System.EnterpriseServices.Internal.Publish();
publish.GacInstall(dll_path);

Here is how you deploy PDB file:

// This is the path to PDB file:
String source_pdb = ...;
// Getting a DLL path for assembly:
AssemblyName name = Assembly.LoadFile(assembly).GetName();
String dll_path = name.CodeBase.TrimStart(@"file:///".ToCharArray());
// In GAC PDB files are sitting next to DLLs:
String target_pdb = System.IO.Path.Combine(System.IO.Path.GetDirectoryName(dll_path),
    System.IO.Path.GetFileNameWithoutExtension(dll_path) + ".pdb");
// Delete PDB file if it is there:
if (System.IO.File.Exists(target_pdb)) System.IO.File.Delete(target_pdb);
// Copy PDB over
if (System.IO.File.Exists(database)) System.IO.File.Copy(source_pdb, target_pdb)

It is a bit hacky to figure out the actual DLL path, though, but it guarantees your PDBs get copied in the right folder.

Based on the code above I wrote NAnt task and a command line tool, which can be downloaded here. Now my build script picks up DLLs and registers them on the SharePoint server automatically, so whenever I attach to W3P process, it always picks correct PDB files. Well, almost ;-)

Once again about remote debugging

This is a follow-up article for Jacob Lauzier’s Cross-Domain remote debugging. Here I will discuss one particular scenario, which, in my mind, should be extremely popular in virtualized environments but, for some reason, is least straight forward to setup.

My development desktop is a Windows XP with Visual Studio 2008 on it. I debug against several virtual machines, my typical setup is Windows 2003 R2 server with WSS on it. Virtual machine obviously is NOT on the domain, because I want to manage my own VMs myself; my desktop obviously IS on the domain, because I use TFS and all other cool things available at Atalasoft. I think this configuration makes sense, since development VMs are normally insecure and disposable, and should not be exposed to company’s production infrastructure. And it feels like I am not the only one adhering to this path, since lots of people run into numerous problems with setting up a remote debugger in similar configuration. Typical issues one will encounter here:

1.       You cannot attach to the remote process due to authentication issues (“The Microsoft Visual Studio Remote Debugging Monitor (MSVSMON.EXE) does not appear to be running on the remote computer”, here);

2.       Remote debugger cannot connect back due to firewall issues or different authentication issues (“The Visual Studio Remote Debugger service on the target computer cannot connect back to this computer”, here);

3.       Publishing your solution (we do SharePoint development) on the target server is not trivial task;

4.       Remote debugging does not pick up debugger databases, so you cannot step through your code (“The breakpoint will not currently be hit. No symbols have been loaded for this document.”, here, here and here);

5.       …or will find them but will refuse to load them (“The symbol file XXX.pdb does not match the module”, here);

6.       …or will skip loading because of “Just My Code” debugging settings;

7.       …or different debugger errors.

Here is some information on how to get this thing to work…

You run debugger on remote server and local desktop. Remote server, as I mentioned above, is not on domain (in my case it is a virtual machine in “Debug” workgroup), and local desktop belongs to Atalasoft domain. According to MSDN, CLR does not provide explicit remote debugging capabilities, so the debugging is done through a proxy application called “Visual Studio Remote Debugging Monitor”.  Your IDE (Visual Studio) talks to that remote debugging monitor, which interacts with the process you want to debug. The way communication is happening is controlled by transport. Default transport is DCOM-based two-way protocol with Windows authentication on top of it. For CLR it is pretty much the only way of remote debugging (the other way is called “Remote” and it is suitable for native code only). Debugging happens in the following way:

1.       First, you deploy assemblies on the target machine;

2.       Then you start the process you want to debug on the remote machine. For SharePoint the process is already running (w3wp.exe, WWW worker process), for other applications you may either use WMI to start it remotely or use Sysinternals’ psexec;

3.       Attach to the remote process from Visual studio (this is done in two directions: first your visual studio connects to visual studio remote debugging monitor, then visual studio remote debugger monitor connects back to your visual studio, and this is where 50% of troubles are);

4.       Have your debugging symbols database (PDB) files loaded  (this is where remainder 50% of troubles are);

5.       Enjoy your remote debugging

Setting up

MSDN has number of articles on this subject. For this particular scenario you will need:

1.       On remote machine:

a.       Install remote debugger (it can be downloaded here). In my case running installer from desktop did not work, so I had do copy it to the temp folder and bounce the server.

b.      Make sure the local security setting is set to “Classic” (look for Local Security Settings under Administrative tools):

c.       Create an account on the remote machine. Visual Studio will use this account to connect to the remote machine. It must have the same user name and password as the user you running the Visual Studio as (i.e. your domain user name). In my case, my domain login is “dbarvitsky@atalasoft.lcl” and the user name I am creating on the target server is “dbarvitsky”. This must be the user local account, since Visual Studio will be logging in as DBARVITSKYLAB\dbarvitsky (where DBARVITSKYLAB is my virtual machine I use for debugging). User must have the same password as well. It also must be an administrator on the virtual machine.

d.      Create second account on the remote machine. This account will be running the remote debugging monitor. I use “Debugger” name on remote machine.

e.      If you are planning to run debugger as a service (so that you do not have to start it manually) make sure “Debugger” user has a permission to run as a service (look for Local Security Settings under Administrative tools or just run secpol.msc):

2.       On local machine:

a.       Create account on the local machine (where you have Visual Studio installed). It must be local user account with the same as the account you created on previous step (“Debugger”). The password must be identical to the “Debugger” account on the remote machine. Remote debugger will use this account to connect back to Visual Studio.

b.      Alter firewall settings to allow inbound TCP 135 port (Remote Debugging DCOM) and grant Visual Studio permissions to create outbound connections:

It normally makes sense to limit scope to “my network/subnet” only (assuming that your VMs are in the same subnet as your machine):

Hints on deploying assemblies

The first step, deploying assemblies (and content) to the remote server implies that you have them built locally and the target platform matches the virtual machine you deploy to. Getting the content on the target machine is not a problem and can easily be done with NAnt:

<copy todir="${destination}" verbose="true"> 
<fileset basedir="${source}"> 
    <include name="pattern\*.*"/> 
</fileset> 
</copy>

For assemblies you have two options:

1.       Simply copy them to the C:\Inetpub\wwwroot\wss\VirtualDirectories\<port>\bin on the target server with their PDBs (debugger database files, where your debugging symbols come from; Visual Studio needs them to relate point in compiled assembly back to the source code, variables, etc);

2.       Install assemblies with PDBs into GAC. You can use AtalaGAC.exe tool for it.

Choose the one that is closer to your production deployment. If you just drop the application in virtual folder and will have all assemblies in bin directory, choose the first option. If you are going to have some fancy installer and will require assemblies to be registered in GAC, go with the second one.

Either way, you will need to have your new assemblies reloaded. Again, there are two ways to do it:

1.       If you are running in share environment (i.e. many people use the same server for debugging), recycling entire IIS is not nice. Instead you can reload only application pool you are working with. Running “iisapp /a ‘SharePoint - 80’ /r” remotely will do the trick (“SharePoint - 80” is the application pool name);

2.        If you can afford recycling entire IIS just run “iisreset” on remote system.

Example NAnt script:

<exec program="C:\Program Files\Sysinternals\PsUtils\psexec.exe"> 
    <arg value="\\${target-server}"/> 
    <arg value="-u"/> 
    <arg value="${target-server.login}"/> 
    <arg value="-p"/> 
    <arg value="${target-server.password}"/> 
    <arg value="iisreset"/> 
</exec>

Attaching to the remote process from Visual Studio

This is a bit tricky part. In order to do this, you need to start remote debugger monitor. Again, two options:

1.       Use “Run As…” and runs as Debugger:

2.       Use remote debugger’s configuration wizard to run it as a service:
 

After you started remote debugger, run your Visual Studio and run “Attach to process”.

If you are running as a service, use DBARVITSKYLAB for qualifier and remember to check “Show process from all users”:

If you are running remote debugger as application, use Debugger@DBARVITSKYLAB as qualifier:

Then select a process (w3wp.exe in my case) and click “Attach”. If you connected successfully, you should be able to see the list of remote modules (Debug, Windows then Modules or Ctrl+D, then M). You should be able to find your assemblies; assemblies must be loaded with symbols. If it is not the case, check your Visual Studio’s Tools\Debugging\General and disable “Just My Code” and make sure you have deployed your PDB files properly. Technically you do not have to have the PDB files on remote machine, but Visual Studio must be able to find them.

Normally you will not have all PDBs loaded immediately after starting application pool, because assemblies did not have a chance to load as yet. All it takes is a single GET request, which I normally include in my Nant deployment script.

References:

1.       http://blogs.msdn.com/jmstall/

2.       http://blog.ray1.net/2008/04/debugging-sharepoint-gac-dlls.html

3.       http://www.belshe.com/2004/03/29/gacutil-gac-install/

4.       http://www.wictorwilen.se/Post/How-to-get-Remote-Debugging-work-properly.aspx

5.       http://www.eggheadcafe.com/software/aspnet/30490153/the-symbol-mymodulepdb-d.aspx

6.       http://forums.asp.net/t/1349247.aspx

New to Atalasoft

I am proud to announce that as of January 6th 2009, I joined VizitSP team in Atalasoft. I am really excited about my new position, the team and the product. I’ll be working with Jacob, David and Rutherford. Woo-hoo!