Lou Franco's ECM Imaging Blog

My colleague, Eric Deutchman, shared Jakob Nielsen’s announcement of the 10 Best intranets with me yesterday, and pointed out this tidbit:

In total, the 10 winners were built on 26 different products — substantially fewer than the 41 used in 2008 or the 49 used in 2007. Most impressively, fully half of the winning intranets used SharePoint, especially the recent MOSS platform (Microsoft Office SharePoint Server 2007). As the following chart shows, SharePoint use has grown dramatically in recent years. This is particularly impressive given that, from 2003–2006, the winning intranets didn't use earlier versions of SharePoint at all.

Another interesting finding of the report:

Some winners supplement their main platform with a few selected tools for specialized purposes.

If supplementation is a necessity, then it’s no wonder that SharePoint is doing so well. Back in October, I speculated on Microsoft’s Five Year Plan for SharePoint. In the post, I said:

Microsoft loves to build platforms. […] You can be sure that any successful Microsoft product will eventually morph into a platform. One way to know the platform is ready is when Microsoft starts delivering new functionality by just using the platform's public API.

Microsoft builds platforms because it knows how to build partnerships. Each of the platforms listed above have a thriving eco-system of ISV's delivering products to run on top of their platform.

SharePoint is a platform for provisioning web applications. Out of the box it comes with a lot of functionality, but its strength is in how easy it is to extend and the third-party market that is forming around it.

Sachin Rekhi’s TuneChimp is on Google App Engine, he says:

The single greatest advantage of Google App Engine is speed to market of an application. […] Since GAE only supports a narrow web app scenario, it makes it extremely simple to setup, develop, and deploy an app. No need to install an OS, configure apache, nor optimize mysql.

This is exactly the point I have been trying to make in my posts about App Engine (GAE should emulate AppStore, the follow-up, and Can this AWS website go on GAE). If something is possible, but not trivial in App Engine, then they lose. In order for App Engine to compete with Amazon or Microsoft’s cloud offerings, they have to implement all of the infrastructure for a web site.

For the features that they address (Web front-ends, database, authentication), they are incredibly easy – I had a basic website with data and authentication with an hour of signing up.

Just to reiterate what I think they are missing:

• A payments API that supports recurring payments
• A way to run non-web tasks either on a timed basis or off of a queue
• Short time-outs and quotas need to be removed (presumably they will when they start charging)

Sachin’s final analysis is interesting:

Probably the greatest detractor from Google App Engine though is the propriety stack that it is built on and the resulting lock-in.

[…] While Google App Engine has become my web development platform of choice for all my weekend projects, I would not yet take the risk of running a production web business on top of GAE.

I have to agree with him. I’m not as concerned with the lock-in, but GAE is only compelling to me if they address 100% of the needs of the service I want to build on top of it.

The Amazon blog has details on how Soocial.com ported their site so that it was 100% on AWS:

In the last few months, they've been working hard to cope with tens of thousands of users and to get ready to easily scale to millions. To make this possible, they decided to move ALL their architecture to Amazon Web Services. Despite the fact that they were quite happy with their previous hosting provider, Amazon proved to be the way to go.

I think this architecture is the perfect illustration of the differences between AWS and Google’s App Engine, and why I think App Engine should strive to be the obvious choice for sites like Soocial.

The blog post breaks down the architecture into 4 parts (1) Web front ends behind a load balancer, (2) Sync servers behind a load balancer, (3) background sync bots run by a queue, and (4) the database.

Each piece has a few moving parts.  Here’s a description of how the database was built:

Database: After evaluating all the different database replication solutions for PostgreSQL, they decided to use pgpool-II. This is PostgreSQL middleware that allows you to do all sorts of stuff including: multi-master synchronous replication, sharding, master-slave replication (although in this case it won't actually do the replication for you, therefore you need to use something like slony for that), online recovery, connection pooling, caching, etc.
The nice thing is that it didn't require any change in their application layer. It behaves like a real PostgreSQL server, therefore they can continue using the exact same previous ActiveRecord configuration. Both the webapp and the sync server use ActiveRecord and the same models as their data abstraction layer.

They use it for different purposes:

- Data shards: they configure rules for where the data goes and comes from, and pgpool-II takes care of it.
- Replication: shards aren't PostgreSQL, they have in fact an extra layer of pgpool-II instances that do synchronous multi-master replication. This way, if ones of their servers goes down, they're not affected. All they have to do is switch the Elastic IP for the given shard which allows to do hot-failover as all the db connections will automatically switch to the new server (not exactly heartbeat, but good enough).

AWS is far more flexible than App Engine – if your plan is to shard PostgreSQL, then AWS is going to be the choice. But, if all you care about is having distributed, highly scalable data, then I replace all of that text above with “let BigTable handle it”. Of course, if you are porting an existing system, it won’t be that easy – but if you are starting from scratch, it’s very compelling.

The same goes for pieces (1) and (2) of the architecture – that’s just handled by AppEngine – just write python using their web framework – load-balancing and elasticity is just handled.

The major problem comes with part (3), the sync bots that are launched by the queue. App Engine has yet to deliver any kind of background process (all work is in response to an HTTP request). But if Google solves this piece as elegantly as BigTable solves distributing the database (will it be a queue and MapReduce?), then they would have all of the pieces for an architecture comparable to this one, and one that is a lot easier to set up if you are starting from scratch.

Here are the blogs Atalasoft hosts and one of my favorite posts from each – I stayed away from product related posts, and they are presented here in no particular order.

Rick Minerich’s Development Wonderland

Rick has become quite the F# expert and has spoken at a few venues about getting started with F#. Back in April he documented some of the work he did to play around with concurrency by developing an Ant Simulator.

10 Hours in F#: Exploring Concurrency Through An Ant Colony Simulation

If you build it, They will come

Dave Terrell’s blog on building might change focus in 2009. I keep referring to this blog post any time I need to debug something that’s hard to attach a debugger to.

Debugging your custom NAnt tasks

If you can’t be a developer, be a developer support

Elaine, our developer support supervisor, is fast becoming an expert on Blueprints.

This post is a step-by-step guide for getting started with Blueprints":

Blueprints, part deux: The question isn't "What are we going to do," the question is "What aren't we going to do?"

If you are in Western Mass, be sure to attend her talk on Blueprints at the Western MA .NET Users Group on February 3rd.

David Cilley’s AJAX Imaging Blog

The end of his three-part series on using YUI sliders and opacity to show live image processing is incredibly cool.

Ajax Image Sliders Part 3: Intervals with Opacity

Bill Bither’s Insights

This was our first year at DMS (industry trade show in Cologne, Germany). Bill documented the experience on his blog:

Our First European Trade Show

Steve’s Tech Talk

Steve’s blog has a lot of advanced .NET programming topics. This post on late binding shows how to get a kind of multiple inheritance in C# (with full source)

Late Binding in C# using Dynamic Compilation

Insert Quality Here

This blog by Adam, our QA Lead focuses on QA topics. He recently compared various web testing frameworks in this post:

Web Testing Frameworks Compared

Jake Opines

Last, but not least is this free app from Jake (with full source) for creating CSS image sprites

Spritify: Writing a CSS Image Sprite Generator with DotImage

31 Apps in 31 Days

Speaking of free apps, in May, Atalasoft delivered 31 free imaging applications built with our toolkit (full source included).

Check out the 31 Apps Gallery

Luis Sala made some good points regarding my post from last week:

So, one can monetize an application through ads or subscription fees. Google clearly offers a means of providing ads, so I’ll move on to subscriptions. Google doesn’t currently provide a recurring billing mechanism. Presumably, at some point in the future, they might do this but in the meantime I’d consider Amazon Flexible Payments Service (FPS), a PayPal-like service that, among other things, supports recurring subscription billing. FPS can be easily integrated to AppEngine using the open source “boto” library, a Python library to Amazon Web Services that can run on App Engine.

Luis is absolutely right that this is possible -- which I alluded to in my mention of using PayPal. But, I don't just want it to be possible -- I want it to be included.

The iPhone AppStore is easiest way to go from idea to sales that I have ever seen in software. Sure, the number of sales can be low, but I'd say that it's roughly proportional to your effort and idea. I think that the AppEngine can be the same way for web-based services, but to do so, they have to provide all of the infrastructure necessary, which includes recurring payments.

The Google Solutions Marketplace that Luis pointed to does look like it offers benefits of the AppStore (a single place where users can find applications they need), but since Checkout doesn't have subscriptions, I can't imagine that payment is integrated -- unless you just charge a single price, which doesn't make sense for a service.

So, Google is getting there, and they are closer than I thought. I don't know what use cases they are have prioritized for AppEngine, but I want to stress that the subscription-paid web-based service model is very appealing to developers and if AppEngine supported it out of the box, a lot of them would take a closer look.

Google put up a sneak preview of their expected pricing for App Engine.

You'll be able to buy capacity based on a daily budget for your app, similar to the way AdWords spending works. You'll have fine-grained control over this daily budget so you can apply it across CPU, network bandwidth, disk storage, and email as you see fit. You'll only pay for the resources your app actually uses, not to exceed the budget you set.

It's pretty much what you'd expect given the precedent set by Amazon. Cloud services are something we pay for.

Even though I was expecting this, there was a part of me that was hoping Google would find another way to deliver these services. It comes from this line of thinking:

1. There are a bunch of companies offering free services on the web 
2. Some of them get really popular
3. Presumably this means they need more server power, which most seem to pay for with VC money.
   
4. Then they get bought and someone gives them a load of money and pays for all their future operations costs.

The problem is that this is a lottery -- you only recoup your expenses if Google buys you. What would be nice is if Google could figure out a way to get you money proportional to your success.

The interesting thing is that this problem has been solved recently. Apple's AppStore provides developers with everything they need to start a business around their app. They handle some level of marketing, fulfillment, notification and delivery of updates, etc, and they take a cut of sale. If you make an app that gets 10 sales a day, you get 10 sales worth of money. It's not much, but you can grow from there -- you have some basis for investing in promotion.

What Google needs is a payments system for AppEngine -- and a way to pay for cloud services from that -- that way moderately successful web applications can provide proportional returns to their developers. I see that Google Checkout can be tied into the App Engine, which is great -- but Checkout doesn't support recurring payments, so it's pretty useless for this -- you can use PayPal, but it seems a natural to want Ad buys, Ad hosting revenue, and payments to all come in and out of the same pot.

Of course, Amazon already has this, but for sheer idea to deployment, the App Engine is amazing -- and it's tuned for these simple for-pay web services that 37 Signals keeps telling everybody to build. Without recurring payments, though, you just that much more friction to building your app, when the App Engine approach makes everything else so easy.

Jim King, who runs the very thorough Inside PDF blog, is starting a series on digital signatures in PDF. This first installment is mostly to set up the series, and makes this important distinction:

The term "electronic signature" is a very general term that includes the idea of putting an electronic image of a signature on an electronic page.  A "digital signature" is much stronger and involves assurances that the document hasn't changed and the signer is who they claim to be. In fact, such digital signatures go way beyond paper and ink signatures with respect to assuring that the document has not been tampered with and that the signature is really a valid signature for that person. We are going to talk primarily about digital signatures.

And he points to this article on electronic signatures from the Adobe security matters blog.

The Google App Engine Blog is reporting that SalesForce has released Force.com for the AppEngine.

a Python library and test suite for apps running on App Engine that enables developers to easily access the Force.com Web services API. This means that developers can now build apps that manipulate and display data stored on Force.com.

I haven't done a lot of SalesForce extending, but from what I looked at it was a lot easier to do client-side applications than server-side. For the client, they offer hosting JavaScript or any other client-side technology (like Flash or Silverlight) without much fuss.

If you need to run server-side, you have a couple of options. SalesForce can host Java, but there are restrictions and it's not clear to me how a potential Apex ISV would do it (it's more clear how direct customers can do it). The other option is to have your own server -- which is fine, but then you lose the cloud benefits that SalesForce offers -- I guess, you could make your server an Amazon EC2, but hosting right in SalesForce offers some benefits over that (mostly pre-baked integration).

So it would seem that this combination gives you some aspects of both -- you get the easy integration combined with having a little more control over what you can put out on the server. Google App Engine isn't currently much better than SalesForce in terms of unrestricted server-side code, and its beta status will scare away some, but for an Apex ISV, I think this option is preferable to hosting in SalesForce, and may one day be better than something built on EC2. To make it more acceptable, Google needs to have a way to pay for usage beyond their limits, and it wouldn't hurt to have some kind of SLA, but the market seems to be accepting that right now.

I saw from Document Imaging Talk, that Ralph Gammon is talking about how Kojax (a leaked MS mobile AJAX platform) might impact Document Imaging.

This, of course, is the hottest segment of the computing market and one that document imaging vendors have been trying to figure out their place in. AJAX, because of the fast and convenient image viewing capabilties it offers, has become a hot platform for browser-based document image viewing.

It doesn't take much to get AJAX to work on mobile devices with a modern browser. I built a DotImage based document viewer for the iPhone for a presentation a few months ago.

I don't think there's really much that a framework needs to do other than being built on AJAX to work on a phone that supports it.

So, what do I think MS is up to? A few things:

1. MS always tries to provide their own API for any technology stack -- they are, after all, a platform company. 
2. The biggest value-add they can bring is some visual tools -- they are good at it and have a platform to build on (Visual Studio)
3. We also need frameworks that make it possible to treat all of the devices the same.  Web developers already have enough trouble with IE vs. Firefox vs. Opera vs. Safari. Throw in the plethora of phone browsers and we really need someone to make sense of it all.
4. All AJAX frameworks provide some kind of easier to use development model -- so I'd expect that to.

But AJAX is inherently suited to viewing images. The client-side has more than enough tools, is ubiquitous and zero-footprint. The fact that AJAX implies that we have a server means that we can do powerful image-processing there.

Last summer, I wrote that AJAX was the first platform that combined Ubiquitous clients, server-side deployment and usability. Part of my ubiquity argument was mobile devices (where Flash, Silverlight, and Java lagged AJAX), and MS concentrating on AJAX shows that they are at least banking something on it being the mobile delivery platform of choice.

Last month Microsoft unveiled the SharePoint Guidance site to help "architects and developers design, build, test, deploy and upgrade SharePoint intranet applications."

The site is organized around developing a SharePoint application to help an HR department manage training in an organization and goes through the entire life cycle of developing the application including unit testing, deploying and upgrading.

Tim Bray (Sun employee) just blogged about what he thinks Sun needs to do to reinvent itself. I think his analysis of Sun's strengths (DTrace, ZFS, HotSpot, Solaris and Storage) are right on, but I disagree with the conclusion:

To my ears, these technologies scream “Deployment time!” In particular server-side deployment, in particular of Web applications.

Therefore, Sun should adopt a laser focus on building a Sun Web Suite and becoming the Web application deployment platform of choice. It’s a large space, a growing space, and one where we can win.

The rest of this piece assumes this strategy and considers how to get from here to there.

Yes, it's true that Java (and maybe PHP, Ruby, etc) web applications might work better on Sun infrastructure, but in the Unix world, hardware and the OS is pretty commoditized. I worked in a Linux shop writing very big web-based Java apps before I joined Atalasoft, and we never had any issue that would be worth moving to Sun. The performance of Linux on commodity hardware was just never an issue for us. Sure, DTrace is awesome, but it's by no means the only way to monitor and debug a runtime service, and I'd just wait for it to come to my OS before undertaking a major project like moving to Solaris.

If Sun wants to prove that their deployment is better, they need to start showing us what they can do with it. Tim talks a bit about the cloud, sort of as an after-thought, but that would actually go a long way to showing me that the Sun deployment story is better.

And they need to start delivering useful enterprise applications on top of their infrastructure. Look at Microsoft as an example -- Word and Excel sold their desktop deployment framework (Windows) and SharePoint is driving sales of their web deployment infrastructure (Window Server+SQL+IIS+ASP.NET). Look at the sales numbers for SharePoint, and tell me that Sun wouldn't want even a fraction of that business. Look at how Apple is driving iPhone sales.

The easiest way to get there:

1. Acquire Alfresco (Java-based open-source competitor to SharePoint)
   
2. Make it scream on Sun hardware/software and especially MySQL
3. Use their storage expertise to bring it to another level
4. Put an on-demand version of it up in the cloud
5. Build a marketplace around creating add-ons to it, and make them easy to buy and activate (like Apple's AppStore or SalesForce's App Exchange)
   

A few weeks ago, I wrote a blog entry on the ISV perspective of CMIS, and made this statement of when I thought we could reasonably rely on it:

The current version of SharePoint doesn't support CMIS and maybe even the next one won't (no commitment yet from MS).  I have to keep SharePoint specific code until we want to drop support for these SharePoint versions -- we are looking at 7-10 years from now.

Today, I read this perspective on CMIS on CMS Watch that I think is right on:

I happen to think CMIS will be a success, in part due to it's simplicity and focus. And we need to remember how success will be measured. True success will have nothing to do with the number of vendors who come out with CMIS implementations, rather it will be the number of onsite applications that use CMIS to integrate systems, that will determine success. It is also worth remembering that standards take many years before they really take hold, typically between 5 and 7, so even the most longsighted of us has no real idea as to what the future really holds for CMIS. All I know for sure is that ECM interoperability for buyers is a must have, CMIS (or something like it) is a necessity.

Ok, so we agree that it will be about 7 years. I also agree that something like CMIS is necessary, and that success is measured by how many applications are written on top of it.

To reiterate from my previous article, I think Atalasoft will have CMIS connectors in our Vizit document imaging suite for SharePoint, but that will be in addition to native connectors until CMIS is performant enough.

Last May, Atalasoft released 31 apps in 31 days, each free, with full source available, and made with DotImage.

I wrote App #7 which was a desktop application to scan, clean, OCR and upload your documents to Scribd.

This month, I went back to the project and wrote a step-by-step guide to building Scan-to-Scribd for CodeProject.

All of the non-uploading code will work in any Scan to X project, where X can be Amazon S3, Azure SQL Data Services, Google Docs, SharePoint, or any CMIS compliant ECM (you just need to write the uploading part). If you want to roll your own service we have this tutorial for building a simple repository with upload service using ASP.NET webservices and SQL Server (DotImage only needed if you want to scan or view documents)

Over at the Document Imaging blog, there's an interesting post on 5 considerations for using SharePoint with document images. I don't agree with the first one, however:

In many organizations, SharePoint projects are departmentalized and are outside of the domain of IT. In fact, according to AIIM’s 2008 State of the ECM Industry, only 42% of SharePoint projects have a formal plan that involved IT. Therefore, your capture integration with SharePoint should be virtually “plug and play” and should not require any modifications to the SharePoint server.

SharePoint is a platform for provisioning websites -- and has quite powerful facilities for deploying and hosting server-side code. In fact, it is this capability that makes SharePoint so effective at hosting document images, because the server-code can preprocess the images and render a zero-footprint viewer, thus making it possible to view document images directly in SharePoint.

Here are my top considerations for document imaging in SharePoint:

#1: Getting scanned content into SharePoint should be streamlined with your scanning process. You have paper in your hand and you want it in a document library -- it should be no harder than (1) load the paper (2) click a button on the scanner. It would be great if your software could read the document and just know where in SharePoint it should go, but barring that, a quick list of choices show up on my screen and I pick one.

#2: Viewing a document should be fast. Document images are potentially very large files. I should be able to quickly bring up a thumbnail of the first page and get others on demand. I should not have to download the entire document image to find out what it looks like.

#3: It should be easy to act on multiple documents at the same time. When indexing or cleaning up a document image it is very likely that I will want to do more than one at a time.

#4: Document Imaging should be pervasive. Should be supported in libraries, lists, web parts, search results, workflows, etc.

#5: Cost should not be a barrier to adoption. Ubiquitous document imaging is only going to happen if you can afford for everyone to use it. No document imaging silos.

Atalasoft is working hard on this, and making document images usable in SharePoint is so important to us that we're giving away SharePoint document capture and SharePoint document imaging previewing (#1 and #2 above -- and I guess #5 -- stay tuned for how we're going to address #3 and #4).

Just saw a report on Microsoft and Open-source stack usage in the enterprise. Its goal was:

The goal of this survey is to research how open source is used within the enterprise, as well as examine the growing impact of Enterprise 2.0technologies and competitive systems such as Microsoft SharePoint.

And it had kind of surprising result:

Microsoft Stack Adoption

• 92% do not use or intend to use Silverlight;
• 86% do not use or intend to use .NET/Web Parts;
• 64% evaluate on Windows;
• 53% use or intend to use a Java architecture; and,
• 52% use or intend to use AJAX.

It's a comprehensive study (25,000 participants), but this is who they are:

Alfresco Software, Inc. recently announced the results of the third global survey of trends in the use of open source software in the enterprise. The Alfresco Open Source Barometer survey is the largest open source enterprise infrastructure/stack survey. The report covers data provided by over 25,000 of Alfresco's 74,000 community members during the period of April to September 2008.

Alfresco is itself an open-source/Java based product. So you need to take that into account when looking at these numbers.

They are totally up-front about this, but I'm not sure how useful Alfresco evaluators are as a barometer to finding out SharePoint usage.