Thursday, April 16, 2009

Does workflow always have to suck?

I evaluate CMS and DAM systems for a living, and one thing I keep coming back to is the fact that so very few of these expensive systems "do workflow" well. I think part of this may be because there are no industry-accepted standards around the kind of workflow I'm talking about (thus, every vendor reinvents the wheel). The closest thing to a standard is BPEL4People, which is an extension of BPEL (and thus too heavy, IMHO). There needs to be a minimalist standard around this domain space, something dedicated to human-facing interactions, supporting process-facing tasks optionally, not the other way around.

I think the other reason so many Web CMS and DAM vendors fail to do a nice job with workflow is that it's just plain hard. Light-duty taskflow or workflow (or "pageflow," as we called it where I once worked) is deceptively difficult to implement, especially if there's a requirement for good UIs around administration, design, and (re)configuration of workflows. And especially if there's a requirement for hot failover (being able to deal with STONITH and other messinesses). And especially if you need to support acyclic (reentrant) flows. And especially if you want to offer good extensibility APIs. And, and, and.

Most systems that support approval workflows (of the type seen in web publishing scenarios) get the basics right, but after that, not. Typically, though, the customer hasn't really thought out his or her use-cases very well before buying a system. And so begins a long process of design, test, rollout, fail, back to the drawing board.

Setting up workflows typically means developers need to touch XML, code, properties files, templates, and/or miscellaneous artifacts, often editing them by hand (since it's unusual to get good tools for this). You may be able to draw a basic flow on a canvas (although even that isn't done well, by many vendors), but applying timeout and retry policies, and handlers for exceptional conditions, may involve a good bit of "dirty fingernails" work. When you're all done, the customer thinks "Okay, done. This should last us for all eternity. Glad we never have to do that again!" But very soon, it becomes clear that a number of corner-cases that were not anticipated at the design stage need to be handled better. So it's surgery time again. Back to messing with a bunch of artifacts and their cobweb of dependencies, then finding a way to test it all, etc.

Administratration is often not well supported. How do you run a report for all workflows of Type X in the past month that either finished abnormally, didn't finish at all, or took too long? (Don't tell me "look in the logs.") What UI tools do you have for simply finding an orphaned workflow, or killing an in-progress workflow instance? How do you know if bouncing one machine (in a cluster) left one or more workflows (of potentially hundreds in progress) in an inconsistent state?

And then there's rights administration. When Sally goes on vacation, how does she give her subordinate, Bob, her "rights" in the system for a workflow of Type A but not for workflows of Type B?

The issues get sticky in a hurry. But I do occasionally see workflow systems that combine decent functionality with a usable graphic designer, or with good administrative tools. But it's hard to get a robust engine, a good feature set, good visual design tools, and good administrative tools all in one package. There are always warts and holes.

So I think the right way to look at all this, if you're a vendor, is that this presents a ripe opportunity. If you're a CMS or DAM vendor looking to differentiate, provide a superior workflow solution.

But there's also an opportunity in the market, right now (IMHO), for someone to come up with a fully productized lightweight workflow product with decent design, development, and admin tools, and easy extensibility (of course), that can be bolted onto a Web CMS or DAM system with little effort, so that customers who are using bespoke WF systems (of the kind that are so common in this industry) can move over to "real" workflow. And get real work done.

I wonder why products like this aren't more common? Again: Probably because it's hard. But again: This is an opportunity . . . for someone.