While talking with a client yesterday I failed to articulate clearly the difference between templates and patterns.

In fact, they represent two methods of problem solving:

• A template is a standard solution for a certain type of tasks. E.g. when I’m going to conclude a partnership agreement I pick up the appropriate template and it becomes a zero version of the document.
• A pattern prinicipally is a standard solution too, but it is used creatively on the basis of a “problem shape” rather than problem’s formal type. Unlike templates, patterns may come with a positive (”how to do”) or negative sign (”how better not to do”).

Vague enough, right? Trying to make it simple, I found an analogy in chess:

• Debuts are templates: learn them and use them, chessbooks plot the game 20 moves ahead from the initial position.
• Typical combinations of the middle game are patterns. E.g. fork is a pattern: if you can see an opportuntiy then use it or threaten to do so. Two rooks on the same vertical is a pattern too while two pawns is an antipattern: try to avoid such a position if possible. But no chessbook contains instructions how to make a fork starting from the initial position.

With regard to business processes, sometimes a business process model is called a process template. Process instances are created from templates about the same way as zero version of the document in the example above.

The second meaning of the term “template” is a typical, standard process model. A vendor or consultant develops a process model on the basis of previous experience and then offers it to prospective customers. Supposedly the customer will be able to use it with small customization, saving efforts in comparison with a development from scratch.

For example, my company Business Console successfully implemented a BPM solution targeting a sales process executed through geographically distributed offices and now we offer it as a template, i.e. a starting point of a BPM project to companies that manufacture and install plastic windows, metal doors, custom-made furniture etc.

Pattern in BPM is a typical process fragment of typical way of communication between processes (some examples).

One may ask: which one is more usefull? My opinion it’s a pattern:

• Templates are specific (one process - one template), patterns are universal. A good pattern can be used in a variety of business processes regardless of the industry.
• A practical benefit of using a template may be less than expected. It usually covers the happy path only and the devil is in details - various workarounds, escalations and exceptions.
• The effect of using the right pattern can be large. For example, there was a case in our practice when the process plotted at 6 A4 sheets glued side-by-side was reduced to the elegant design with just 15 activities by using the right pattern.
• The value of the antipattern is its ability to preserve you from mistakes. The price of a mistake is unlimited in theory and sometimes it’s really big in practice.

Another reason of my skepticism towards templates is the fact that in case of BPM we are targeting continuous improvement, not one-time automation. Templates, by definition, can be applied only once: during initial modeling. The ability to use patterns will remain open forever. It’s quite possible that after discovering a new pattern you will look at the processes you designed from a different perspective and will find a way to improve and/or simplify them.

An example:

We can draw whatever we wish of course but as soon as we tried to make this diagram executable we’d find out that in the real world the payment doesn’t always arrive. As well as in the case of “one-way activity” antipattern, we communicate with an independent subject having a free will (the buyer) and we can’t assume that it will follow our prescriptions.

We should consider at least three outcomes in this example:

1. payment arrived
2. buyer informed about refusal to pay
3. neither payment nor refusal arrived in due time

There is a special BPMN construct exactly for such a case called “exclusive event gateway”:

The beauty of this construct is that the process executes in parallel after passing the gateway, expecting all three events simultaneously. But as soon as one event happens, the other branches stop executing automatically and immediately.

The big regret is that BPMS vendors tend to consider the event gatway as luxury. I know several systems being declared as supporting BPMN yet not supporting this construct.

This is mistake I’m afraid because alternative messages processing can’t be avoided! The only possible way to solve the task in the absence of the event gateway is parallel gateway. But here we hit the problem of stopping other routes as soon as one of possible events happens. It can only be resolved with ugly diagram constructs and/or program code. Of course the process visibility and standard compiance become lost on the way.

This antipattern is special because it comes not from process analysts but from BPMS developers. On the other hand, it’s BPMS vendors will to eliminate this antipattern - they only need to change their attitude towards the event gateway.

As long as particular BPMS does not support the exclusive event gateway, one cannot effectively utilize the message flow. The software supports orchestration but not choreography. An old good workflow to me, whether it has the BPMN label or not.

This anitpattern is almost trivial. Yet watching how different people make the same mistake at different projects I come to conclusion that it’s quite popular

Example 1: Concluding a contract. The draft must must be signed on behalf of our company by CEO:

Please respect the boss by giving him the opportunity not to sign the contract - put the gateway immediately after “sign by the CEO” activity.

Example 2: while processing a customer’s order, an agent company places an order with a partner:

Recognize that the partner has a free will and take into account the the possibility that he will not accept our order - add the gateway to the process.

One-way modeling is OK at early stages of process discovery, it’s the so-called “happy path”. On the other hand, is there any activity with the predetermined outcome? Maybe it’s better to check the result after each and every activity?

Putting gateways everywhere will clutter the scheme. The lesson learned from this antipattern is this: as a minimum, put the checks after activities where the free will is clearly visible - for example, assigned to decision makers or organizations independent of our.

BPM conquers the new territory which is called in different ways: Dynamic Processes, Unstructured Processes, Knowledge Worker Processes, Barely Repeatable Processes, Case Management.

BPM now reached the maturity level where the management of repetitive and predictable processes has become a matter of technique. It grants reliable process execution by unreliable employees - low-paid, with low skills and low motivation. Such processes are common for state and semi-state organizations and also for businesses following an extensive path of development.

But only “very talented” individuals can believe that every process can be rigidly structured:

Leaving aside the extremes - fully structured and completely ad-hoc processes - there are two combinations: either a structured process launches the ad-hoc or vice versa.

1. “A Little Help From a Friend” pattern: a structured process launches ad-hoc subprocess. An example: system integrator’s “request-to-order” process. Let’s assume that sales rep’s meeting with a client was positive meaning that he found  ”pain points” - issues that the client will pay for if they were resolved by the integrator. The next step of the process is finding a solution. However, the client’s problems may vary at very wide range (note that integrator’s value is exactly his ability to solve a wide range of problems) starting from the simplest need of a boxed software to complex projects. In the latter case the process will follow a trajectory unknown in advance that may involve an architect, developer, sales manager, systems engineer, vendor’s tech support etc. Traditional BPMS can only represent this unpredictable subprocess as a single task (see “Process Antipattern: One Man Show“). This is a poor-man’s solution indeed because it isn’t visible who is addressing the issue at the moment, what progress has been made and what the current timeframe is.
2. The opposite situation, the “Process Toolset” pattern: unpredictable process at the top launching well-formalized subprocesses. An example: a law firm having client’s case as the top-level process. It’s absolutely impossible to predict in what direction the case will turn next day, e.g. a new document may be submitted by the opposite party that will change radically both  the case prospects and our plan for actions. Yet many of the actions initiated within the case are standard, making it possible to formalize each as a subprocess. They are mostly preparing applications or other documents for the court. Such subprocesses is executed by a dedicated specialist - a common resource not assigned to a specific case. From business perspective it’s interesting to monitor performance not only of the case as a whole but also of subprocesses and resources usage.

Going down from the top-level business processes (company’s value chain) to the lowest level (micromanagement) we must be prepared to encounter both structured and ad-hoc process. Support for the latter in today’s BPMS is far behind but this subject is widely discussed by researchers, analysts and vendors:

• The following estimate was made at Gartner’2009 conference: as much as 60% of an organization’s processes are unstructured – and probably also unmonitored, unmanaged, unknown and unruly. These 60% are like “average temperature by the clinic” but the “invisibility” of these processes can indeed be the major issue from business standpoint.
• “Tapping into Collective Knowledge Will Drive Unstructured Process Activity” - Jim Sinur predicts that organizations acceptance of collective knowledge, industry networks and even social networks will result in fundamental changes in BPM. His another post on the same subject: “White-collar and unstructured processes go together like cheese to wine“.
• The boom of social networks pushes the idea to borrow approaches evolved there for communication within dynamic processes - see the Workshop on BPM and social software at BPM conference in Ulm. For example, when confronted with a problem, I can publish the question on the corporate social network (”a help from a friend”). It’s visible to my “friends” including the project manager and team lead. The best answer gets a bonus.
• SAP shows how Google Wave collaboration technology can be used - not for process execution but for modeling. SAP Gravity is a BPMN modeler implemented within Google Wave. Taking into account that the ability to redesign a process on the fly is essential for dynamic process execution, SAP makes progress not only in process modeling and discovery but in the the execution too.
• Oracle talked about collaborative and dynamic BPM at Oracle OpenWorld’2009. They emphasized commitment to SCA that makes possible to combine different kinds of processes with business rules, analytics, etc. That’s no surprise taking into account that they have acquired about 50 companies in two years and hence face huge integration challanges.
• HandySoft, ActionBase and other vendors claim dynamic processes support in the latest versions of their products.
• The most authoritative industry experts gather to discuss dynamic processes in general and their support in BPMN.

So we can see a number of minds attacking the problem from different directions. Well the more alternative approaches, the better final solution should be.

Yet for BPMS vendors that may be a bloody battle. There is probably more BPM vendors than the market needs and now a new front  opens for the competition: dynamic processes. It’s quite wide front if all aspects of dynamics are taken into account so I’m afraid not all vendors will be able to fight it in the current economic situation. But at the end of the day those who provided full support to dynamic processes will posess a competitive advantage clearly visible for the users.

Alternative title: “Micromanagement”.

The typical first BPM project issue: how deep to dig into a business process details?

We were engaged into a process called “Purchase order by the three-tier agreement” in one of our first projects. Here is the brief:

• A three-tier agreement is concluded by and between the buyer, the manufacturer and the supplier - manufacturer’s authorized partner.
• The agreement is designed as a framework: it specifies prices, terms and conditions, but not the items to be purchased nor the volume. Each purchase shall be specified by a separate specification containing items and quantities. When signed such a specification enforces contractual obligations to all parties.
• The process is managed by the supplier and there were no intention to directly attach the buyer or manufacturer to BPMS. The whole process runs inside supplier’s organization yet there were such activities as “send the specification to manufacturer for signature” or “ship the goods to buyer”.

The process consists of four majour phases:

1. Agreeing the order.
2. Signing the specification.
3. Delivery.
4. Payments and deal closure.

Let’s consider the first phase “Agreeing the order”. Despite terms and conditions are set by the agreement, the bargaining may happen here, e.g. the buyer may request an additional discount for a large order or tougher delivery terms tied to the internal project schedule. In such cases the account manager should agree on the conditions within the organization and with the manufacturer; if the buyer requested too much then a compromise acceptable for all parties should be found. Several iterations may be necessary lasting for months. It’s a delicate work highly depending on account manager’s skills. And this is the key point of the process - the remaining activities are pure routine.

The first version of the process diagram looked like this:

Then it became more compliacated. First, if the manufacturer made a counter-offer that we found acceptable then we should only agree it with the buyer on the next round; the same is true with respect to buyer’s counter-offers. Then it was rightly pointed out that the process may be speed up if negotiations with the manufacturer and the buyer was made in parallel. And so on.

Let me skip the evolution of the process and proceed directly to the resulting diagram:

The key point of this process is that there is a single performer: account manager. No one else cares about where we are inside. Agreeing started / agreeing ended successfully / agreeing failed - that’s all the business (the process owner) is interested in.

I’ve seen process diagrams similar to the first one several times. For example, there was customer’s process of accepting goods to the warehouse with about twenty activities, all performed by a  storekeeper. It doesn’t make sence. You get cumbersome scheme prone to frequent changes yet the details do not add value to the process.

Target BPM on the overall process performance and cross-functional problems which are responsible for poor performance in most cases.

It is not easy - you must be ready to find yourself under crossfire if your studies affect existing borders between organization departments. But don’t go the path of least resistance and don’t use BPMS to document a sequence of activities performed by a single person.

Sorry, this entry is only available in Русский.

Another frequently asked question from a forum.

Question - When a business process template is modified (activity added/removed), what happens with process instances started by this template (previous version). What happens with analytics?

Answer - Most BPMS will finish running instances by following old template and will create new instances based on new teplate. It’s acceptable only if the template modification was planned. If - like it happens in most cases - the process became stuck because the template doesn’t have a desired gateway or a flow then you have nothing to do but abort the process instance and start it again by new template making the analytics inconsitent.

Yet there are systems allowing process scheme modification “on the fly”. Recommended reading:

• ebizq.net article by Glen Smith from Appian explains why this functionality is important from methodology perspective. You can launch the process into operations faster if you don’t have to discover details of all possible exceptions.
• Keith Svenson from Fujitsu notes that implementing process modification on the fly is hard to implement in a system that uses BPEL for excution. Fujtsu Interstage BPM lets you edit the scheme of a running instance the same way you edit a template; you can save the new instance scheme as a new version of the template if you wish.

Please refer to BPTrends report for the information about specific BPMS.

It’s hard to implement on the fly modification if a business process editor (modeler) is implemented as a desktop application which is the case for most systems. If a process instance stuck one should be able to correct scheme online promptly. Hence the modeler must be implemented as a thin client. Fujitsu Interstage for example has both desctop and online (java applet) modelers.

Oracle BPMS (aka BEA AquaLogic aka Fuego) took another way. The modeler pallette of this system has a special “magic” activity which can get control from any activity and/or pass it to any other acitivity. It solves the major part of the problem - absence of a desired flow on a scheme.

But you shouldn’t rely on a tool only - the process sheme should be designed the right way.

Let’s consider a long-term contract with a customer for example. We are going to deliver goods, obtain money and also renew contract’s terms and re-negotiate its conditions many times during several years. If we tried to implement it technically by a single process then for sure its scheme will be changed many times. Better solution is to create a long-living yet very simple process on the top level - a kind of a state machine keeping the state of our relations with the customer: “contract is beeing agreed”, “contract is in effect”, “contract is being agreed” etc. Every action starting from contract negotiations should be implemented by a workflow-like process. This workflow can be started if the contract is at some specific state and it can send a message to the contract process which will bring it from one state to another. Since the state machine is strictly passive one can modify a workflow scheme freely and/or create several alternative versions of e.g. delivery workflow.

As mentioned at “End-to-End Process Orchestration anti-pattern“, business doesn’t work like “one-two-three”: shop floors don’t switch on/off by every client’s order even at produce-to-order scenario; material purchases are arranged for a production program rather than for a single finished product. Hence an end-to-end business process is modelled not by a sequence of activities within a single process (so called process orchestration) but by several processes executed independently and exchanging data and/or messages.

In BPMN each process is modelled by a separate pool. A process chain “Order to Cash” - “Manufacturing” - “Material Supply” for example may look as follows:

“Order to Cash” process is initiated by an incoming message - an order submitted by a client. “Manufacturing” is triggered by a timer, e.g. at the end of each working day, and handle queued orders within a cycle. If necessary materials are not available then “Manufacturing” waits for a signal from “Material Supply” that they are delivered. “Material Supply” is scheduled by a timer too and arranges purchasing for materials having the projected stock balance below the limit. (The internal orders chaing may have more links than in this example, e.g. a metallurgical plant may have inter-shop orders.)

Such asyncronous inter-process communication implies buffers between them. The buffers accumulate orders passed from process-requestor to process-provider. In the example above these are “Manufacturing Orders” and “Planned Stock Balance”. Technically such buffers may be implemented by a number of ways: message queue, database records, ERP objects at some specific state. Of course the buffer internals better to hide by wrapping them with services like “insert”, “traverse”, “extract”.

Modern BPM Suites are able to model and execute diagrams like above and this is the major BPMS advantage over traditional workflow systems. However such diagrams turned out to be difficult for analysts as Bruce Silver noted at “BPMN to Requester: Get Outta My Pool“. The major issue isn’t the notation but “asynchronous thinking”. One should develop an ability to extract separate asynchronous process from what business present to you as a single process. The answers to following questions should help: 1) what business object corresponds to a business process instance; 2) which events correspond to process instance start and stop.

For example, even in the relatively simple process of hiring an employee we can find a set of business objects: 1) a headcount item; 2) manager to HR request to fill the item; 3) a vacancy passed to a specific recruiting channel; 4) a candidate; 5) hired employee. They do not correspond to each other as 1:1 and their lifecycles aren’t synchronous. E.g. a candidate may submit his resume not caring about whether we have a vacancy for him - it’s HR task to assess which vacancy (or vacancies) it’s worth to consider for him. So a single process will hardly suffice; it depends on your business how much process there will be at the end of the day.

Worth to note that employee hiring is a classic business process example that BPM vendors love to use for their products demonstration. Yet they try to do it with a single process! Obviously such demos are made by developers, not consultants.

Final warning: please don’t consider the above as a call-up to breed too many asynchronous processes. In fact, the choice between synchronousness and asynchronousness is a non-trivial managerial decision but we’ll talk about it next time.

Definition:

• “Enterprise Process” (equivalent term “End-to-End Process”) is a business process connected to external customer at both ends and going through more than one top-level company’s departments.

Axiom:

• A BPM initiative would pay for itself only if you target end-to-end processes.

Otherwise you’ll be guilty of suboptimization sin according to Lean methodology or will “shoot sparrows with a cannon” according to Russian proverb. Yet it doesn’t mean you must begin with such a process - you may use something else for training but you won’t win a medal for training only.

Typical errors:

1. BPM vendors love to illustrate their products by supporting processes like “Employee Onboarding” or “Expenses Report” thus provoking prospects to do the same. It’s OK for training but there is simply not enough money beghind these processes to justify enterprise-wide project which what BPM project is by definition.
2. Many prospect wish to go for “Negotiating a Contract” process. There are money behind it but is this a process really? For me it’s rather a fragment of end-to-end process “Sales of Goods or Services”. Customer’s concern being the end result - i.e. performance of the process as a whole - it may happen that the bottleneck wouldn’t be this particular fragment. Narrowing a scope leads to suboptimization.
3. And here is the worst case: “We have a process here, well-studied and alredy automated, now let’s implement it in BPM for comparison”. In other words, let’s make a race between good and better. For your project to be acknowledged by the management you must not only implement some process in BPM but also significantly improve it. You’d hardly make it with a process that was well worked on already.

OK, let’s assume that we do everything right and consider “Order to Cash”, a classic example of end-to-end process. (Good questions would be: “Why exactly this process?” or “How to pick up the best one from many company’s processes?” These deserve detailed answers however so let’s leave them for the next time.)

“Order to Cash” process in “produce-to-order” business scenario may consist of prepayment, manufacturing, delivery and closure subprocesses:

What’s wrong with the above diagram? It assumes (since we have only one pool) that manufacturing works synchronously with sales: no order - manufacturing is idle, order arrives - manufacturing starts working on it. If we got deeper into manufacturing subprocess, we would probably found materials and/or parts purchasing subprocess, sychronized with manufacturing in turn.

But businesses just don’t work like “one-two-three”!

Manufacturing does not switch on/off by every client’s order even if we produce to order. Clients’ orders are accumulated and picked up on regular basis, say daily by production scheduling process. Similarly, purchasing is not triggered by a single order usually but rather runs regularily or is triggered by stock level going below a limit.

Technically such work is implemented in BPM not by a singe sychronous process like shown on the diagram above but by several asynchronously executed processes that communcate with each other by data and/or messages. Such scheme can be drawn with BPMN and executed by BPMS - which, from my point of view, is a big advantage of BPMS over workflow. But let’s consider the correct diagram next time - what I’m going to say here is that asychronous execution is a core feature not only of this particular but of every end-to-end process.

Definitions:

• “Process Orchestration” means tasks execution sequence and logic within a single process frame.
• “Process Choreography” means the logic of several processes asynchronous execution coordinated by data/message flows.

Theorem:

• End-to-end process should be modeled by the choreography, not by the orchestration.

It was confirmed by practice and could be proved by the following consideration: since end-to-end process by definition goes through several top-level departments, you’ll have to take into account the working rhytm of each one of them, which means - modeling asynchronous execution.

An attempt to model it by pure orchestration is nothing else but following an anti-pattern which I will call “End-to-End Process Archistration”.