Modeling workflow loop execution for user reporting in Box Automate

Imagine a form where users submit a contract with two attachments for review. A reviewer rejects the contract, so your workflow sends it back (Loop Back) to the user. On the second pass, the contract is approved, and metadata is applied to both attached files in parallel (Loop Over). But when the customer opens the execution history, they see one flat list: events from both approval passes and both files have been mixed together, with no way to tell which belong together.

In this post, we’ll explain how we turned that list into a navigable history.

Box Automate is a workflow orchestration platform which helps teams turn manual, content-heavy processes into workflows. You can leverage this product to turn work such as routing contracts for approval or extracting metadata into an automated process that takes minutes.

To describe our process, we’ll use Business Process Model and Notation (BPMN), which was designed to help translate business processes into a format readable by both technical and non-technical people. Another advantage of BPMN is that it can be exported to an XML-based format for direct implementation. 

Getting started: what does your workflow require?

For a workflow to run, it requires: 

1. Flow triggers. An event that starts workflow execution
2. Outcome. The step of workflow that defines an action which will be taken, like File Move, HTTPS Request or running an AI Agent
3. Gateways to manage the flow of the workflow. They determine branching, merging to outcomes (can be exclusive or parallel). Those can be Split, Conditional Split, and Loops gateways.

Each workflow must contain a trigger and at least one outcome. It can contain zero or more gateways. 

Within a workflow, Box Automate supports two kinds of loops:

1. Loop Over groups outcomes and will repeat them for a particular set of items. Those can be files that triggered Manual Start or files attached to form submission. Iterations run in parallel. This is like a foreach loop.
2. Loop Back redirects execution flow backwards, to the outcome that was run before. An example usage is Task Outcome when dealing with customer approval. At first the task is rejected, and on the second pass it is approved. The behavior of this gateway can be described like a goto statement.

Now that we’ve covered the building blocks, let’s go to the history page to see how workflow execution is happening. The goal of User Reporting is to show clients which outcomes were executed for confirmation, verification and debug purposes. 

As this automation system is about asynchronous execution, we rely on emitted events to track what is happening. Each workflow execution has metadata attached to it, which contains an array of events. On each outcome we emit an event on start, finish and optionally an event on intermediate action as shown below. 

The intermediate event is used in case of async actions when actions require interaction from another user, such as approving a task or signing a document. So, each outcome generates at most 3 events.

Why can't we just render the event log?

Before loops existed, workflows could only move forward, which meant that each new event was either the finish event of an outcome or start event of the next one. By following events, we could trace execution flow naturally.

Linear execution paths are easy to visualize from an array of internal events. Even with parallel execution, we can’t go backwards, so each outcome will be executed at most once. Even though events will be in mixed order, they can still be rendered from top to bottom. We could simply draw outcomes one by one. When we started designing loops, we asked ourselves whether the current solution scales and whether we can reuse it.

Starting off with the simplest solution, just appending events as they are, we end up in a situation where outcomes can be run multiple times within one execution. How can we progress from that point? To show something meaningful to the clients, we should present which outcomes were executed with what parameters and group events related to specific outcomes. 

This is better than nothing, but we still miss a link between outcomes. Our goal is to show the whole path of execution so users can see how parameters in each loop iteration are changing. The best would be to follow what happens step by step. 

If the workflow is linear, that’s trivial. When we add Split or Conditional Split, we can imagine that we “fold” parts where branching happens from split to merge and as a result we get a linear workflow.

When loops came into play, we tried a similar approach of “folding” and started with Loop Back. We realized that we don’t know which path would be executed in the next iteration and can’t determine which part of the workflow to fold. For example, we loop back after some decision was negative and go later with a positive path which later can come back to the first decision point which leaves us with multiple loops which are convoluted.

Below is an illustration of outcomes in “table of contents” form. “Folding” in this simplified example would mean treating the repeated block of outcomes 3 and 4 as one unit. In one unfortunate scenario, imagine having 10 Loop Back passes and trying to find a specific outcome in the list – this would almost be like searching for a needle in a haystack. Very inconvenient.

We realized that relying solely on an events array is not enough.

How to model storage? Constraints and requirements to paint a clear picture

Let’s take a look at what a simplified workflow execution looks like. 

First, Box Automate receives an internal event and checks if there’s any workflow that should be triggered on such an event. If yes, execution is started, which consists of emitting a workflow start event and scheduling the first outcome to be executed.

Later, we repeat actions while there are next outcomes to be run: a processor receives the result of an action from workers, updates execution history and sends another job to the queue, from which workers pick it up.

We emit events whenever an outcome changes state – like when an outcome is submitted to the queue or results are returned. In case of async outcomes (those which require human interaction, like Box Sign or a Box Task Center notification) we emit another event so that we know the system did its job and action will be resumed once action by user is finished.

Each event also carries a flowID, which records where in the workflow the event happened and which branches and loop iterations led to it. We describe how flowID is built in the upcoming “The Segment Tree” section.

1. The Processor consumes internal Box events (e.g. a file upload) and matches them against workflow triggers.

2. On a match, it starts an execution: it puts the first outcome on the Command queue and publishes a history event to the History queue.

3. A worker picks up the command, calls downstream services (Box API, Box Sign, AI…) and returns the result on the Response queue. The Processor reads it, schedules the next outcome and publishes another history event. Steps 2-3 repeat until the workflow ends.

4. The History consumer, which runs in the same service, reads history events in order per executionId (message key).

5. It appends them to the event log in Execution metadata and updates the Segment Tree described in the next section.

Now we know how events are emitted, so let’s focus on constraints.

In order to remain secure and compliant we can use only approved storage. And to support sharing functionality, we’re using our platform database solution. It is a NoSQL database that we use to store workflow metadata. For the rest of this article, we can imagine metadata as a JSON object.

As mentioned above, we store events of actions that happened during workflow. As a result there are multiple events that describe one outcome. We knew that there would be use cases we couldn’t foresee yet, so we had to build a solution that is future-proof and would be easy to extend. We also wanted to avoid coding current product understanding into schema. Data should be agnostic so we could change the presentation layer without migration.

Some questions that we were asking ourselves were:

• How should we distinguish between flows, and how should we render them?
• What would happen if a workflow contains both loop types, nested in each other?
• How should we deal with out-of-order events?
• How should we deal with race conditions?

Initially, we were considering two options:

1. Group events into separate objects per loop iteration
2. Keep events list as is and add separate data structure on top of them, something similar to indexes in a database.

We chose option 2 because we already had an event log and could build an index on top of it without migrating data.

Making writes safe: an ordered history queue

Updating metadata synchronously from every place that emits an event causes race conditions. Concurrent updates to the same execution conflict and the retries they trigger cause further conflicts. 

Instead, we publish each event to a history queue and consume it in the same service. The broker’s message key guarantees that events with the same key are delivered in order to one consumer at a time. We use the execution ID as this key. If the consumer fails, its partitions are reassigned to another instance, which continues in the same order.

This has two consequences. First, all events of one execution are processed sequentially, including parallel Loop Over iterations. Iterations wait for each other instead of conflicting on writes. Second, a message that keeps failing after the queue’s own retries is moved to the dead-letter queue, the standard handling for poison messages, instead of blocking the events behind it. 

We deliberately kept the consumer in the same service that emits the events: it reduces operational overhead and avoids an extra service-to-service hop, while the partition key still guarantees per-execution ordering.

The Segment Tree

Now we know what we need: a data structure that tells us which events happened within a given iteration.

We call this structure a Segment Tree. The name comes from competitive programming, where a segment tree answers range queries over an array. Ours does not answer range queries: it splits the event log into segments of the execution path and records how those segments nest. Treat the name as a loose analogy, not the textbook algorithm. We will build up to it step by step, writing first about approaches that did not work.

To tell which events belong to one iteration, we introduced flowID: a path from the root through every gateway passed on the way and the branch or iteration taken. Events that share a flowID belong to the same iteration. We show the exact format later in this section. 

Our first attempt built on this: matching the longest common prefix of flowIDs. A shared prefix tells us which gateways were visited during an execution and lets us group flows under the gateway they came from.

This approach simplifies runtime as no extra action has to be taken. At the same time, it makes rendering more complex as change on one gateway has to recalculate all “following” outcomes and gateways which can include loops when one loop iteration is changed. This caused problems with Loop Back ambiguous boundaries. The boundary problems of Loop Back gateways are described below. We ditched the idea because it conflicted with our goals:

• Keep it general and flexible in case of future reworks
• Get it right (or close to being right) the first time 
• Avoid workflow metadata migration
• Keep this logic out of the frontend, to minimize coupling between them in case new consumers would appear

But what about the data structure? How did we actually build it? Zooming out, we looked for a familiar model and found one in interpreters.

When an interpreter calls a function, it pushes a call frame that keeps the callee’s local variables separate from the caller’s. When the function returns, the frame is popped. Recursion works because every invocation gets its own frame. Our workflow has the same shape:

With that idea in mind, we started prototyping and performing “paper executions” to see where our ideas fail before we code them. Instead of call frames we used Scopes, and instead of function names the gateways on the current path.

When workflow execution starts, we create the first Scope object named root and events are assigned to it. At this moment, the scope object is an imaginary container with an array where events would be stored.

During execution when we encounter a gateway, we modify the scope stack depending on which gateway type we reached: 

• For a Split gateway, we create a new Scope, put that object at the top of stack and link events that happened within
• For Merge, we end a particular scope and pop it from top of stack. Scope that is currently on top of the stack is considered an active one.

Based on that, we have some constants:

• We always have at least one scope on scope stack, as we have root scope added at the beginning
• We won’t have more “pop” than “push” operations, as we do validation of a workflow during publishing and in execution it’s already valid. This ensures that during the whole execution of a workflow there will not be a moment when we have more merge gateways executed than split gateways
• Each event belongs directly to exactly one scope

To validate this idea we started drawing examples of different workflows and stepping through them, to see if we could spot any issues or edge cases. The basic cases worked, so we moved on to loops.

Soon enough we realized that we have a problem with Loop Back. Under the hood Loop Back is context-less and just redirects execution flow to another outcome. We might run into it and jump up in the workflow and then go down different paths – never coming back to this gateway. In such a situation, it's hard to explain where iteration ends.

How do you present events before Loop Back and how do you present events after it?

To simplify it, we decided to treat all events before Loop Back as iteration 1, but this doesn’t solve the problem with rendering information on the UI. How should we determine where iteration ends? We only know iterations that “led” to Loop Back and triggered it. 

This solution makes it possible to have events scattered between different Scope objects. This makes it impossible to quickly decide which events belong to which Loop Back iteration. 

Pushing this job to the frontend to iterate through Scopes couples the data layer with the presenting layer. This makes frontend logic way more complex, contradicts making our design future-proof, and would block advanced pagination of events in the future.

On the other hand, Loop Over has internal split and merge gateways that distinguish events that are run before and after this loop. On top of that, each iteration runs in parallel, in an isolated scope so each can have a separate Scope object and causes no problems. Now the number of scopes created on a Loop Over gateway is equal to executed iterations.

We don’t need Scope objects at all: the flowID already records every Split and Loop Over on the current path. Since those gateways isolate their events by construction, only Loop Back needs extra state — and that state is just a pass counter. We append it to the flowID and increment it whenever a Loop Back fires, so a counter greater than one marks a repeated pass. A pass ends when the next Loop Back fires or the execution ends. Every event in between carries the same counter, so the boundary is no longer ambiguous.

Internally Loop Over is called for_each.

Example: flowID: root(1)>for_each:gateway_1a2b3c...>iteration_0(1)

root(2)>parallel_gateway:gateway_4d5e6f...>edge_7a8b9c...(1)

Each flowID has three kinds of parts: the root part, the gateway, and the branch or iteration (the flow). To make our lives easier we copy the last gateway ID into a separate field called originID and extend events with that field. This allows us to build up an execution tree structure with parallel execution and makes it easy to group events into subflows that are correlated with specific gateways. originID can be gatewayID or triggerID if no gateway was found before. For the flowIDs above, originID is: for_each:gateway_1a2b3c... and parallel_gateway:gateway_4d5e6f...

Each outcome saves to a log: flowID and originID. By appending those to events, we can easily rework the index in the future if needed without changing original workflow metadata, because the index is stored in a separate database object.

Structure

We started with an imaginary Scope object and managed to keep the same behavior while replacing it with modifications to the flowID. We distinguish between segments and flows. Their details are presented in the table below:

To make this concrete, take the workflow from Figure 2. It starts when a form with two attached files is submitted. Outcome A is an approval task: “Approve contract”. If the reviewer rejects it, a Loop Back sends execution back to A, at most three times. 

Once A is approved, a Loop Over runs outcome B, “Apply Metadata”, for each of the two files in parallel. In the run shown in Figure 8, A is rejected on the first pass and approved on the second, so its events end up with four distinct flowIDs. Panel B lists them in order, and panel C shows the tree they build.

How is the Segment Tree built?

Once we know all the details on the structure, let’s dive into the builder algorithm. When a workflow is triggered, we initialize a Segment Tree with an initial segment with id root, type LINEAR and add the first flow of id root(1), type INITIAL and empty childSegments array.

Segment Tree after the workflow is triggered:

{
    id: "root",
    segmentType: "LINEAR",
    flows: [
        {
            id: "root(1)",
            flowType: "INITIAL",
            originID: "trigger_id",
            childSegments: []
        }
    ]
}

When execution enters a branch or Loop Over iteration, the tree is extended with a structural part and new flow path. On the other hand, when Loop Back continues, the trailing counter of the current flow is incremented. Both operations are done by modifying the flowID.

1. Loop Over / Split: val newFlowID = currentFlowID + ">" + gatewayID + ">" + edgeID + "(1)";
2. Loop Back: val newFlowID = incrementCounter(currentFlowID)

When the next outcome is started, we read the current flowID from context and ensure such flow exists already in the tree. If yes, nothing happens. Otherwise, new flows or child segments are added. 

We do this in a recursive manner. For linear nodes, we either merge into existing flow or build a flow subtree. For gateway nodes, we either map flows to child segments or create a linear child and build subtree with incoming flow. For example both flows edge-a(1) and edge-a(2) map to child segment edge-a. To make it easier to debug and reason about, we return new records / objects rather than mutating the existing tree in place.

Rendering on the frontend

 We talked a lot about technical details of the algorithm, but the part about rendering data on the client is also worth mentioning. First, we retrieve segment data. Later we parse it, to find nodes and decide which nodes need selectors. Those will be used to choose different iterations.

For linear segments with multiple flows we add a selector on Loop Back gateway, as multiple flows mean multiple iterations, and for Loop Over gateways we save segmentIDs to have correct data source as gatewayID may not be unique which would make workflow traversal nondeterministic. 

Next, we calculate the range for each selector. By default we show the last index on each selector as this data would be the most recent one.

Finally, we derive relevant events and render UI.

Execution events say which flowID they belong to. Segment Tree explains how flowIDs relate to Loop Back passes, Loop Over iterations, and nested selector instances. The user sees one coherent path instead of every event at once.

Conclusion

As a result of our work with loops in Box Automate, customers can observe progress, track workflows, and self-diagnose which outcome fails. This is one of many engineering problems that we solved in Box Automate — and for me, personally, one of the hardest and most satisfying. 

We managed to resolve problems with loop iteration preview while building opinionated solutions which do not close doors for future ideas and minimize cost of potential changes by building something that resembles a database index. Like an index, the Segment Tree makes reads cheap at the cost of extra storage and extra writes, and it can lag behind its source of truth. If the history consumer falls behind, the UI briefly shows an incomplete execution until it catches up. 

In return, the event log itself never changes shape. A new view of an execution is a different walk over the tree plus a filter on flowIDs, not a schema migration - and if the tree ever needs a different shape, we can rebuild it from the events we already store.

This post was about sharing knowledge and showing how workflow-like systems start from an idea, and how engineers work their way through issues to resolve them. This whole project was not a solo endeavor, but a group effort from members of the Platform and Experience teams on Box Automate. I’m grateful for your help, and for the time we’ve spent together on brainstorming and later implementation. In alphabetical order: Adam Kaniasty, Aliaksandr Karaliou, Lukasz Kielbus, and Mustafa Bas.

If you're interested in learning more, we're always happy to help. Contact us here.