Efficiently Combining and Processing Large Index Datasets in ControlUp Workflows Without Native Loop Constructs

A community member faced challenges building a workflow in ControlUp to combine two indexes dynamically for a nightly dashboard summarizing the previous day’s environment status. The primary difficulties arose from limitations in the workflow engine compared to PowerShell scripting, particularly related to looping constructs, array size management, and efficient data handling. Attempting to accumulate all data into a single large array caused performance degradation, and efforts to split data handling into groups of 100 records ran into issues because workflows lacked a traditional for-loop or the ability to reinitialize variables inside loops.

ControlUp experts clarified that appending to arrays in loops is generally inexpensive, but performance issues often stem from JSON modification or calculation nodes that operate on the entire array each iteration. Instead of keeping large arrays in memory, a recommended pattern is to flip the subflow logic: allow the subflow to write records directly to the target custom index and return no data to the main flow. The main flow can paginate data requests using Get Custom Data (which supports paging via from/size parameters), looping over pages where each subflow processes one page end-to-end, avoiding large in-memory arrays and reinitialization challenges.

To emulate traditional loop behavior, a “fake while loop” workaround was suggested by creating a dummy array representing the number of iterations desired (e.g., 20), then looping over that array. Combined with OpenSearch’s partitioning (`include > partitions` and `include > num_partitions`), this approach divides large datasets into manageable buckets processed in separate iterations. The configuration can be sized to match the environment—for example, capturing 4,000 records per iteration across 20 iterations to cover up to 80,000 results, adjustable as needed.

A practical example from the discussion involved combining device data from two ControlUp indexes: one filtering Windows devices by specific event logs, and the other fetching device information filtered by the extracted device IDs. Merging these on the device ID with the “Preserve structure from: both” option enabled results in a unified dataset ready for further processing or custom indexing. This workflow ran efficiently in about four seconds on test data. To control data volume, filtering steps were added to isolate devices exhibiting significant event counts (e.g., more than 5 events in 24 hours), which reduced processing load while focusing on impactful network issues.

The conversation highlighted the current lack of a native “for” loop feature in ControlUp workflows, but noted this capability is planned for future releases. Meanwhile, users can leverage paging, subflows writing directly to indexes, and OpenSearch partitions to manage large datasets effectively. Per-iteration writes to custom indexes are feasible if API rate limits and retry logic are properly handled. For large data reads, Get Custom Data is preferred over Get Data Index because of its paging capabilities. These insights provide a reliable framework for users needing to aggregate and analyze dynamic multi-index data in ControlUp workflows while balancing performance and complexity.

For additional guidance on ControlUp workflow nodes and custom data handling, refer to ControlUp’s official knowledge-base and documentation at https://docs.controlup.com and https://cuacademy.controlup.com.

Read the entire ‘Efficiently Combining and Processing Large Index Datasets in ControlUp Workflows Without Native Loop Constructs’ thread below:

I’m struggling with the logic of a workflow with the limitations given. In powershell this would be very easy to do, but in workflows its proving more difficult.

I have 2 indexes that I want to combine to be able to make a dashboard table out of them, and the number of rows will change around depending on what is happening in the environment. I plan to run this nightly and get a summary of the past day.

I first ran into problems when the array seems to have grown too large and bogged things down, so I shifted the movement from the queries to an array over to a subflow and got that working right, now the main flow just had to drop everything from the subflow into the array, and once it processed everything it would then drop it into the custom index.

that still didn’t work well, so now looking at other ideas. I tried an If statement in the loop, so if it hit 100 it would reinitialize the dataholdingarray, but for whatever reason you can’t do that from within a loop, even if the variable was created outside of the loop. Frustrating but okay. Then make groupings of 100 and just feed each grouping into the array.

The loop had another issue, there is no For type loop I can find (for x=0 to x=100 do this) so that is also a no.

Tried making an oldschool loop with if statements, but can’t loop an if statement back around until the array count is over a certain amount because you can’t do cyclical connections.

I can’t just build out 5 groupings, because what if things happen and then there needs to be 8 groupings, or 2 groupings?

My next thought is to push things outside of ControlUp, maybe in a google spreadsheet and pulling them back again, but that really feels like it would have much MORE overhead and pieces to break.

If I could just plop a powershell script in there to work with the data this would be dead simple. Even if the powershell was limited to only object input/output, no access to anything other than what it is fed directly.

What other paths can I try with this? (I’ve been resisting just writing to the custom index on every iteration because I would think that has to be costly to the server)

Could you share the names of the two indexes and a sample of their structure (a few example records from each)?
Also, how exactly do you want the data combined?
• One row per device: records from both indexes are matched/merged based on a common identifier, such as the device ID.
• Rows underneath each other: all records from both indexes are simply combined into one dataset, regardless of device.
• Something else: for example, aggregating multiple records into a single row per device.
Knowing the index structures and the intended relationship between the records should make it much easier to determine the right approach.

Hi,

> for x=0 to x=100 do this
We will add this functionality very soon.

Rico asked good question. Could you share which indexes do you try to use and samples of their data? Also please share approximate volume if you know.

Thanks for the detailed writeup Mike, really useful. Rico’s questions hit the key point, but I dug into the engine side in the meantime and there are a few things that should unblock you regardless:

1. The slowdown was probably not the array itself. Appending in a loop is cheap. What grows expensive with every iteration is a JSON Calculate or JSON Modify Object node inside the loop, or any node that references the whole array in its inputs. If you had one of those, that’s your culprit.

2. Flip the subflow: a subflow runs as its own execution with its own memory, so have it write its records to the target index directly and return nothing, with the main flow just orchestrating. Get Custom Data supports paging (from/size), so the parent can loop over page numbers and each subflow handles one page end to end. No big array anywhere, no need to reinitialize anything.

3. Per-iteration writes are fine in principle, the real constraint is your org’s API rate limit. Set retries on the write node (default is one attempt), and let us know your rough nightly row count so we can sanity-check the volume.

One node-choice note: prefer Get Custom Data over Get Data Index for large reads, only the former can page past the first batch.

Share the structures and the row count and we’ll sketch the exact flow.

Hey Mike, I know we worked on this last week, I’m not sure why I didn’t think of this solution then.

1. Retrieve index A. Devices with a certain Windows event from win_event_log.
2. Extract the device IDs.
3. Retrieve index B. _devices filtered on device ID.
4. Merge results from 1 and 3 on device ID again. `Preserve structure from: both` is important here. It causes the arrays to be merged.
I haven’t sent this data to a custom index yet. But the entire workflow took 4 seconds to run on your data set. See the combineNetworkData workflow in your tenant.

If you want to ensure that you capture the entire environment, even if the results might be very large, you can use a fake while loop.
1. By creating a dummy array of lets say 20 items.
2. Loop over that dummy array – this gives you a 20 instance for loop.
3. Use the include > partitions and include > num_partitions OpenSearch mechanism to cut the entire result set into 20 buckets.
“`”terms”: {
“field”: “_device_id.keyword”,
“size”: 4000,
“include”: { “partition”: {{IterationHelper.result[i]}}, “num_partitions”: 20 }
}“`
This won’t be needed when the workflows team introduces a for loop mechanism

The configuration above will capture at most 4000 results per iteration and does 20 iterations. So if your environment is larger:
• Increase the size from 4000 to something bigger
• Add more iterations, remember to increase both the dummy array and the num_partitions value
• Consider if the use case needs to cover more than 80k results

Thanks, taking a look, meanwhile I was able to get things going, I was trying to combine devices with computers that have a specific event in win_event_logs, so combine them per unique ID.

I was able to get it to complete by having another loop that just filtered out so only devices with more than 5 of those events in 24 hours got moved to a different array, then went through that other small array to grab the records.

As this learning is more for what devices are having network problems, I am not going to be missing much by not looking at ones dropped only a few times a day, as I want to know the ones that are dropping hundreds and thousands of times a day.

Dennis your way is far better than mine, time to start learning more on the other blocks.

Continue reading and comment on the thread ‘Efficiently Combining and Processing Large Index Datasets in ControlUp Workflows Without Native Loop Constructs’.  Not a member? Join Here!