Logs Related Training & Support Archives

A community member faced challenges building a workflow in ControlUp to combine two indexes dynamically for a nightly dashboard summarizing the previous day’s environment status. The primary difficulties arose from limitations in the workflow engine compared to PowerShell scripting, particularly related to looping constructs, array size management, and efficient data handling. Attempting to accumulate all data into a single large array caused performance degradation, and efforts to split data handling into groups of 100 records ran into issues because workflows lacked a traditional for-loop or the ability to reinitialize variables inside loops. ControlUp experts clarified that appending to arrays in loops is generally inexpensive, but performance issues often stem from JSON modification or calculation nodes that operate on the entire array each iteration. Instead of keeping large arrays in memory, a recommended pattern is to flip the subflow logic: allow the subflow to write records directly to the target custom index and return no data to the main flow. The main flow can paginate data requests using Get Custom Data (which supports paging via from/size parameters), looping over pages where each subflow processes one page end-to-end, avoiding large in-memory arrays and reinitialization challenges. To emulate traditional loop behavior, a “fake while loop” workaround was suggested by creating a dummy array representing the number of iterations desired (e.g., 20), then looping over that array. Combined with OpenSearch’s partitioning (`include > partitions` and `include > num_partitions`), this approach divides large datasets into manageable buckets processed in separate iterations. The configuration can be sized to match the environment—for example, capturing 4,000 records per iteration across 20 iterations to cover up to 80,000 results, adjustable as needed. A practical example from the discussion involved combining device data from two ControlUp indexes: one filtering Windows devices by specific event logs, and the other fetching device information filtered by the extracted device IDs. Merging these on the device ID with the "Preserve structure from: both" option enabled results in a unified dataset ready for further processing or custom indexing. This workflow ran efficiently in about four seconds on test data. To control data volume, filtering steps were added to isolate devices exhibiting significant event counts (e.g., more than 5 events in 24 hours), which reduced processing load while focusing on impactful network issues. The conversation highlighted the current lack of a native "for" loop feature in ControlUp workflows, but noted this capability is planned for future releases. Meanwhile, users can leverage paging, subflows writing directly to indexes, and OpenSearch partitions to manage large datasets effectively. Per-iteration writes to custom indexes are feasible if API rate limits and retry logic are properly handled. For large data reads, Get Custom Data is preferred over Get Data Index because of its paging capabilities. These insights provide a reliable framework for users needing to aggregate and analyze dynamic multi-index data in ControlUp workflows while balancing performance and complexity. For additional guidance on ControlUp workflow nodes and custom data handling, refer to ControlUp’s official knowledge-base and documentation at https://docs.controlup.com and https://cuacademy.controlup.com.

Read the entire article here...

---