Multiple users discussed issues with triggers, specifically the event log trigger causing an error message and items getting stuck in the monitor queue. Suggestions were made to check the machine column in the incidents pane, clear the queue or restart the monitor service. It was also mentioned to check and possibly increase the MaxAutoExecQueueSizeTotal and MaxAutoExecQueueSizeTarget settings. A ControlUp team member offered to look into the issue further.
Read the entire ‘Troubleshooting Triggers in ControlUp’ thread below:
Been having issues with triggers and found this in the event log. Is this expected? Can anything be done with this? Error: The maximum number of invoked automation jobs by a single automated followup action is exceeded. The operation failed.
What kind of trigger?
Scheduled?
in this case it was an event log trigger
but I also see others
and it says the operations count over current limit
can I increase this?
it implies you have over 10k actions queued. I don’t think you want to increase that limit. You’ll probably want to understand why they are queued
what version are you on?
I have lots of triggers haha
9.0.1546
can you run this on that monitor?
“`$files = get-childitem "c:\program files\smart-x\controlupmonitor" -filter "monitor.dll"
$files | foreach-object {import-module $_.fullname}
Open-controlupmonitor
Get-cuaaqueue“`
actually the first 2 lines might be generating some errors in v9. You should be able to ignore those
the open-controlupmonitor isnt recognized as a cmdlet
yeah its written from memory. Let me get on a monitor and correct the script
ah ok thank you@!
“`$monPath = ((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\cuMonitor").imagepath).replace("\cuMonitor.exe",$null).replace(‘"’,$null)
Get-Item "$monPath\powershell.dll"|import-module
Open-ControlUpMonitor
Get-CUAAQueue“`
oh wow
yeah def an issue
I think I see my problem!
well, the queue shouldn’t be a problem generally speaking
So if items gets stuck there’s a reason for it
but thats too many I think
I candisable this trigger
What is listed as those items queue? Target or monitor?
monitor
Is the list growing or decreasing if you run it multiple times? Items near the start should go away if the queue is processing
it doesnt look to be processing
can we delete the queue
they are all the same and not relevant
and the most recent one shown is 4/1
clear-cuaaqueue
or something like that
nice
thank you!
Keep an eye on it. Because the queue shouldn’t grow. Even if you use silly amounts of triggers. Items getting stuck on the monitor queue means the monitor has over 1000 actions running across multiple agents. Which seems unlikely if the oldest is 4/1
well the oldest is 3/30
the most recent was 4/1
That’s not better haha
Also the events trigger shouldn’t generate
"The maximum number of invoked automation jobs by a single automated followup action is exceeded. The operation failed"
I suspect that the machine wasn’t passed to the trigger. I think you should be able to see that in the incident pane for that event
ok
im gonna do a few more tests after clearing that queue
unless you think its unrelated to the event log trigger
You can see the size of the queue in the web console, starting in v9 by the way
The same view will also show you if the monitor is healthy in general but also AA health in general
oh sweet!
hmm where do you see that?
I searched for the monitor, but I dont see what you see
As for the event trigger. I’m not sure if that is just the event trigger itself or event triggers in general. But an event trigger shouldn’t cause
"The maximum number of invoked automation jobs by a single automated followup action is exceeded. The operation failed"
That is either because of a bug
Or if a trigger is trying to run more than MaxAutoExecJobsTrigger actions. Which in v9 is 5000. I don’t think you added more than 5000 actions to a single trigger.
In v9, unless you set a schedule trigger on the process table and set no filter, you’re very unlikely to hit that error
If you get a chance. Take a look at incidents
Double click on windows events
Check trigger and uncheck everything else
Find your trigger that caused the error message. Double click anywhere except the trigger name itself. Clicking on the trigger name will open the trigger settings for that trigger. We don’t want that.
Do those incidents contain something in the machine column?
is this where you mean?
yeah
So machine isn’t empty. :thinking_face:
my queue is building again
should it process quickly? is this normal?
Depends on how long the scripts take to complete
so it could be queueing from the action that is a script?
so the script action runs in seconds
Yeah if your script action takes 10 minutes to complete for example
if I do it manually
just ran it manually and it was about 5 seconds
Im reviewing the event log and there might be something going on with the script when run as the monitor context
ill do some digging there
Key: HKU\S-1-5-20\Software\Smart-X\ControlUp\MonitorSvc
Value: MaxAutoExecQueueSizeTotal
Data: 1000
Type: DWORD
it hasn’t changed so idk
Assuming the queue is set to monitor
yep
I’m not sure if clear-cuaaqueue clears the "list of running actions" or just the queue. If that makes sense.
If I use a themepark example. Clearing just the queue without clearing whatever is stopping the people on the ride currently from being unable to get out of the ride, won’t do anything. Because the queue will just fill up again
MaxAutoExecQueueSizeTotal expands the amount of people that can go on the ride. My analogy is breaking down a bit
MaxAutoExecQueueSizeTotal can be set while the monitor is running. No need to restart
ah ok
well i went ahead and set it
but all my items are still in queue
so its still getting mroe items in there
as you said, the line was cleared then filled up again
can you set MaxAutoExecQueueSizeTarget as well? Set it to 5
It controls how many actions each target (agent) can run concurrently
Default is 1
ok done
oh thats a great option
I could just try restarting the monitor
service
yeah, that’ll clear the queue for sure 🙂
would that clear the queue or
oh ok
its all in memory
got it
CC: @member for VSO 193579. No immediate action required. Just an FYI since I’m on PTO for the next 2 week.
In case you need someone that can reproduce the stuck in queue issue
so it seems like things are still stuck here
so I guess Ill try to restart the monitor and see if the queue fills up again
Continue reading and comment on the thread ‘Troubleshooting Triggers in ControlUp’. Not a member? Join Here!
Categories: All Archives, ControlUp Scripts & Triggers