Troubleshooting scripts or applications executed by Cerberus Events and Scheduled Task Actions

One powerful feature of Cerberus is the ability to have Event Rules and Scheduled Tasks execute scripts or run applications. However, sometimes getting the actions to execute correctly can be a challenge. This guide covers some of the most commonly encountered errors and how to troubleshoot them.

Things to be aware of

The Cerberus Service Account MUST have permission to execute the script or application.

You may find executing the script or application 'manually' outside Cerberus works fine, but then using Cerberus results in an error. Bear in mind that when you test outside Cerberus, most likely you're using the account you logged into the server with. This is generally not the Cerberus Service Account.

To check what account is running the Cerberus service:

• Open 'services.msc'.
• Select 'Cerberus FTP Server'
• Right-click and select 'Properties'.
• Click the 'Log on' tab on the window.

You may see 'Local System' selected. In this case, you need to ensure user 'SYSTEM' can run your script or application. You may see a specific local or domain account specified, especially if you are running Cerberus 12 or above. In both those scenarios, you'll need to ensure that the account can run the script or application.

Testing tip: Log into the Windows server or PC running Cerberus using the Cerberus service account credentials, then test running your script or application. If you get it to run correctly by executing it 'manually' outside of Cerberus, then there is a good chance the Cerberus Event Rule or Scheduled Task will be able to run it. 

Non Windows scripts must be run by their host application.

If you are looking to execute a script that is not a Windows script, like a Powershell script, for example, you must either set up the path to the application as an executable Event Target (e.g. for Powershell, the path to the application will be something like C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe) or set up the path to the Windows command line (cmd) as an Event Target and specify the application in the execution command. (e.g. "powershell.exe -ExecutionPolicy Bypass -File "C:\SCRIPTS\HelloCerberus_test.ps1"") 

Specify the hostkey if using a scriptable client like cURL, WinSCP or PSCP

With scriptable clients, such as cURL, WinSCP, or PSCP, you must specify the hostkey in your client command, regardless of whether that's in the 'Command' field of a Cerberus action, or within a script calling one of these clients. If not provided, the client will expect someone to manually accept the hostkey. Since there is no way to do that with an automated script or Cerberus-triggered application, the action will 'hang' and you will see a 259 error once Cerberus times out the action execution.

Using Cerberus Logging to troubleshoot

Turning on debug logging on the Cerberus User Interface may help troubleshoot your Cerberus action execution. Please note that Cerberus logging will NOT display logging of applications called within a script. For that, you may need to enable logging on to your script. See below for details.

To turn on debug logging in the Cerberus User Interface:

• In the Cerberus User Interface, navigate to 'Log'
• Click the 'bug' icon to turn on debug logging
• Once you are finished testing, click the 'bug' icon again, to turn on debug logging.

Script return codes / Adding logging to popular scripts and applications

When Cerberus runs a script or runs an executable, Cerberus will display a return code in the Cerberus logging. The return code given may indicate the script or executable was successfully run or will give any number of error codes. If Cerberus was not able to execute a script, you will see a failure return code in the log. Error codes can vary depending on what service or application Cerberus is executing and many are very generic, which will require further investigation. We cannot list all the possible error codes here, so you may need to look into the documentation for your chosen application for more details.

Successful Script Execution

If Cerberus executed the script successfully, you will see the following entry in the Cerberus log:

executed with return code: 0 (success)

This simply means that Cerberus was able to run the script with no permissions errors. It does not necessarily mean the script executed with no issues. Even if you see this return code, you may still need to dig into your script to ensure it's performing its required actions. Enabling logging on the script itself may help reveal issues within the script that Cerberus did not report. See below for some tips on how to activate script logging on some popular packages. You may need to consult the documentation for your chosen application to learn how to start logging.

Sample error codes common to several scripting applications

There are a few error codes that are common across different applications:

return code: 1 (failure)

return code: 2 (failure)

• Cerberus may not have permission to execute the script
• The target path that the script is supposed to be operating on is incorrect or Cerberus does not have permission to see the specified path

return code: 16 (failure)

• Permission issues with the folders targeted by the script (especially if you are using Robocopy) 

return code: 255 (failure)

• Cerberus can't find the script file to execute or doesn't have permission to execute it.

return code: 259 (failure)

• The action is "Still Active". The script process was still running after the wait for process time elapsed. Basically, the process was running for more than the max wait time set in the action
• You may be trying to execute a non-Windows script directly without specifying the application to execute it with. For example, trying to execute a Powershell .ps script that you have directly set up as an Event Target. In this example, you will need to either use a Powershell Event Target or a Windows Command Prompt Event Target and specify "powershell.exe" in the execution command.
• A 259 error may mean that the service account running Cerberus does not have permission to execute the actions in the script.
• It can also mean the script is waiting for some interactive input. For example, if you do not specify the hostkey in an action calling a scriptable client, the client will 'wait' looking for input approving the hostkey. Since there is no interaction with a command executed by Cerberus, this is not possible and the script will eventually time out.

Also, this error happens when the client called by the script is waiting for the interactive acceptance of the hostkey. Ensure you specify the hostkey when calling a scriptable SFTP client like cURL or WinSCP.

Error codes and adding logging to some popular applications

Windows Batch (.bat) Script

You can refer to the Windows System Error Codes when running batch scripts, which can be found here:

https://docs.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-

To troubleshoot issues with Windows batch scripts, we recommend adding logging for the script. This gives you a specific window into what the script is doing. See https://community.spiceworks.com/topic/897323-how-to-create-logs-while-running-batch-script

Powershell

return code: 267 (failure)

The directory name is invalid

For example, ensure you wrap paths in double quotes and not single quotes.

-ExecutionPolicy Bypass -File "D:\GSRSCRIPTS\pgpdecryptcerberus.ps1"

return code: 1 (failure)

1 (failure) can mean "Incorrect function". Check your Powershell call for any errors.

Powershell also uses the Windows System Error Codes, which can be found here:

https://docs.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-

To add logging output to your Powershell Script, see http://woshub.com/write-output-log-files-powershell/

cURL

return code: 3 (failure)

URL not properly formatted. Check the FTP/SFTP URL the script is calling and ensure it's properly escaped and formatted.

return code: 26 (failure)

The error code 26 means "There was a problem reading a local file or an error returned by the read callback." Check the path to the file you are calling and also ensure the Cerberus service account can see the file.

For a good summary of other cURL errors, see https://curl.haxx.se/libcurl/c/libcurl-errors.html

To export logging to a file, use the --trace or --trace-ascii options with a given file name to log to, like this:

curl --trace trace.txt www.haxx.se

WinSCP

Refer to this link for WinSCP error codes:

https://winscp.net/eng/docs/sftp_codes

Refer to this link to enable logging if you have Cerberus executing WinSCP commands:

https://winscp.net/eng/docs/logging_xml

PSCP

While using the PSCP scriptable client to transfer files is a great tool, specifying the command call in your script can be tricky. To assist, here is a sample of a successful PSCP execution command:

-hostkey 70:28:75:16:2c:d4:09:14:49:13:7b:1a:af:35:a0:02 -sshlog c:\ftproot\sshlog.txt -P 22 -scp -noagent -batch -i "C:\ProgramData\Cerberus LLC\Cerberus FTP Server\certificates\1082631.ppk" someuser@192.168.1.158:/sourcepath c:\ftproot\backup.bak

Success requires the following for successful key authentication:

• -hostkey > You must specify the hostkey.
• -sshlog > This writes an execution log. This is very handy for troubleshooting.
• -P > Specify the port (SFTP port 22 in this example).
• -SCP > Specify you wish the transfer to be SCP if you want to use SCP instead of SFTP.
• -noagent > disable use of Pageant, the interactive PSCP client.
• -batch > disable interactive prompts.
• -i > Specify path to private key for public/private key authentication.
• someuser@192.168.1.158:/targetDirectory > the remote system user name, IP address or hostname, target directory
• c:\ftproot\backup.bak > the file and destination absolute path.

The source or destination path should be in quotes if there are spaces in the path

See https://the.earth.li/~sgtatham/putty/0.60/htmldoc/Chapter5.html

Logging the output of SCP (and similar commands) doesn’t work the straightforward way of a simple redirection. SCP uses direct terminal codes and if it detects a redirection, it doesn’t output anything.

scp $REMOTE_SERVER:~/test/test.file /dev/null

# fetch the file, but don't store it locally
script -q -c "scp $REMOTE_SERVER:~/test/test.file /dev/null" 2>&1 | tee scpLog
# replace carriage return with newline
sed -i -e 's/\r/\n/g' scpLog