Since go-live, we have experienced four core Process Scheduler issues. They are:
1) Remote Call
2) Core dumps related to excessive number of log directories
3) Crystal Reports doesn't run
4) Process Instance > 1M and problems with the proposed workarounds

Remote Call
Certain Application Engine processes that called COBOL would intermittently generate an "unable to open file" error at the point where they call the COBOL procedure and crash. Restarting these processes was problematic because a simple restart would skip the COBOL section of code -- restart had to be performed by clearing the temporary tables and resetting the run control at the database level. After several weeks of pain, PeopleSoft delivered new Application Engine executables and the problem was resolved.

Core Dumps
After operating about three days, the Unix process scheduler would start core dumping every minute and could not keep up with the load. This was related to how the process scheduler manages log files -- it creates a separate subdirectory to store log files for every job that runs. There is a Solaris limitation of approximately 32,000 subdirectories in a single directory. After about three days, around 32,000 jobs run creating 32,000 subdirectories. Solaris wasn't able to accomodate additional subdirectories, and the process scheduler would crash. A workaround was implemented outside of PeopleSoft to purge these log subdirectories on a nightly basis and the problem was resolved.

Crystal Reports
Crystal Reports currently do not run. Resetting the process scheduler allows a few Crystal Reports to run, but inevitably one job will hang up and no more Crystal jobs will run. This forces us into manual workarounds to do core business tasks like printing AP checks. There is a PeopleSoft incident that addresses this issue, and the fix is included in the latest PeopleTools upgrade. Preliminary testing indicates that the PeopleTools upgrade will indeed solve the problem, and we're working toward implementing it.

Process Instance > 1M
Due to the large number of processes we run, the process instance counter has rolled over 1 Million in only three months. When this happened, Application Engine jobs ceased to function. Since application engine jobs are necessary for almost every core batch process in PeopleSoft, the impact was immediate and severe. Once again a PeopleTools upgrade was identified as the ultimate solution, but it takes time to get a Tools upgrade moved to Production. For the interim, PeopleSoft offered two workarounds: 1) Restart the process instance counter at 1; or 2) Re-configure process scheduler so that Application Engine jobs run outside of Process Scheduler on the command line.

After some preliminary testing, we determined that restarting the process instance counter at 1 was problematic. Many tables are keyed by process instance, and these tables would have to be purged before this would work. Since the impact to the business was so severe, it was decided that there wasn't enough time to do the analysis, build a solution, test it and put it into production.

So by default, we implemented the workaround to reconfigure process scheduler so that AE jobs run on the command line. This was a simple configuration change that could be reversed if the problems got worse. Preliminary testing was successful, so we left the configuration change in place. This allowed the scheduled jobs to run.

The one problem with the workaround was that running Application Engine processes on the UNIX command line cannot invoke LDAP for authentication, so end-users were not able to manually run Application Engine jobs. The workaround to the workaround is to manually create a PeopleSoft (non-LDAP) password for users that run Application Engine jobs. The user must remember to use the new PeopleSoft password and not their Network password, otherwise the PeopleSoft password gets deleted and Application Engine jobs don't run any more.

Comments (0)

Write comment

Name

Email

Website

Title

Comment

Write the displayed characters

Add Comment