ERP Associates, Inc.

Comment (0) Read More...

"A blog containing development tips I have learned through the years as a PeopleSoft developer."

Comment (0)

Even with a modern packaged application like PeopleSoft, it's possible (and actually quite likely) that performance problems exist and won't show up until after go-live when all of your production users are actively hitting the application.

Why? The reasons have less to do with poorly written software (PS/Oracle usually writes relatively efficient code), and more to do with the complexities of the application architecture. In a typical implementation, IBM HTTP Server handles all HTML traffic and hands off the work to IBM WebSphere. WebSphere manages sessions and communicates with the BEA Tuxedo Application Server which handles the business logic layer and communicates with the database server. The Operating System(s), system hardware, SAN's, load balancers, clustering software, network, etc., all play a part in performance.

Each component has its own tuning parameters and if any one of these components are not tuned appropriately performance suffers across the board. That's why it's a good idea to test your performance.

Fortunately, performance testing doesn't have to be a long, drawn out ordeal nor does it have to be expensive. It just requires a plan, a performance testing tool, the right people on the performance testing team, and a way to capture performance data for each component.

The performance testing plan should provide the framework and scope of the testing effort. It should outline the purpose of the testing, define basic terms like "Concurrent User", and include what tests will be conducted, what will be measured, and how you'll know if you're successful. I generally like to name team members in the plan along with contact information just so there's no confusion about what will be expected.

I've used Segue's Silk Performer in the past and it works very nicely for PeopleSoft performance testing. But if you don't have a commercial tool, Jakarta JMeter from the Apache Software Foundation (http://jakarta.apache.org/jmeter/) is an open source tool that has all of the features you'll need.

When you're writing the performance testing scripts, don't try to write a script for every system component. Keep it to just the most commonly used on-line activities. Also, don't try to test the "worst case". Performance testing tools can generate way more transactions than a person could ever enter. Be sure your performance testing tool at its peak won't generate more transactions than you'll find on a busy day in the database. A few quick queries should show if you're in the ballpark. I have heard that PeopleSoft benchmark reports define a concurrent user for Financials as someone who enters one transaction every 5 minutes. In my opinion, that's one very busy person.

Assembling the performance testing team is generally a matter of talking with the folks that already support your application. In addition to your PeopleSoft Application Administrator, you'll want to have a System Administrator to monitor the OS and the Kernel; a DBA to monitor the database; and a Network Engineer to monitor the network. If you feel like you're weak in a particular area, go find someone in your organization that can help. Or if you think you can cover multiple areas, the team can be smaller.

Capturing performance data doesn't have to be a huge deal either. If you're running on UNIX or Linux, it's easy to redirect VMSTAT or SAR output to a file during the test for memory and CPU info. Windows provides a way to capture performance data to a file as well. I have some scripts that use TMADMIN to track # of users logged on, # of Tuxedo request, and whether or not queuing is happening. I generally use WebLogic console to watch threads and JVM heap info as the test is running. On the database, TKPROF is a great way to measure many things in an Oracle database, and SQL Profiler has a lot of functionality for SQL Server.

I generally divide my tests up so that we start with 1/3 of the target load, then 2/3rds for the next test, and finally the full load. While each test is running, I have the performance testing team on a conference call so we can discuss what's happening in real time. I've found that a 20-minute test seems to provide good information, but I don't mind cutting it short especially if problems become obvious. While the test is running, I like to log on to the system and navigate around, just to get a human's opinion of performance. I keep a performance testing narrative document up to date with the "play-by-play" info which always includes at least the time the test started, what memory, CPU, threads and JVM Heap were doing at various times in the test. This narrative also helps me make sense of all of the data the team members will be sending at the end of the test. The narrative also includes any changes that were made between tests so we can tell what worked, and what didn't.

Once we're sure the system will work at the required load, and if time allows, I like to do a stress test to find exactly at what load the system will break down. This gives an idea of the overall system capacity which is a great statistic to know for planning purposes down the road.

Anyway, that's more or less how I approach performance testing. If you have any other ideas or approaches, please share.

Comment (1)

I spent some time this summer implementing Oracle's CoreID product (now known as Oracle Access Manager) as a solution to bypass the PeopleSoft sign in screen. It works as an add-in module in Apache, IIS or IHS that intercepts requests and checks to see if the URL is protected in the Access Manager component. If it is protected, Access Manager executes the security rules as specified. Our rules redirect the user to an IIS web server which performed Integrated Windows Authentication. After the user was successfully authenticated, it redirected the user back into PeopleSoft using the original URL that the user requested. But in addition to redirecting the user back, it added the user's EMPLID (which was retrieved from Active Directory) as an HTTP header in the request.

On the PeopleSoft side, I enabled the WWW_Authenticate signin PeopleCode to retrieve the EMPLID header, look up the Operator ID, and do a Switch User to sign in as that operator. I did put some code to make spoofing the EMPLID header a bit more difficult for a wannabe hacker.

But CoreID is a relatively complicated product meant to work with multiple web applications. If you're wanting to just do Kerberos authentication with PeopleSoft to bypass the sign in screen, you might look at the following options:

• WebLogic "should" be able to do this, but the version that's delivered with PeopleSoft seems to be missing the Kerberos security policy functionality. It might be worth checking with PS to see if you can get a full version, otherwise consider purchasing it


• Investigate WebSphere or Oracle Application Server's Kerberos authentication options.


• Use IIS as a reverse proxy server (it has native Integrated Windows Authentication). Having been down this path, I know IIS authenticates the user but doesn't want to pass the username/password over to your app server. You might have to write your own ISAPI filter to make it happen.


• There are network devices that do the authentication and cache application username/password combinations. When the user hits the web site, the network device signs the user in based on the authentication so the user never sees the signin screen. Although this sounds like a maintenance nightmare, but I know of one company that is VERY happy with this solution and it's relatively cheap to deploy.


• Probably the most common approach I've seen is to write some type of ASP script that users could hit via URL which would use IIS's native Integrated Windows Authentication to authenticate the user, generate a PSTOKEN (via Component Interface) and redirect them back into PeopleSoft. Not bad if your users start from the same point in the app, but it's not so good if users expect to be able to click on any URL and get straight into the app w/ no signin.

Please let me hear from you if you've tried any of these approaches and let me know what worked and what didn't.

Comment (7)

I ran across this in one of my RSS feeds the other day. Apparently one of Oracle's partners is making demo versions of Oracle software available on-line. I thought it might be interesting to check out some of the technology and functionality could make it in to Fusion. (Yeah right, in my spare time...)

From the article:

Solution Beacon, a certified Oracle partner and a provider of Oracle resources, offers instances of Release 11i Vision for public access. Many releases are available and registration is fast and simple. Once you have filled out the required registration form, you will receive a username and a password to your email. These public instances are great tools to use if you simply want to experience EBS applications or for testing new functionalities. However, I wouldn't recommend using these for a client's demo though because instance refreshes are on as-needed basis.

Here's the quote from the site:

As a member of the Oracle Partner Network we are providing the following instances of Release 11i Vision for public access. Please note that these environments will be refreshed on a periodic basis without notice. There are no explicit or implied guarantees with regards to availability, performance or data retention. Should you have any special needs requiring a dedicated instance for a short period of time or if you experience any issues with the instances provided here please send us an email to r11ivision@solutionbeacon.com.

I don't know of any services like this for PeopleSoft Enterprise. If you do, please share!

Comment (0)

According to this ComputerWorld Article, Oracle took the opportunity to bash SAP after a strong first quarter.

Ellison criticized SAP's recent decision to retain the current version of its enterprise resource planning software, mySAP ERP 2005, through 2010 and provide new functionality through regularly issued optional enhancement packages.

"SAP has delayed the next release of its applications until 2010," he said. "They'll be two full years behind our Fusion release." Oracle is working on a new suite of enterprise applications, known as Project Fusion, which is due out in 2008.

The digs at SAP appeared to be almost personal. Ellison, noted for using strident rhetoric about rivals, even said, "SAP appears to be rethinking their strategy as they lose application market share to Oracle and confront the difficulties of moving their application software to a modern service-oriented architecture [SOA]." He also claimed that SAP CEO Henning Kagermann is talking about an acquisition strategy to augment SAP's slowing organic growth. "These are major changes in direction for SAP."

Ellison's remarks drew a response from SAP. "Larry Ellison's statements in [Tuesday's] Oracle earnings press release about SAP's product and acquisition strategy are a complete misrepresentation," said Bill Wohl, vice president of product and solutions public relations at SAP. "Since January of 2003, SAP has consistently articulated and delivered on its vision for enterprise SOA following a course of organic growth combined with strategic acquisitions. SAP offers customers market-leading, enterprise SOA applications today, while Oracle's next-generation applications exist only in PowerPoint and won't be delivered until 2008 or beyond."

Comment (0)

Oracle announced today that PeopleSoft Enterprise CRM 9.0 is generally available. According to the press release:

September 18, 2006—Furthering Oracle’s ongoing commitment to enhance PeopleSoft applications as part of the “Applications Unlimited” program, Oracle announces the availability of Oracle’s PeopleSoft Enterprise Customer Relationship Management (CRM) 9. Highlights of this release include increased use of Oracle Fusion Middleware and new enhancements designed specifically for the financial services and communications industries and usability enhancements designed to increase productivity.

Comment (0)

I ran across this on Yahoo's peoplesoft-fans group today. Apparently Oracle is making PeopleSoft documentation available from their web site. This includes the elusive PeopleTools Installation Guide. Check it out:

Archived: http://www.oracle.com/technology/documentation/psftarch.html
Latest: http://www.oracle.com/technology/documentation/psftent.html

Comment (2)

I was going through some old e-mails and ran across one from 2003 about three months after go-live on a Financials 8.4, PeopleTools 8.43 implementation. Performance testing didn't prepare us for how heavy the process scheduler load would be and how process scheduler would fail under that load.

The e-mail was one of those uncomfortable ones to the project sponsor trying to explain in simple terms what went wrong, why, and what was being done to fix it. Hope you never have to do one of these, but if you're in IT for any length of time I'll bet you do!

Oh, and just so you know all of these issues have since been resolved.

Since go-live, we have experienced four core Process Scheduler issues. They are:
1) Remote Call
2) Core dumps related to excessive number of log directories
3) Crystal Reports doesn't run
4) Process Instance > 1M and problems with the proposed workarounds

Remote Call
Certain Application Engine processes that called COBOL would intermittently generate an "unable to open file" error at the point where they call the COBOL procedure and crash. Restarting these processes was problematic because a simple restart would skip the COBOL section of code -- restart had to be performed by clearing the temporary tables and resetting the run control at the database level. After several weeks of pain, PeopleSoft delivered new Application Engine executables and the problem was resolved.

Core Dumps
After operating about three days, the Unix process scheduler would start core dumping every minute and could not keep up with the load. This was related to how the process scheduler manages log files -- it creates a separate subdirectory to store log files for every job that runs. There is a Solaris limitation of approximately 32,000 subdirectories in a single directory. After about three days, around 32,000 jobs run creating 32,000 subdirectories. Solaris wasn't able to accomodate additional subdirectories, and the process scheduler would crash. A workaround was implemented outside of PeopleSoft to purge these log subdirectories on a nightly basis and the problem was resolved.

Crystal Reports
Crystal Reports currently do not run. Resetting the process scheduler allows a few Crystal Reports to run, but inevitably one job will hang up and no more Crystal jobs will run. This forces us into manual workarounds to do core business tasks like printing AP checks. There is a PeopleSoft incident that addresses this issue, and the fix is included in the latest PeopleTools upgrade. Preliminary testing indicates that the PeopleTools upgrade will indeed solve the problem, and we're working toward implementing it.

Process Instance > 1M
Due to the large number of processes we run, the process instance counter has rolled over 1 Million in only three months. When this happened, Application Engine jobs ceased to function. Since application engine jobs are necessary for almost every core batch process in PeopleSoft, the impact was immediate and severe. Once again a PeopleTools upgrade was identified as the ultimate solution, but it takes time to get a Tools upgrade moved to Production. For the interim, PeopleSoft offered two workarounds: 1) Restart the process instance counter at 1; or 2) Re-configure process scheduler so that Application Engine jobs run outside of Process Scheduler on the command line.

After some preliminary testing, we determined that restarting the process instance counter at 1 was problematic. Many tables are keyed by process instance, and these tables would have to be purged before this would work. Since the impact to the business was so severe, it was decided that there wasn't enough time to do the analysis, build a solution, test it and put it into production.

So by default, we implemented the workaround to reconfigure process scheduler so that AE jobs run on the command line. This was a simple configuration change that could be reversed if the problems got worse. Preliminary testing was successful, so we left the configuration change in place. This allowed the scheduled jobs to run.

The one problem with the workaround was that running Application Engine processes on the UNIX command line cannot invoke LDAP for authentication, so end-users were not able to manually run Application Engine jobs. The workaround to the workaround is to manually create a PeopleSoft (non-LDAP) password for users that run Application Engine jobs. The user must remember to use the new PeopleSoft password and not their Network password, otherwise the PeopleSoft password gets deleted and Application Engine jobs don't run any more.

Comment (0)

One of my clients upgraded from a Troy 4300 to a Troy 4350 model. The old one was starting to have problems, the bank was complaining, and overall it was a necessary investment. The problem was that the escape sequences necessary for printing the MICR fonts and accessing the Signature card were slightly different. But no one realized that until the printer was here.

When it arrived it was installed and a test check run was sent to it. It didn't exactly get the expected result. The MICR font wasn't MICR, and the complete signature didn't print. After tweaking the control codes and asking the user to "try again" and "tell me what it looks like" over the phone for the upteenth time, it was time to do something different.

The only way I've ever been able to get a check printing program working with a check printer is to either sit the developer next to the printer, or move the printer next to the developer. This time we were able to move the printer next to the developer, which allowed the developer to see first-hand what the problems were and to try multiple iterations of code quickly.

Here were some things I was reminded of as I worked with the developer to resolve the problem:

• When running to a Troy printer from an SQR, be sure to select HP as the Format on the Process Scheduler Request screen. Printing to PS (postscript) will make the printer ignore all of your control codes and printing with a TXT format will ignore all formatting altogether.


• PAY003.SQR really only has a few places (two I think) where changes need to be made. Don't look for problems where they don't exist.


• If you need a sanity check, write a stand-alone SQR or Crystal that does nothing but print the MICR font and the Signature just to make sure you know how it's done.


• The bank can provide a MICR ruler to make sure the characters line up.


• Be sure your testing involves sending a batch of voided checks to the bank for their approval


• Give the "check printing program modifications" tasks a generous number of hours when doing a project plan.

Comment (10)