ERP Associates, Inc.

HI,
Facing critical issues in Production PIA page. From last 2-3 months the PIA page is very slow, when we bounce the Webserver it then connects OK for some time, but after few hours it becomes slow again.These days loging into PIA is also taking lot of time.

Dont know whats happening to our systems.Production is down since 2 days. Please please help me out...

Tools version is PT845, HRMS88 Oracle 9.2 and all are on windows 2000 edition. We dont have load balance for webserver, but have for App server.

It could be several things, but start in configuration.properties. Make absolutely sure the Application Servers that it tries to connect to are all up and running, the servers are reachable from the web server, and you don't have any invalid ones in your list.

Next, keep an eye on your application servers. Use the Server Status function of psadmin to make sure you have plenty of idle PSAPPSRV processes during the slow times. If they're all busy, you either need to start more, or figure out if something is getting locked on your database server.

We had a situation at my current client where all of the PSAPPSRV processes were busy and users just had to wait while the browser spun. Turns out we had a bottleneck on the SAN that was causing the database IO to take forever. We increased the available memory on our database server (reducing I/O), and identified another application we were competing with for SAN I/O and fixed it.

Finally, watch your Java Heap memory on your web server. If you use weblogic you can do it at <piaserver>:<piaport>/console. Go to Servers > PIA. Then click the Monitoring tab and Performance. If you're doing garbage collection too frequently you might notice some slowness -- although probably not to the level you're describing.

Hi Bert,
Thank you for your reply.

First thing regarding Configuration.properties, everything is fine, App servers are running fine and looks everything good.

Second thing about PSAPPSRV services, I do see some are in Idle state in Status menu.

And regarding DB tuning, I have asked my Oracle DBA to look into the memory. So he is telling that there is some memory problem in the instance.so can u tell me more in brief how can we tune DB?

And abt heap size I have increased the size to 1024. but still no luck.

And abt garbbage collection, we dont collect it regularly, sometimes.

But 1 thing I observered in the console is that , in Servers->PIA->Monitoring->performance graphs, I see that queue is just increasing like anything. once the queue lenth increases, page starts troubling, when again I bounce the Webserver it will work fine for some time and again queue increses same trouble.
Still we are working on this issue. till now no output has come.

When the queue increases, that just means there's a bottleneck somewhere and requests aren't being handled as quickly as they need to be.

If you have idle app server processes when the system is the slowest, the bottleneck is probably on your web server. From what you said I think you're ok on heap size, so check the threadcount settings. You might need more threads to service all of the user requests that are coming through. My rule of thumb is 4+(# of CPU's on web server)*2. But feel free to go above that threshold if task manager says you have plenty of idle CPU and memory. If task manager says your box is maxed out in CPU, you might consider scaling back the threadcount and adding a second web server -- But I've never seen that scenario.

If you find that application servers actually are all (or almost all) busy when the system is the slowest, I'd guess you really have a problem on the database server. I won't attempt to tune your database in a forum post, but I generally look for long-running SQL on the database to get a feel for what's slow -- is it one particular set of statements or is it across the board. It's easy to do SQL tuning to make one particular set of statements run faster, but if it's everything then you're looking at tuning internal database settings or investigating your disk solution to see if there is an I/O bottleneck. Your DBA should be able to help with this piece.

Hi Brent,
Thanks for your reply. Now page is slightly improving. As I have increased the PSAPPSERV services and also I have increased the Thread count in Config.xml file in Weblogic server. I have one query that wht is the maximum limit of Thread count we can increase? As we have nearly 800 to 900 concurrent users. Right now I have increased the value till 150. Can you suggest me anything. And one more thing is that I have observed is, the login into the application takes time when any hot backups are rnunning on the DB server, then I observe is that none of the PSAPPSERV services are idle. Queued status also shows me in high number. This slowness logging takes place whenever the hot backups run. So, what can we do to still increase the performance of the application.Do we need to do any tuning on DB server. Is there any thing to increase the hardware like RAM or something like that. Is 8GB RAM sufficient on Oracle with this much usage? Please let me know...As this is month ending we need to run time sheets and payrolls. Some what hectic these days...

One more issue we are facing these days is that: There are some Scheduled PSjobs, which are running twice at a time,ie Duplicate jobs are running. Can you tell me what might be the issue?

Your ThreadCount sounds reasonable for your concurrent users. I'd spend some time on the database next.

Obvious tip of the day: Can you move your hot backups at a time when there's less going on in the system?

What runs on the database server? Just the DB, process scheduler, app server and/or web server? Also, how many CPU's and what speed do your servers have? Also is this Unix or Windows?

I'm not sure if 8GB on the database is enough for so many concurrent users and processes. Use tools like SAR or vmstat to figure out if the database is using the memory that's available to it, and if not take steps to make that happen.

I'm really not an Oracle DBA and I'm a bit weak on database tuning. But as I recall hot backups use a lot of rollback space. Depending on your Oracle version, you might try to create more, larger rollback segments.

If you'd like to bring in someone with PeopleSoft and Oracle DBA skills for a few days to resolve this let me know. I know several good people.

Regarding the process scheduler problem: If you pay for Oracle support, open a case and make them figure it out for you. Process Scheduler is a real pain to troubleshoot, and a lot depends on your Ptools version and whether or not you're running Master Scheduler.