PSEM Agents Not Starting

Posted by Vasudev Seeram on November 27, 2008 Leave a comment (0) Go to comments

Problem:

When the StartAgent.bat command is executed, the following appears:

E:PSOFTFS900PSEMAgent>StartAgent.bat

ERROR main EMF_CATEGORY – Error while connecting to ‘http://glwebapp:8001/PSEMHUB/hub’.
ERROR main EMF_CATEGORY – Error: Internal Exception: – Error registering peer
ERROR main EMF_CATEGORY – Error while connecting to server – retrying attempt 1
ERROR main EMF_CATEGORY – Error while connecting to ‘http://glwebapp:8001/PSEMHUB/hub’.
ERROR main EMF_CATEGORY – Error: Internal Exception: – Error registering peer
ERROR main EMF_CATEGORY – Error while connecting to server – retrying attempt 2

Troubleshooting:

In Command Prompt, stopped the PSEMHUB profile with the following commands:

set PATH=%PATH%;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin

cd /d E:PSOFTFS900webserv

PSEMHUB_fsdmobinstopServer.bat server1 -profileName PSEMHUB_fsdmo

Moved all the files in E:PSOFTFS900webservPSEMHUB_fsdmologsserver1 to backup_20081127a.

Then restarted the PSEMHUB profile:

set PATH=%PATH%;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin

cd /d E:PSOFTFS900webserv

PSEMHUB_fsdmobinstartServer.bat server1 -profileName PSEMHUB_fsdmo

The E:PSOFTFS900webservPSEMHUB_fsdmologsserver1SystemOut.log contains the following error:

************ Start Display Current Environment ************
WebSphere Platform 6.1 [ND 6.1.0.3 cf30646.29] running with process name PSEMHUB_fsdmoNodeCellPSEMHUB_fsdmoNodeserver1 and process id 260
Host Operating System is Windows Server 2003, version 5.2 build 3790 Service Pack 1
Java version = J2RE 1.5.0 IBM J9 2.3 Windows Server 2003 x86-32 j9vmwi3223-20061001 (JIT enabled)
J9VM – 20060915_08260_lHdSMR
JIT – 20060908_1811_r8
GC – 20060906_AA, Java Compiler = j9jit23, Java VM name = IBM J9 VM
was.install.root = E:IBMWebSphereAppServer
user.install.root = E:PSOFTFS900webservPSEMHUB_fsdmo
Java Home = E:IBMWebSphereAppServerjavajre
ws.ext.dirs = E:IBMWebSphereAppServer/java/lib;E:PSOFTFS900webservPSEMHUB_fsdmo/classes;E:IBMWebSphereAppServer/classes;E:IBMWebSphereAppServer/lib;E:IBMWebSphereAppServer/installedChannels;E:IBMWebSphereAppServer/lib/ext;E:IBMWebSphereAppServer/web/help;E:IBMWebSphereAppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = E:PSOFTFS900webservPSEMHUB_fsdmo/properties;E:IBMWebSphereAppServer/properties;E:IBMWebSphereAppServer/lib/startup.jar;E:IBMWebSphereAppServer/lib/bootstrap.jar;E:IBMWebSphereAppServer/lib/j2ee.jar;E:IBMWebSphereAppServer/lib/lmproxy.jar;E:IBMWebSphereAppServer/lib/urlprotocols.jar;E:IBMWebSphereAppServer/deploytool/itp/batchboot.jar;E:IBMWebSphereAppServer/deploytool/itp/batch2.jar;E:IBMWebSphereAppServer/java/lib/tools.jar
Java Library path = E:IBMWebSphereAppServerjavajrebin;.;E:IBMWebSphereAppServerbin;E:IBMWebSphereAppServerjavabin;E:IBMWebSphereAppServerjavajrebin;E:PSOFTFS900binclientwinx86;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;E:beaTuxedo9.1bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;C:PS_HOMEbinclientwinx86;C:Program FilesMicrosoft SQL Server80ToolsBinn;C:Program FilesMicrosoft SQL Server90Toolsbinn;C:Program FilesMicrosoft SQL Server90DTSBinn;C:Program FilesMicrosoft SQL Server90ToolsBinnVSShellCommon7IDE;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin
************* End Display Current Environment *************
[11/27/08 11:14:53:265 VET] 0000000a ModuleManifes E UTLS0002E: The shared library psSharedLibrary contains a classpath entry which does not resolve to a valid jar file, the library jar file is expected to be found at E:PSOFTFS900webservPSEMHUB_fsdmoinstalledAppsPSEMHUB_fsdmoNodeCellPSEMHUB.earPSIGW.warWEB-INFlibmail.jar.
[11/27/08 11:14:53:390 VET] 0000000a ModuleManifes E UTLS0002E: The shared library psSharedLibrary contains a classpath entry which does not resolve to a valid jar file, the library jar file is expected to be found at E:PSOFTFS900webservPSEMHUB_fsdmoinstalledAppsPSEMHUB_fsdmoNodeCellPSEMHUB.earPSIGW.warWEB-INFlibactivation.jar.
[11/27/08 11:16:00:546 VET] 0000000a TCPPort E TCPC0003E: TCP Channel TCP_4 initialization failed. The socket bind failed for host * and port 8500. The port may already be in use.
[11/27/08 11:16:00:546 VET] 0000000a WSChannelFram E CHFW0034W: The Transport Channel Service detected transport chain WCInboundDefaultSecure failed. The service will retry to start chain WCInboundDefaultSecure every 5000 milliseconds for up to 60 attempts.
[11/27/08 11:16:00:562 VET] 0000000a TCPPort E TCPC0003E: TCP Channel TCP_4 initialization failed. The socket bind failed for host * and port 8500. The port may already be in use.
[11/27/08 11:16:00:562 VET] 0000000a WSChannelFram E CHFW0034W: The Transport Channel Service detected transport chain SOAPAcceptorChain3 failed. The service will retry to start chain SOAPAcceptorChain3 every 5000 milliseconds for up to 60 attempts.

Netstat showed the following output:

E:PSOFTFS900webservPSEMHUB_fsdmo>netstat -na|more

Active Connections

Proto Local Address Foreign Address State
TCP 0.0.0.0:86 0.0.0.0:0 LISTENING
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1025 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2809 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2810 0.0.0.0:0 LISTENING
TCP 0.0.0.0:2811 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8000 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8001 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8002 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8500 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8502 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8880 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8881 0.0.0.0:0 LISTENING
TCP 0.0.0.0:8882 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9043 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9044 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9045 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9060 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9061 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9062 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9100 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9101 0.0.0.0:0 LISTENING
TCP 0.0.0.0:9102 0.0.0.0:0 LISTENING
TCP 10.18.14.30:139 0.0.0.0:0 LISTENING
Note that port 8501 is not used.

Check ports of PIA: pointed web browser to http://glwebapp:9060/ibm/console/. Logged on as admin. On the left pane, navigated to Servers > Application Servers. Click server1, and under Communications, click Ports.

Note that the WC_defaulthost_secure is 8500.

Check ports of PSEM: pointed web browser to http://glwebapp:9061/ibm/console/. Logged on as admin. On the left pane, navigated to Servers > Application Servers. Click server1, and under Communications, click Ports.

Note that the WC_defaulthost_secure is also 8500. It should be 8501(?)

Click WC_default_secure, and then change the port number from 8500 to 8501. The following message appears:

One or more virtual hosts match the previous host name and port combination of “*:8500” and do not match the new combination of “*:8501.” You might need to update the host aliases.
Changes have been made to your local configuration. You can:

• Save directly to the master configuration.
• Review changes before saving or discarding.

The server may need to be restarted for these changes to take effect.

The PSEM profile is stopped and started using the steps shown above.

After stopping PSEM web profile and restarting, the following error appears:

Recover prior changes

Your prior session timed out before the changes could be published to the master configuration. Would you like to recover the changes made in the prior session or work with the master configuration?

Work with the master configuration
Recover changes made in prior session View items with changes

Changed Items Status
cells / PSEMHUB_fsdmoNodeCell / nodes / PSEMHUB_fsdmoNode / serverindex.xml Updated

Also:

Enterprise Applications > Save
Save your workspace changes to the master configuration.

Click Save to update the master repository with your changes.
Click Discard to discard your changes and begin work again using the master repository configuration.
Click Cancel to continue working with your changes.

Total changed documents: 2

Changed Items Status
cells / PSEMHUB_fsdmoNodeCell / nodes / PSEMHUB_fsdmoNode / serverindex.xml Updated
cells / PSEMHUB_fsdmoNodeCell / applications / PSEMHUB.ear / deployments / PSEMHUB / deployment.xml

After saving the master configuration, PSEM was stopped and restarted. The file E:PSOFTFS900webservPSEMHUB_fsdmologsserver SystemOut.log showed:

************ Start Display Current Environment ************
WebSphere Platform 6.1 [ND 6.1.0.3 cf30646.29] running with process name PSEMHUB_fsdmoNodeCellPSEMHUB_fsdmoNodeserver1 and process id 2580
Host Operating System is Windows Server 2003, version 5.2 build 3790 Service Pack 1
Java version = J2RE 1.5.0 IBM J9 2.3 Windows Server 2003 x86-32 j9vmwi3223-20061001 (JIT enabled)
J9VM – 20060915_08260_lHdSMR
JIT – 20060908_1811_r8
GC – 20060906_AA, Java Compiler = j9jit23, Java VM name = IBM J9 VM
was.install.root = E:IBMWebSphereAppServer
user.install.root = E:PSOFTFS900webservPSEMHUB_fsdmo
Java Home = E:IBMWebSphereAppServerjavajre
ws.ext.dirs = E:IBMWebSphereAppServer/java/lib;E:PSOFTFS900webservPSEMHUB_fsdmo/classes;E:IBMWebSphereAppServer/classes;E:IBMWebSphereAppServer/lib;E:IBMWebSphereAppServer/installedChannels;E:IBMWebSphereAppServer/lib/ext;E:IBMWebSphereAppServer/web/help;E:IBMWebSphereAppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
Classpath = E:PSOFTFS900webservPSEMHUB_fsdmo/properties;E:IBMWebSphereAppServer/properties;E:IBMWebSphereAppServer/lib/startup.jar;E:IBMWebSphereAppServer/lib/bootstrap.jar;E:IBMWebSphereAppServer/lib/j2ee.jar;E:IBMWebSphereAppServer/lib/lmproxy.jar;E:IBMWebSphereAppServer/lib/urlprotocols.jar;E:IBMWebSphereAppServer/deploytool/itp/batchboot.jar;E:IBMWebSphereAppServer/deploytool/itp/batch2.jar;E:IBMWebSphereAppServer/java/lib/tools.jar
Java Library path = E:IBMWebSphereAppServerjavajrebin;.;E:IBMWebSphereAppServerbin;E:IBMWebSphereAppServerjavabin;E:IBMWebSphereAppServerjavajrebin;E:PSOFTFS900binclientwinx86;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;E:beaTuxedo9.1bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;C:PS_HOMEbinclientwinx86;C:Program FilesMicrosoft SQL Server80ToolsBinn;C:Program FilesMicrosoft SQL Server90Toolsbinn;C:Program FilesMicrosoft SQL Server90DTSBinn;C:Program FilesMicrosoft SQL Server90ToolsBinnVSShellCommon7IDE;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin
************* End Display Current Environment *************
[11/27/08 14:39:35:968 VET] 0000000a ModuleManifes E UTLS0002E: The shared library psSharedLibrary contains a classpath entry which does not resolve to a valid jar file, the library jar file is expected to be found at E:PSOFTFS900webservPSEMHUB_fsdmoinstalledAppsPSEMHUB_fsdmoNodeCellPSEMHUB.earPSIGW.warWEB-INFlibmail.jar.
[11/27/08 14:39:36:093 VET] 0000000a ModuleManifes E UTLS0002E: The shared library psSharedLibrary contains a classpath entry which does not resolve to a valid jar file, the library jar file is expected to be found at E:PSOFTFS900webservPSEMHUB_fsdmoinstalledAppsPSEMHUB_fsdmoNodeCellPSEMHUB.earPSIGW.warWEB-INFlibactivation.jar.

Note that there is no reference to the port number.

The file E:PSOFTFS900webservPSEMHUB_fsdmologsserverstartServer.log showed:

************ Start Display Current Environment ************
Host Operating System is Windows Server 2003, version 5.2 build 3790 Service Pack 1
Java version = J2RE 1.5.0 IBM J9 2.3 Windows Server 2003 x86-32 j9vmwi3223-20061001 (JIT enabled)
J9VM – 20060915_08260_lHdSMR
JIT – 20060908_1811_r8
GC – 20060906_AA, Java Compiler = j9jit23, Java VM name = IBM J9 VM
was.install.root = E:IBMWebSphereAppServer
user.install.root = E:PSOFTFS900webservPSEMHUB_fsdmo
Java Home = E:IBMWebSphereAppServerjavajre
ws.ext.dirs = E:\IBM\WebSphere\AppServer\java\lib;E:\IBM\WebSphere\AppServer\classes;E:\IBM\WebSphere\AppServer\lib;E:\IBM\WebSphere\AppServer\installedChannels;E:\IBM\WebSphere\AppServer\lib\ext;E:\IBM\WebSphere\AppServer\web\help;E:\IBM\WebSphere\AppServer\deploytool\itp\plugins\com.ibm.etools.ejbdeploy\runtime
Classpath = E:PSOFTFS900webservPSEMHUB_fsdmoproperties;E:IBMWebSphereAppServerproperties;E:IBMWebSphereAppServerlibstartup.jar;E:IBMWebSphereAppServerlibbootstrap.jar;E:IBMWebSphereAppServer/lib/j2ee.jar;E:IBMWebSphereAppServer/lib/lmproxy.jar;E:IBMWebSphereAppServer/lib/urlprotocols.jar;E:IBMWebSphereAppServerjavalibtools.jar
Java Library path = E:IBMWebSphereAppServerjavajrebin;.;E:IBMWebSphereAppServerbin;E:IBMWebSphereAppServerjavabin;E:IBMWebSphereAppServerjavajrebin;E:PSOFTFS900binclientwinx86;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;E:beaTuxedo9.1bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;C:PS_HOMEbinclientwinx86;C:Program FilesMicrosoft SQL Server80ToolsBinn;C:Program FilesMicrosoft SQL Server90Toolsbinn;C:Program FilesMicrosoft SQL Server90DTSBinn;C:Program FilesMicrosoft SQL Server90ToolsBinnVSShellCommon7IDE;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin;E:PSOFTFS900webservPSOL_fsdmoinstalledAppsPSOL_fsdmoNodeCellPSOL.earPSOL.warWEB-INFlibwinx86;E:PSOFTFS900veritywinx86_nti40bin
Current trace specification = *=info:com.ibm.*=all
************* End Display Current Environment *************

[11/27/08 14:39:29:515 VET] 0000000a ManagerAdmin I TRAS0017I: The startup trace state is *=info:com.ibm.*=all.
[11/27/08 14:39:29:515 VET] 0000000a WsServerContr 1 Executing executeUtilityOnWindows with args: E:PSOFTFS900webservPSEMHUB_fsdmoconfig PSEMHUB_fsdmoNodeCell PSEMHUB_fsdmoNode server1
[11/27/08 14:39:29:515 VET] 0000000a WsServerContr 1 Getting WindowsService obj with: server1 and E:PSOFTFS900webservPSEMHUB_fsdmoconfig PSEMHUB_fsdmoNodeCell PSEMHUB_fsdmoNode server1
[11/27/08 14:39:29:515 VET] 0000000a WindowsServic > getWindowsServiceNameForServer() Entry
[11/27/08 14:39:29:515 VET] 0000000a WindowsServic 1 In executeCommand, args are: E:IBMWebSphereAppServerbinWASService.exe -getServiceName server1 -profilePath E:PSOFTFS900webservPSEMHUB_fsdmo
[11/27/08 14:39:29:531 VET] 0000000a WindowsServic 1 Runtime.exec() returned process java.lang.ProcessImpl@456a456a
[11/27/08 14:39:29:531 VET] 0000000a WindowsServic 1 retCode from childProcess.waitFor() is 0
[11/27/08 14:39:29:531 VET] 0000000a WindowsServic 3 exitCode = 0
[11/27/08 14:39:30:531 VET] 0000000a WindowsServic 1 Trying to new up WindowsService using serverName=server1, winServiceName=null
[11/27/08 14:39:30:531 VET] 0000000a ManagerAdmin I TRAS0018I: The trace state has changed. The new trace state is *=info.
[11/27/08 14:39:30:562 VET] 0000000a AdminTool A ADMU0128I: Starting tool with the PSEMHUB_fsdmo profile
[11/27/08 14:39:30:562 VET] 0000000a AdminTool A ADMU3100I: Reading configuration for server: server1
[11/27/08 14:39:32:828 VET] 0000000a AdminTool A ADMU3200I: Server launched. Waiting for initialization status.
[11/27/08 14:40:41:796 VET] 0000000a AdminTool A ADMU3000I: Server server1 open for e-business; process id is 2580

Solution:

Ensure that the web server is started

References:

http://www-128.ibm.com/developerworks/forums/thread.jspa?threadID=169775&start=15&tstart=0