Java Weblogic SSL

This is a simple WebLogic SSL configuration. If you’re not using WebLogic with SSL you probably should be. Minimally have SSL setup to encrypt your passwords to the administration consoles. I actually force SSL and disable the standard http listen port. My authentication provider is active directory and I’m using SSL there as well. The node manager is also setup using SSL as are the managed servers (JVMs). I’ll get into all that a bit later.

I’ll be using java’s keytool for all of this. We will create two keystores, one for the identity and one for the trust. You could also use the standard java trust and simply add your root certificates to it.
Some Notes (WebLogic Default Keystore Passwords - In case you want to mess with the demo trust or demo keystore)
Trust store password: DemoTrustKeyStorePassPhrase
Key store password: DemoIdentityKeyStorePassPhrase
Private key password: DemoIdentityPassPhrase
Java standard trust store password: changeit

Using java keytool:
Create the identity keystore and keypair.
I cd directly to the directory where WebLogic stores its demo trust and demo identity keystores. In my case /opt/oracle/middleware/wlserver_10.3/server/lib. My two keystores weblogic_identity.jks and weblogic_identity.jks will be created and stored there.

/opt/oracle/middleware/java/bin/keytool -genkey -alias weblogicServer -keyalg RSA -keysize 2048 -keystore weblogic_identity.jks -dname "CN=myhost.domain.com,OU=Middleware, O=MyOrg"

You will be asked to create a password for this keystore, so make sure to save it or remember it.

Create the certificate signing request (CSR):

/opt/oracle/middleware/java/bin/keytool -certreq -alias weblogicServer -file myhost.csr -keystore weblogic_identity.jks

Take the contents of the myhost.csr and submit it to your internal certificate authority (CA) or another external CA. In my example I get three certificates back. The root certificate, the intermediate certificate and the newly signed certificate we just submitted our CSR for. I get all of these back in base 64 encoding. Once you have these you can begin importing them into the proper keystores.

Create the trust keystore & import the root certificate:

/opt/oracle/middleware/java/bin/keytool -import -trustcacerts -alias myRoot -file /path/to/myRoot.cer  -keystore weblogic_trust.jks

Import intermediate certificate to trust keystore

/opt/oracle/middleware/java/bin/keytool -import -trustcacerts -alias entRoot -file /path/to/entRoot.cer  -keystore weblogic_trust.jks

Import root certificate to identity keystore

/opt/oracle/middleware/java/bin/keytool -import -trustcacerts -alias myRoot -file /path/to/myRoot.cer  -keystore weblogic_identity.jks

Import intermediate certificate to identity keystore:

/opt/oracle/middleware/java/bin/keytool -import -trustcacerts -alias entRoot -file /path/to/entRoot.cer  -keystore weblogic_identity.jks

Import signed certicifate to identity keystore:

/opt/oracle/middleware/java/bin/keytool -import -trustcacerts -alias weblogicServer -file /path/to/mySignedCert.cer -keystore weblogic_identity.jks

That’s all we have to do with keytool. We now have the two java keystores we need to configure Weblogic SSL.
WebLogic WSLT script for SSLThis script will setup the above keystores for your admin and all your managed JVMs.

#!/usr/bin/python
# Read Properties File
loadProperties("/path/to/scripts/my.props")

# Split if more than one.
WLmgdName = WLmgdNameList.split(',')
 
# Connect String
connect(username,password,'t3://'+adminHost+':'+adminPort)
 
# Get your edit on son! DO WORK!
edit()
startEdit()
 
# Admin Server SSL & Keystore
cd('/Servers/'+adminName)
cmo.setKeyStores('CustomIdentityAndCustomTrust')
cmo.setCustomIdentityKeyStoreFileName(wlHome+'/server/lib/weblogic_identity.jks')
cmo.setCustomIdentityKeyStoreType('jks')
cmo.setCustomTrustKeyStoreFileName(wlHome+'/server/lib/weblogic_trust.jks')
cmo.setCustomTrustKeyStoreType('jks')
cd('/Servers/'+adminName+'/SSL/'+adminName)
cmo.setServerPrivateKeyAlias('weblogicServer')
 
for mgdServer in WLmgdName:
       # Managed Server SSL & Keystore
       cd('/Servers/'+mgdServer)
       cmo.setKeyStores('CustomIdentityAndCustomTrust')
       cmo.setCustomIdentityKeyStoreFileName(wlHome+'/server/lib/weblogic_identity.jks')
       cmo.setCustomIdentityKeyStoreType('jks')
       cmo.setCustomTrustKeyStoreFileName(wlHome+'/server/lib/weblogic_trust.jks')
       cmo.setCustomTrustKeyStoreType('jks')
       cd('/Servers/'+mgdServer+'/SSL/'+mgdServer)
       cmo.setServerPrivateKeyAlias('weblogicServer')
 
save()
activate()
exit()

WLST properties file (my.props)

username=weblogic
password=weblogic123
adminName=my_admin
adminHost=myadmin.domain.com
adminPort=30000
WLmgdNameList=jvm01,jvm02,jvm03,jvm04,jvm05,jvm06
wlHome=/opt/oracle/middleware/wlserver_10.3

Now you should log into the admin console and change your passwords under Servers SSL and Keystores to use the password I told you to save or remember back at the start of this post. You can probably add the password bits to the script if you want, I’ll have to check that out.Weblogic node manager SSL
Edit nodemanager.properties
You should make sure you are using SecureListener=true and add the following:

KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=/opt/oracle/middleware/wlserver_10.3/server/lib/weblogic_identity.jks
CustomIdentityKeyStorePassPhrase=t0ps3cret
CustomIdentityAlias=weblogicServer
CustomIdentityPrivateKeyPassPhrase=t0ps3cret
CustomTrustKeyStoreFileName=/opt/oracle/middleware/wlserver_10.3/server/lib/weblogic_trust.jks

These passwords will encrypt on first start.
Set node manager type to SSL
Log into your admin console and make sure node manager type is set to SSL.
After all that make sure you save and activate any changes you made and restart everything and you should be good to go. Rock on…..

Weblogic Cluster SSL:
Change cluster address port
I assign addresses and ports on my cluster page, don’t forget to change the port to your secure port.
Secure Replication
Oh yea, if you disable all your regular listen ports and change cluster communication to use SSL make sure you change your cluster replication to “Secure Replication Enabled” or else things wont work. This setting is under clusters > cluster name > replication. You will see an error similar to:

server subsystem failed. Reason: java.lang.AssertionError: No replication server channel for osb_01 java.lang.AssertionError: No replication server channel for osb_01

HOW TO CLEAR WLI_PROCESS_EVENT

Can I clear WLI_PROCESS_EVENT, the system is using 29 GB of LOB space and I have an idea that WLI_PROCESS_EVENT is responsible for it. The reason of clearing is DB is getting full and we don't want historic data, and when server crash it took longer to come back.

Some Counts from the DB (Those with CLOB's/BLOB's):
SELECT COUNT(*) FROM WLI_CALENDAR
-- 1
SELECT COUNT(*) FROM WLI_PROCESS_DOCUMENT
-- 12
SELECT COUNT(*) FROM WLI_PROCESS_EVENT
-- 638564
SELECT COUNT(*) FROM WLI_PROCESS_TRACKING
-- 81
SELECT COUNT(*) FROM WLI_WORKLIST_DATA
-- 0
SELECT COUNT(*) FROM WLI_MT_CONTENT 
-- 0
SELECT COUNT(*) FROM WLI_PROCESS_INSTANCE_INFO;
-- 112

Solution: There are large amount of orphaned events which has a LOB column, so basically get rid of them:
Run this SQL Query and commit

$ Update  <DB_SCHEMA_NAME>.WLI_PROCESS_INSTANCE_INFO SET PROCESS_STATUS = 5 WHERE PROCESS_STATUS = 1;

Go to wliconsole and do a manual purge. Run this SQL Query and commit:

Run this SQL Query and commit:
DELETE FROM WLI_PROCESS_EVENT WHERE PROCESS_INSTANCE IN (SELECT WLI_PROCESS_EVENT.PROCESS_INSTANCE FROM WLI_PROCESS_EVENT LEFT OUTER JOIN WLI_PROCESS_INSTANCE_INFO ON WLI_PROCESS_EVENT.PROCESS_INSTANCE = WLI_PROCESS_INSTANCE_INFO.PROCESS_INSTANCE WHERE WLI_PROCESS_INSTANCE_INFO.PROCESS_INSTANCE is NULL)

NOTE: Because I'm using Oracle, the LOB segment did not shrink, it only freed up blocks. To free up space I need to drop the column (which will drop the lobsgment) and recreate the column OR truncate the table.

How to enable GUI while connecting to Remote Redhat / Linux machine using Putty.

This post cover details about how we can enable GUI interfacing using Xserver while connecting to Remote Redhat 5.6 / Linux Machine using Putty on Windows based local machine.

First all of you need to install Xserver in your local box. Xserver will be installed using Xming.

After downloading install the Xming server in your localbox and run it with option ":0 -clipboard -multiwindow -ac". To do this, right click the short cut of Xming -> go to properties -> and in target  its should look similar to this "C:\Program Files\Xming\Xming.exe" :0 -clipboard -multiwindow -ac (here double quates are part of the string itself), depending on the location of Xming installation path may change.

On Redhat / Linux machine has feature called X11Forwarding, depending on the value of this parameter it enables or disables the display of graphics on the server.

1. login on Linux/BSD system called myserver.mydomain.com
2. Open /etc/ssh/sshd_config file using text editor:
1. # vi /etc/ssh/sshd_config
3. Find out parameter X11Forwarding and set it to yes:
1. X11Forwarding yes
2. Save file & exit shell prompt.
4. Restart sshd service under Debian Linux:
1. # /etc/init.d/ssh restart
2. Alternatively, if you are using Fedora / Red Hat Linux restart sshd:
3. # /etc/init.d/sshd restart

Client Side Setting on Windows Machine
For connecting to Redhat Linux box I am using the most popular SSH client putty. Below are the steps to configure putty on Windows machine.

1. Run the putty.exe
2. provide Host Name (or you can use IP address of host machine as well)
3. Select SSH as Connection Type.
4. Port should be 22 default
5. Enter again the same name as you entered in Hostname in to Saved Session Input box.
6. In Connection Category, Find out the Connection Tree. In SSH, expand it and you will see "Enable X11 Forwarding"..
7. Enable X11 Forwarding by selecting the check box
8. X Display location  should be set to localhost:0
9. Save this entire information as a session by click on Save button
10. Now start the Xming Server on location machine
11. Now connect to the Redhat / Linux Machine using saved session from putty
12. And to verify that Graphics are enable use this command xclock &
13. You should be able to new graphical window coming up.

Now you are all set to roll, you can execute and run any GUI based application from the Redhat / Linux box and it will get displayed on you local machine.

How To Encrypt Clear Text Passwords With WebLogic Server

WebLogic Server encrypts all the plain text passwords stored in its domain configuration XML file(s). This is to prevent access to sensitive information. When passwords are entered using administration console or scripting tools, it will automatically get encrypted before they are stored in the configuration XML files(s).

Prior to WebLogic Server 9.0: If those passwords need to be reset either the configuration tools (Console or scripting tools) can be used which will automatically re-encrypt the passwords or by directly changing the configuration files using a text editor. When files are directly modified using a text editor the passwords will get encrypted during the subsequent restart.

Starting from WebLogic Server 9.0: Using clear text passwords in the configuration files are supported only for Development domain and it will not re-encrypt the passwords. If the domain is a Production domain then you cannot set the passwords in clear text. You have to either use a dedicated command-line utility or WLST to encrypt the clear text passwords. If the server encounters a clear text password when parsing the configuration file(s) while starting in Production Mode, then you will get an error similar to the following:
<Oct 20, 2008 9:05:35 PM UTC> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: java.lang.IllegalArgumentException: In production mode, it's not allowed to set a clear text value to the property: PasswordEncrypted of ServerStartMBean>
Depending on the configuration the MBean name value of the error message may change. In this case the ServerStartMBean has clear text value for a password property. Either the dedicated Java utility to encrypt clear text values can be used or WLST cant be used to re-encrypt. To run the encrypt utility follow the instructions below:

1. Change directory to your domain's bin folder (For Eg. cd c:\bea\user_projects\domains\mydomain\bin)
2. Execute the setDomainEnv script (For Eg. setDomainEnv.cmd)
3. Execute java weblogic.security.Encrypt which will prompt for the password and will print the encrypted value in stdout.
4. The following are some sample output from running the utility
1. C:\bea\user_projects\domains\mydomain>java weblogic.security.Encrypt
   Password:
   {3DES}9HWsf87pJTw=
2. You should execute this utility from the domain folder as it requires the domain's password salt file (SerializedSystemIni.dat) for encrypting the clear text string. You can also pass the clear text string as an argument: C:\bea\user_projects\domains\mydomain>java weblogic.security.Encrypt testpwd
   {3DES}9HWsf87pJTw=
5. You can also use WLST to encrypt clear text strings as below:C:\bea\user_projects\domains\mydomain>java weblogic.WLST
   Initializing WebLogic Scripting Tool (WLST) ...
   Welcome to WebLogic Server Administration Scripting Shell
   Type help() for help on available commands
   wls:/offline> es = encrypt('testpwd')
   wls:/offline> print es
   {3DES}9HWsf87pJTw=
   wls:/offline>
1. When running WLST from a location different than the domain folder you can pass in an argument to specify the domain directory. Once you have the encrypted value, the configuration files can be modified to include this encrypte value instead of clear text passwords. These features will make your domain to operate when resetting the encrypted passwords on a Production domain's configuration XML files. These methods not only can be used to encrypt configuration XML (config.xml) but also the JDBC or JMS descriptor XML files.

Use PuTTYgen to generate a private/public key pair

Use PuTTYgen to generate a private/public key on the SSH client PC:
The example installation is from an installation on a Windows Vista Ultimate laptop PC

In the following procedure a 4096-bit RSA private/public key pair will be generated on the SSH clients PC. The public key will be transferred to the copSSH server PC for installation by the copSSH server administrator in the users .ssh folder.
Important Perquisites
    Make sure the user is activated on the copSSH server prior to performing this procedure.
    Make sure the user can login to the copSSH server PC using PuTTY and/or WinSCP with a password.

Create a new private/public key pair encrypted with a strong pass phrase using PuTTYgen
PuTTYgen is automatically installed on the client PC when WinSCP is installed or it can be downloaded from the PuTTY site. On the client PC go to Start | Run and run the C:\Program Files\WinSCP\PuTTYgen\puttygen.exe program from the command line.
Select SSH-2 RSA and 4096 bits. Click on the Generate key.

Image 1
Enter an appropriate/descriptive Key comment, enter a strong pass phrase in the Key passphrase window then reenter the pass phrase in the Confirm the passphrase window.

Good pass phrases are 10-30 characters long, are not simple sentences or otherwise easily guessable (English prose has only 1-2 bits of entropy per character, and provides very bad pass phrases), and contain a mix of upper and lowercase letters, numbers, and non-alphanumeric characters.

Do NOT forget the pass phrase. Lost pass phrases can NOT BE RECOVERED.

Highlight and copy the text in the Public key for pasting into OpenSSH authorized_keys file window into a Notepad text file. Name the file with a descriptive file name with a .pub extension and save on the client PC in a temporary folder. For example the user Al might save his public key file as Al.pub in the client C:\Temp folder.

Image 2:
Click on the Save private key button to convert and save the private key in the C:\Program Files\WinSCP\PuTTY\Key Files folder on the client PC. Create the folder if needed.

Image 3:
Move the public key to the copSSH server PC

In order to use a private/public key pair the newly created public key must be moved to the copSSH server PC. Use whatever means required to accomplish this, ie. floppy, flash drive, etc. The copSSH server administrator will then copy the new public key to the users C:\Program Files\copssh\home\\.ssh\authorized_keys file. For example the copSSH server administrator might copy the Al.pub file to the C:\Program Files\copssh\home\Al\.ssh\authorized_keys file on the server PC.

Configure PuTTY and/or WinSCP on the client PC to use the private key file

Open PuTTY and load a previously saved session, or create a new session. Enter the path and file name of the private key file in the Private key file window. I also recommend configuring PuTTY to only use the SSH2 Protocol. Click on SSH and select 2 only as the Preferred SSH protocol version. Continue configuring PuTTY including an alternate Port number for the SSH server (if used) as desired then save the session.

Image 4:
Open WinSCP and load a previously saved session, or create a new session. Enter the path and file name of the private key file in the Private key file window. I also recommend configuring WinSCP to only use the SSH2 Protocol. Click on SSH and select 2 only in the Protocol options window. Continue configuring WinSCP including an alternate Port number for the SSH server (if used) as desired then click on Save.

Image 5