Issues with Oracle with Connections 2.5 RESOLUTION

as posted by me on the Connections Blog earlier today :

IBM have now released new trigger code to resolve this issue.

The steps are simple

* Stop the application
* Backup the DB
* Run through the code to remove the Trigger
* Recreate the Trigger
* Start the primary server and test
* Assuming all is well start the other App Servers in the clusters

As yet IBM are unsure as to how they are going to release the fix as it falls out of the typical iFix scope.
If you are seeing this specific issue please contact the Lotus Connections Support team who will furnish you with the appropriate code to resolve the issue.
As soon as I have confirmation on how this will be distributed I will add what will hopefully be the final update to this on-going saga.

Big thanks to Kieran Reid at IBM and Andrew Frayling and his team at Cardiff Uni for assistance and support in resolving this issue. Great work all round.

Big maintenance weekend coming up

This weekend I am really going to Maintenance town on my live connections cluster.

I have the infamous DB issue to fix. Big props to Kieran Reid from IBM for all the help with this. So triggers to delete and re-add .. no bigg just have to have the system down to solve this ..

That got me thinking .. while its down I may get some other things sorted.
I have some LC config changes to make which should hopefully fix some of the issues I am seeing using Connections with TAM. Hopefully these will work straight away which will lend more power to my elbow that some TAM config changes need to take place (long story will blog when I find the solution) ..

And as we have a new shiny fix pack for connections, I am going to take the opporuinity to take the cluster over the whole weekend and get everything on before we potentially turn it lose on students.

I shall be a busy little nerd girl over the weekend but I am really looking forward to it 🙂

Issues with Oracle on Solaris with Connections 2.5 UPDATE

As I posted on the Connections Blog earlier today …

At last it appears there may be a *real* fix for this .. IBM have changed the trigger code which should hopefully fix the mutating trigger issue we see when deleting files ..

I will be testing this today and if it works rolling it out to my live environment over the next week.

I will attempt to get some confirmation if this code will be fixed in all new fix packs and APARS – watch this space and hopefully some great news

Issues with Oracle on Solaris with Connections 2.5 – UPDATE

After some testing with the SPARC version of this fix – which actually did work we were pleased to find out that Oracle had released a version for x86.

We applied this – this morning, and I am sorry to say it doesn’t work. If you try to delete a file from the DB directly or through the connections interface, the DB is still throwing the mutating trigger issue.

Plot thickens – time to go back to oracle 🙂

Issues with Oracle on Solaris with Connections 2.5

There is an issue when running Connections with Oracle on Solaris
Symptoms of the problem are you can not delete certain files and / or the files widget from communities

The error in the logs is – table FILES.MEDIA is mutating

08/02/10 00:01:00:569 GMT] 0000005d Library E EJPVJ9166E: Unable to delete the library with id b855660b-d6bc-4b19-891f-2087aa3d9a0c. [UserImpl@26ce26ce id=64377ea3-e571-4323-922a-dc0723fead36 directoryId=2BE4B3FF-4AB4-48FF-9B83-73689537A16A]
java.sql.SQLException: ORA-04091: table FILES.MEDIA is mutating, trigger/function may not see it
ORA-06512: at “FILES.PKG_MED_DOWNLOAD_UPD”, line 45
ORA-06512: at “FILES.MED_DOWNLOAD_UPD_S”, line 2
ORA-04088: error during execution of trigger ‘FILES.MED_DOWNLOAD_UPD_S’

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)

We have since discovered (thanks to Kieran Reid in Connections Support for doing the leg work) that this is an issue with Oracle 10.2.0.4 on Solaris – the triggers have an issue which is fixed in 10.2.0.5 – which is a big no no as far as connections go. There is a fix that you can apply to 10.2.0.4 that will resolve the problem.

From support.oracle.com search the knowledge base for 4574851
You should get three results, select the third match
Click on the link for Patch.4574851
Select the 10.2.0.4 release for the Solaris platform
Download, install and test.

*NOTE* this fix is only available for SPARC not x86

So far this appears to have fixed the issue on the backup of the Prod database (I have put a stand-alone LC25 in front of it to test which involved all sorts of DB hacking to get it to work – not recommended unless you are desperate for a quick test). I am hoping to schedule moving our prod DB from x86 to SPARC applying the patch and then plugging my LC25 cluster into it.

Changing the title on the Connections 2.5 Homepage

Changing the title on the homepage is a bit of a pain .. The steps are as follows :

Make a back up copy of the COMPRESSED homepage.ear file from the deployed application config

ND – < was ROOT >profiles< profile> configcells< cell Name >applicationsHomepage.earHomepage.ear

StandAlone – < was ROOT >profiles< profile >configcells< cell Name >applicationsHomepage.earHomepage.ear

extract it to a temp folder ie D:tempextractedhomepage

find the dboard.common.jar and extract that to a temporary folder i.e D:tempextractedDashboard

drill down into the extracted file >

> com/ibm/lotus/connections/dashboard/nls/

and find the file jsp_resources.properties

change the jsp.homepage.title = < “your new title” >

change any instances of “IBM Lotus Connections Home Page” in this file to < “your new title” >

save and close the file

do the same for any additional languages that you are supporting

re-compress the dboard.common.jar and copy the newly edited compressed version into the extracted directory of the homepage ear file.

re-crompress the homepage ear file

stop all server instances that are running the homepage application replace the newly edited and compressed homepage.ear file in the deployed application config

you will also need to replace the newly edited dboard.common.jar in the installedApps folder on your primary / standalone server.

< was_root >profiles< profile name >installedApps< cell name >Homepage.ear

once the servers are restarted they will use the new title in the homepage app

Connections 2.5 – WebSphere Tips

WebSphere Tip : 1

When clustering Connections you may encounter issues when the wizard attempts to federate the node into your deployment manager. This is a known WAS issue as the JVM suffers out of memory errors (is you delve deep in the add node log file / dmgr log you will find them).

There is a quick work around that can solve this:

Increasing the WAS HEAP size
In order for the add node command to work correctly when running the cluster wizard please do the following:

Connection servers
On each of the connections servers browse to bin and edit the addNode file (.bat or .sh depending on your OS).

Insert the line set WAS_HEAP=-Xms256M -Xmx1500M at the top of the file to set a variable (for example – under the set CMD variable)

set CMD_NAME_ONLY=%~n0
set WAS_HEAP=-Xms256M -Xmx1024M

at the bottom of the file find the “%JAVA_HOME%binjava” line and add the variable

“%JAVA_HOME%binjava” -Dcmd.properties.file=%TMPJAVAPROPFILE% %WAS_HEAP% %WAS_DEBUG% %WAS_TRACE% %CONSOLE_ENCODING% “%CLIENTSOAP%” “%JAASSOAP%” “%CLIENTSAS%” “%CLIENTSSL%” %USER_INSTALL_PROP% “-Dwas.install.root=%WAS_HOME%” “-DWAS_HOME=%WAS_HOME%” “com.ibm.wsspi.bootstrap.WSPreLauncher” -nosplash -application “com.ibm.ws.bootstrap.WSLauncher” “com.ibm.ws.runtime.NodeFederationUtility” “%CONFIG_ROOT%” “%WAS_CELL%” “%WAS_NODE%” %*

save the file.

Deployment Manager
On the deployment manager machine.
Open the Administrative Console.
Open System Administration > Deployment Manager > Process Definition > Java Virtual Machine.
Specify 256 for Initial Heap Size and 1500 for Maximum Heap Size.

Save your changes and restart the Deployment Manager.

This should resolve the issue – you may need to increase the Dmgr maximum heap slightly more but I found 1000 was just not enough and 1500 did the trick.

When you run the cluster wizard now it should run as expected 🙂

WebSphere Tip : 2

A handy tip to note if you are not a huge WebSphere guru.

To enable commands to be run from the command line without the need of the -username and -password arguments configure SOAP security.

Every WebSphere profile has a file called soap.client.props which hold soap connector client information. The path to the files is as follows : /profiles//properties

SOAP connector security is disabled by default.

When enabled with the correct information it is possible to run the standard WAS start , stop and status commands for instance by just running the .bat or .sh command without passing the extra credentials.

### EXAMPLE ###

###############################################################################
#
# JMX SOAP Connector Client Properties File
#
# This file contains properties that are used by the JMX SOAP Connector Client
# of the WebSphere Application Server product. SOAP Connector executes on WebSphere
# java servers and client systems with java applications that access WebSphere servers.
#
# ** Encoding Passwords in this File **
#
# The PropFilePasswordEncoder utility may be used to encode passwords in a
# properties file. To edit an encoded password, replace the whole password
# string (including the encoding tag {…}) with the new password and then
# encode the password with the PropFilePasswordEncoder utility. Refer to
# product documentation for additional information.
#
###############################################################################

#——————————————————————————
# SOAP Client Security Enablement
#
# – security enabled status ( false[default], true )
#——————————————————————————
com.ibm.SOAP.securityEnabled=true

com.ibm.SOAP.loginUserid=wasadminuser
com.ibm.SOAP.loginPassword=wasadminpassword

#——————————————————————————

Connections 2.5 Clustering – how to avoid some pain

All was going exactly to plan when I installed my primary node – federated correctly worked as expected and I even managed to change it fairly easily to point to a different DB and shared content store I was a very happy bunny UNTIL I decided to add node2 – then it all went “pear shaped”.

So here is a quick over view of the issue and how I have got around it – but I really want to know how this happened and if I can do anything to fix this for the future – I have a PMR open and IBM are trying to recreate the issue now.

I created node1 using the Connections install wizard to create a primary node – I supplied DB(jdbc:oracle:thin:@ < my Original DB server name >:1521:conn1) and file system info (//< my Original File server name >/LotusConnectionsData/< featureName >) and it clustered successfully and node 1 was fine.

I then moved the DB to another machine and also moved the file system. I edited the data source info at cluster and server level (jdbc:oracle:thin:@< my NEW DB server name >:1521:conn1) and also changed the file system (< my NEW File server name >/portal_collabdata$/< featureName >) in the Websphere variables section of the ICS as per the instructions in the info center. Node 1 has always worked as expected even after moving these.

When I added any subsequent node it configured the server with the original file store information (//< my Original File server name >/LotusConnectionsData/< featureName >)) and defaulting back to the original DB data source (jdbc:oracle:thin:@< my Original DB server name >:1521:conn1).

If I change these manually and resynch and restart the servers they work as expected – the Datasource although it is set at Cluster level .. is also set at server level – I had to change the datasource EVERYWHERE to fix the issue (as I have 4 servers per machine and 4 machines that is a lot of editing).

This has prompted me to ask these questions of IBM:

The WebSphere Variables for the file stores are also picking up the original path – It appears that when Node1 was federated and the config was created that it has made some kind of *template* which it creates further nodes/servers from. As I have changed the config the template is not getting updated (if this is how it is doing it).

Am I doing anything wrong?
If so what?
And if no how to I prevent this from happening in the future?

== IBM’s Response ==
I received an email back from IBM regarding the issues that I experienced after changing some settings in my cluster. The bad news is it is a limitation, the good news is they are going to fix it:

The customer is right, this is a limitation in the LC 2.5 install and is being addressed for the next release.

In LC 2.5, variables/datasources/providers/etc are created at the server level, then this is used as a template for additional servers…
the problem is that server level settings like this override higher (node, cluster, cell) level settings, causing the difficulty updating the customer experienced.
ideally, these settings would be at cluster level.

Since the customer has this working, they do not have to change anything, but, if they wish to simplify future changes they can do the following:

1. create cluster level variables, datasources, providers, etc
2. [optional… for testing] create a new node — this node will have all the server level settings by default
3. only if you did 2… delete the server level settings for the items you created at cluster level in step 1
note: if you don’t delete the server level settings for this new node, it would continue to use the server level settings
4. only if you did 2… test that the applications deployed on the new node behave correctly (basically you are verifying the cluster level settings)
5. after verifying (or reviewing) the cluster level settings (variables, datasources, etc), you can delete the server level items corresponding to the new cluster level items
note: if you don’t delete the server level settings for this new node, it would continue to use the server level settings
6. now, when you make changes to the cluster level variables thru the deployment manager, you just need to save changes and synchronize nodes
all the nodes and servers that don’t have node or server level instances of the same variables will get the cluster level values

Again, the order of precedence for finding variables, datasources, etc is….
first, is it defined for the Server? If yes, the server level item is used
second, is it defined for the Node?
third, is it defined for the Cluster?
fourth, is it defined for the Cell?