Performance of web services in IBM BPM 85x

Follow

Introduction

When we talk about IBM BPM web services (WS) we typically refer to two types - inbound or outbound web services.

Inbound web services (WS) in IBM BPM is when you create a web service in BPM, it's hosted by BPM and you use an external call to BPM calling that web service to trigger a specific action in your process.

Outbound web service (WS) IBM BPM is when you're making a call to an external web service (WS) that is hosted outside of IBM BPM.

Web services (WS) stack in IBM BPM is implemented by underlying WebSphere JAX-WS web service engine. It's possible to fall-back to a previous web service engine that was used in earlier IBM BPM (Lombardi edition) releases but this is outside of the scope of this article. So, we will assume that JAX-WS is used which is the default web service engine in IBM BPM 8.5.x.

Examples of web services performance issues

Performance depends on too many factors. You might have a situation where you make a number of WS calls within the BPM and these calls average around 0.5 seconds (500 ms) per call, when executed through BPM, but have response times 5-10 faster if called outside of the BPM context (SOAP UI as an example). 

1) One of the common reasons why IBM BPM SOAP connectors (actually pretty much any JAX-WS implementation) are going to be slower than HTTP GET and soapUI is that the SOAP connectors parse and validate the WSDL and XSDs as well as serialize/deserialize the SOAP Request/Response to/from BPM engine variables. All these operations have a cost (cost of simplicity, reliability and speed of development). Therefore, all these aspects need to be taken into account. In other words, comparing performance HTTP GET to Web Service Integration is NOT "apples to apples" comparison.

2) Another aspect to keep in mind is the caching of parsed WSDL and XSDs on the BPM side (for outbound web services), which means that the FIRST invocation of the web service after JVM restart or WSDL/XSD update is always going to be slower than subsequent invocations (because of all the parsing of WSDL/XSD BPM has to do). Therefore, when you test for performance, it is better to make sure you execute the test multiple times and measure the moving average.

3) If you use environment variables in web service integration service then consider measuring how long it takes to retrieve those. This is especially true if your running on a version of IBM BPM that is older than IBM BPM 8.6. CF 2018.03. In IBM BPM 8.6 2018.03 IBM introduced a cache for environment variables, so, it supposedly should work better/faster there. In earlier versions environment variables are retrieved from the database each time. I have worked with one client reporting slow web services calls within IBM BPM (it works fast in SOAP UI). We have determined that retrieval of environment variables took at least 50% of the overall time of the execution of this outbound web service. So, we ran a test where we defined values using "inline" configuration in web service integration instead of using a "Server" scope level with ENV vars. These modifications has given us great performance boost.  The calls were running 2 to 4 seconds whereas previously it was from 10 to 32 seconds.

4) It's also important to keep in mind that most operations related to WSDL and XSDs parsing is CPU intensive operations. So, you might be getting significantly different results on different servers with different HW configuration(s).

How to troubleshoot web services performance issues

  • It's best to narrow down the problem to something specific when using WS integration or plain SOAP connector. In order to do that you can use generic WS traces from the following technote or use so called WSPerf trace documented in the following article. The advantage of WSPerf trace is that it's lightweight, when compared to the full web services engine trace.
  • Make sure to monitor OS resources during your WS performance tests. Monitor from an OS level (CPU, memory) as well as from a WAS/JVM level (memory/heap, verboseGC, threads). If your IBM BPM server is running in a Virtual Machine make sure you analyze CPU consumption from a Virtual Machine ESX server perspective because if you do have shared resources in your VM appliance looking at the operating system resource pages (free CPU, free memory) can show that there are available resources. However, these resources are being shared or constrained at a deeper level in the larger VM server setup. So, use VM tools as explained in the following technote.
  • Initially I thought there is a WSDL caching mechanism involved when calling inbound WS within the BPM. So, I was really hoping that I could check numbers in instrumentation monitor page using extended page URL:

https://bpm_host:port/teamworks/cs_instrumentation.lsw?userVisibleOnly=false

Using the above URL I have found the following section which was exactly what I was looking for but unfortunately it's NOT working in IBM BPM 8.5.x. E.g. no any numbers for either inbound or outbound WS calls are getting reflected/populated here, it just always stays at 0 ... 
InstrLogWebServices.png
So, this is more for your awareness that this does not work unfortunately. And the reason I brought it up here anyway is because I started to investigate why it's not working and spent some time working through this with IBM L2/L3 Support and here is what I found out -

  • 1) There is no WSDL cache for inbound web services in IBM BPM 85x. Period. As such, it does not make sense to expect the Instrumentation page to reflect anything for Inbound.
    Instead, that's the process application version context cache. The PA version context cache is a small and unfortunately not adjustable cache. The WSDL is generated on every WSDL query call which is not cached. 
  • 2) For outbound web services, if you enable the "use-pre856-wsdl-cache" setting (in 100Custom.xml configuration), a non-persistent cache could be used, otherwise the WSDL would be loaded into the database ("local repository"). Even in this case, the Instrumentation is for persistent content cache used by web service. because WSDL cache is not a persistent content .. it's not mentioned. For outbound web services there was an engine switch in BPM 8.0 that completely redesigned the whole web service stack and since then the corresponding instrumentation data is not populated anymore. You can fall back to an old engine by turning the flag in 100Custom.xml to false but that is not recommended (<use-jaxws>true</use-jaxws>)

You may also use "Instrumentation monitor" but using the "Start logging" / "Stop logging" and then parse it as explained in this article.

If you look at the previous section item/example #3 where I talked about environment variables impacting the speed of web service execution we were able to determine this using the data from the collected instrumentation monitor. Here is an example of a snippet with the instrumentation data extract that shows how much time environment variables retrieval operation takes -

>> THREAD WebContainer : 409 <<
13:08:22.755 period 30529ms 'Resume Workflow Engine' {
13:08:23.441    period 29843ms 'Service Requests' Process Application=TestWebservice_PA {
13:08:24.065       period 1061ms 'Do Job' Worker=com.lombardisoftware.component.twscript.worker.ScriptWorker {
13:08:24.065          period 1061ms 'Eval Script' {
13:08:24.065             period 1030ms 'EJB Calls' Method=EnvironmentServicesCore.lookupVariable {
13:08:25.126       period 17847ms 'Do Job' Worker=com.lombardisoftware.component.wsconnector.worker.WSConnectorWorker {
13:08:25.142          period 2293ms 'EJB Calls' Method=EnvironmentServicesCore.getAdminServerVariables {
13:08:27.435          period 3588ms 'EJB Calls' Method=EnvironmentServicesCore.getAdminServerVariables {
13:08:31.038          period 11935ms 'SOAP Execution' {
13:08:31.038             period 11872ms 'Server Lookups' Name=XMLTypeDescriptorContext {
13:08:43.285       period 9609ms 'Do Job' Worker=com.lombardisoftware.component.wsconnector.worker.WSConnectorWorker {
13:08:43.285          period 3042ms 'EJB Calls' Method=EnvironmentServicesCore.getAdminServerVariables {
13:08:46.327          period 6567ms 'SOAP Execution' {
13:08:46.327             period 6333ms 'Server Lookups' Name=XMLTypeDescriptorContext {

Improving the performance of web service connectors in IBM BPM

Luckily there is a number of ways how you can improve the performance of your web service connectors in IBM BPM. 

  • Simplify the WSDL and XSDs, i.e. merge multiple includes, flatten down the XSD (turn tree of XSD into single XSD).
  • Cache the outputs of the SOAP calls, if the data is static enough
  • Call the SOAP services via AJAX wrappers from Coach, instead of sequential invocation of the SOAP services before the coach, if requirements permit
  • Make sure your WSDL cache configured and is sized accordingly. See the following doc.
  • Nice overview of WSDL caching in IBM BPM 856+ in the following post.
  • There is a section in IBM BPM Performance Redbook that is specific to web services tuning that you may want to pay attention to. It's on page 91, section 4.6.5. Copying that section here -

4.6.5 Web services tuning
If the target of the web services import binding is hosted locally in the same application server, you can further improve the performance by taking advantage of the optimized communication path that is provided by the web container.
Normally, requests from the web services clients are sent through the network connection between the client and the service provider. For local web services calls, however, WebSphere Application Server offers a direct communication channel, bypassing the network layer completely.
Complete the following steps to enable this optimization. Use the administrative console to change these values:
1. Click Application servers and the then the name of the server that you want.
Click Container Settings → Web Container Settings → Web container → Additional Properties → Custom Properties.
Set the web container custom property enableInProcessConnections to true.
Do not use wildcard characters (*) for the host name of the web container port.
Replace it with the host name or IP address.
The property can be accessed by clicking the following path: Application servers → messaging engine → Container Settings → Web Container Settings → Web container → Additional Properties → Web container transport chains → WCInboundDefault → TCP inbound channel (TCP_2) → Related Items → Ports → WC_defaulthost → Host.

2. Use localhost instead of host name in the web services client binding. Using the actual host name (even if it is aliased to localhost), disables this optimization.

To access the host name, use the following steps:
a. Click Enterprise Applications and select the application name.
b. Click Manage Modules and select the application EJB JAR file.
c. Click Web services client bindings → Preferred port mappings and select the binding name. Use the localhost (for example, localhost:9080) in the URL.

3. Make sure that there is not an entry for the server host name and IP address hosts file of your server for name resolution. An entry in the hosts file inhibits this optimization by adding name resolution processor usage.

Have more questions? Submit a request

Comments

Powered by Zendesk