WSO2 Identity Server - Concurrency Issue

Question

We are working with WSO2 IS, v5.1.0. When testing, we got the following results:

When running one suite of tests, all works ok.
When running concurrent tests, meaning - concurrent requests are being sent, we got NPE. Moreover, We got "200" on adding two users, for example, but then when trying to query and get both of them, we got a message that 2 values were expected but only 1 is returned.

Any idea how can I solve this issue? what is causing it? Let me know if any further info is required.

Thanks!

No stacktrace is written, but I'll attach a few lines prior to the NPE: `Entering deleteAttachments() {org.apache.axis2.transport.TransportUtils} Exiting deleteAttachments() {org.apache.axis2.transport.TransportUtils} Reading Role Mapping of Application 2816 {org.wso2.carbon.identity.application.mgt.dao.impl.ApplicationDAOImpl} Setting config system registry instance. {org.wso2.carbon.context.PrivilegedCarbonContext} Setting governance system registry instance. {org.wso2.carbon.context.PrivilegedCarbonContext} {org.apache.axis2.transport.http.AxisServlet} java.lang.NullPointerException` — Nitz, Apr 14 '16 at 06:54
To give you any reasonable hint, more information is needed. Is it a single node / clustered environment? What datastore is used? Did you upate any claim definitions? *Do you run the IS as a (wrapper) service?* And the most important - the **stacktrace**. — gusto2, Apr 14 '16 at 07:20
@GabrielVince It's a single node environment. We're using MySQL as our datastore. We didn't configured the default claim (we use the default..should we configure?). We run the IS as a service. Regarding the stacktrace, we don't get any when the NPE happens, I attached in the above comment the prior messages from the log (the log was in TRACE mode). Thanks !! — Nitz, Apr 14 '16 at 08:34
I want to mention one more important thing - We get this error in **one specific case** - when trying to **attach permissions to a role**. We're trying, for example, to attach 2 permissions to a single role. Sometimes both are added, sometimes one is added and the other isn't, and sometimes both are not added. In all cases, we got "200" messages. Moreover, adding permissions is also causing some problems - sometimes when adding a new permission to the system it DELETES all existing ones. — Nitz, Apr 14 '16 at 08:47
Are you doing these operations via UI or by service calls? Did you check repository/logs/wso2carbon.log for stacktrace? — Chamila Wijayarathna, Apr 14 '16 at 11:25

gusto2 · Answer 1 · 2016-04-14T10:28:37.737

Now I see..

there are two issues in the WSO2 IS 5.1.0 you can / have to fix.

Use of the embedded JSP pages

First - see there's a difference in the parameters of the wso2server.bat/.sh and the bin/yajsw/wrapper.conf. In the wrapper.conf add:

wrapper.java.additional.27 = -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false

please change the parameter order to fit your parameter list

StringUtil bundle dependency

Here comes the NPE stacktrace in play and without it, you cannot pinpoint the exact problem. Apparently the some UI bundles are having invalid dependencies specified, particularly there's no dependency specified to the Commons-Lang StringUtil package which is used.

We have solved it by following actions:

download and copy commons-lang-2.6.jar into the repository/components/dropins
create a new OSGi bundle (assuming you know Java and how to create OSGi bundle fragments) which imports org.apache.commons.lang;version="[2.6,3)" and is a fragment to the org.wso2.carbon.identity.mgt.ui bundle. Copy this bundle to the dropins folder.
create a new OSGi bundle which imports org.apache.commons.lang;version="[2.6,3)" and is a fragment to the org.wso2.carbon.identity.application.mgt.ui bundle. Copy this bundle to the dropins folder.

Edit:

part of the maven plugin to generate the bundles

        <plugin>
            <groupId>org.apache.felix</groupId>
            <artifactId>maven-bundle-plugin</artifactId>
            <version>2.3.7</version>
            <extensions>true</extensions>
            <configuration>
                <instructions>
                    <Fragment-Host>org.wso2.carbon.identity.application.mgt.ui</Fragment-Host>
                    <Import-Package>org.apache.commons.lang.*</Import-Package>
                    <!--
                    <Export-Package>org.apache.commons.lang.*</Export-Package>
                    <Embed-Dependency>commons-lang</Embed-Dependency>
                    -->
                </instructions>
            </configuration>
        </plugin>

There are multiple bundles having this issue, but to get the IS 5.1.0 usable, at least these bundles were necessary to fix.

In all cases, this answer is still based on assumptions and our experience rather than evidence (the stacktrace).

g

First of all - thank u so much for the detailed answer!! I did what you wrote in the **Use of the embedded JSP pages** section. I don't know if it relates but suddenly I can see the stacktrace. Here it is: http://pastebin.com/hZFE7HuK — Nitz, Apr 14 '16 at 13:59
Nope, there still needs to be more in the wso2carbon.log even on the INFO / ERROR level — gusto2, Apr 14 '16 at 14:15
you're right. I changed to ERROR log level. Here's the full stacktrace: http://pastebin.com/JNBUYPsm — Nitz, Apr 14 '16 at 14:46
Is there a known concurrency issue when trying to update users role's permissions? https://wso2.org/jira/browse/IDENTITY-1135?jql=project%20in%20(CARBON%2C%20IDENTITY)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22)%20AND%20text%20~%20%22deadlock%22%20ORDER%20BY%20created%20DESC — Nitz, Apr 17 '16 at 15:33
Indeed, the IS has some issues with concurrency but this is not the case. In this case I'd bet you have to fix some database settings to resolve the DB deadlock. Try to search for the mysql exception you see in the stacktrace — gusto2, Apr 17 '16 at 20:47
I see there are inner selects used in the statement, so it may take time until the select returns, try to increase the wait time (http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout) and as well - the stacktrace should be present even on lower debug levels (by default you should run on INFO or WARN) — gusto2, Apr 18 '16 at 08:28

WSO2 Identity Server - Concurrency Issue

1 Answers1