Case:
Customer wants to use nested groups in Access control for Communities, also it should be reflected in I'm a Member when user is looking for their communitys and so on. Connections was 4.5CRx

Google search Links that where tried, but did not work for me (for some reason unknown).
https://www.lbenitez.com/2015/11/how-to-enable-nested-ldap-groups-in-ibm.html
https://www-01.ibm.com/support/docview.wss?uid=swg21321308
https://www-10.lotus.com/ldd/lcforum.nsf/869c7412fe5d56b7852569fa007826e3/4aa9a40d4818785f85257b3b004e3240?OpenDocument
https://www.communardo.de/home/techblog/2014/06/04/nested-groups-ibm-connections/

Found something that worked for me (seems logical looking at the description).
https://social.technet.microsoft.com/wiki/contents/articles/5392.active-directory-ldap-syntax-filters.aspx
coming from thread
https://social.technet.microsoft.com/Forums/scriptcenter/en-US/f238d2b0-a1d7-48e8-8a60-542e7ccfa2e8/recursive-retrieval-of-all-ad-group-memberships-of-a-user?forum=ITCG

Description:
All groups specified user belongs to, including due to group nesting (Notes 10, 19)
eg. (member:1.2.840.113556.1.4.1941:=cn=Jim Smith,ou=West,dc=Domain,dc=com)
All members of specified group, including  due to group nesting (Note 10)
eg. (memberOf:1.2.840.113556.1.4.1941:=  cn=Test,ou=East,dc=Domain,dc=com) 
Note 10.
The string 1.2.840.113556.1.4.1941 specifies LDAP_MATCHING_RULE_IN_CHAIN. This applies only to DN attributes. This is an extended match operator that walks the chain of ancestry in objects all the way to the root until it finds a match. This reveals group nesting. It is available only on domain controllers with Windows Server 2003 SP2 or Windows Server 2008 (or above).

NOTE: 
All of this is of course done in the context of Deployment Manager.
After doing the changes a full resynch needs to be done with all nodes in the cluster (sometimes also take down node and use synchNode from the node) and restart the node.

Solution is to change my setting in Websphere to reflect this:

Also changed for performance reasons the following (optional):
Reason:
https://www.ibm.com/support/knowledgecenter/SSAW57_8.0.0/com.ibm.websphere.wim.doc/disablingnestedgroupsearches.html
Solution is to change according to instructions

How does it look in the files before and after the change, here are snippets of this:

wimconfig.xml before the change:
      <config:groupConfiguration>
        <config:memberAttributes name="member" objectClass="group" scope="nested"/>
        <config:membershipAttribute name="memberof" scope="nested"/>
      </config:groupConfiguration>

wimconfig.xml after the change:
      <config:groupConfiguration>
        <config:memberAttributes name="member:1.2.840.113556.1.4.1941:" objectClass="group" scope="nested"/>
        <config:membershipAttribute name="memberOf:1.2.840.113556.1.4.1941:" scope="nested"/>
      </config:groupConfiguration>

security.xml before the change (you can not cut and paste any of these because some parameters are unique to your environment):
  <userRegistries xmi:type="security:WIMUserRegistry" xmi:id="WIMUserRegistry_1" serverId="" serverPassword="{xor}" realm="defaultWIMFileBasedRealm" ignoreCase="true" useRegistryServerId="false" primaryAdminId="wasadmin" registryClassName="com.ibm.ws.wim.registry.WIMUserRegistry"/>

security.xml after the change (you can not cut and paste any of these because some parameters are unique to your environment):
  <userRegistries xmi:type="security:WIMUserRegistry" xmi:id="WIMUserRegistry_1" serverId="" serverPassword="{xor}" realm="defaultWIMFileBasedRealm" ignoreCase="true" useRegistryServerId="false" primaryAdminId="wasadmin" registryClassName="com.ibm.ws.wim.registry.WIMUserRegistry">
    <properties xmi:id="VMMURProperty_1" name="com.ibm.ws.wim.registry.grouplevel" value="1"/>
  </userRegistries>
 

Shortcut to this document: https:// https://https://infoware.com/?p=7180
Thats all folks