Monday, October 17, 2011

The “reassociation” business

Since Fusion Middleware, OPSS (Oracle Platform Security Services) support 3 types of security stores: file, OID (Oracle Internet Directory) and Oracle database. When a Weblogic server domain is first created, OPSS is “associated” to a file-based security store by default, which is ok for development purposes. But for production, that is not recommended (Please check Multiple Nodes Servers Environments section in OPSS docs). That would be ok if your whole environment is a single Weblogic domain with only one server in a single machine. But 99,99% of the cases are not like that. Usually, an SOA or WebCenter environment is composed of multiple servers in clusters spread across different machines. A file-based security store is not a scalable option. In these cases, you should look at OID or the database. Fusion Applications, a gigantic set of apps, adopt OID as the security store.

The OPSS security store is a composite of policies, credentials, keys and audit services. Notice that I am leaving the identity store service out. OPSS delegates the identity store service to the identity providers configured in WebLogic server.

As a side note, OPSS is not a product, but a set of security services used by Fusion Middleware. If you’re a Fusion Middleware user, trying to understand OPSS is a great idea.

This post is about the nitty-gritty details of configuring (or reassociating) a Weblogic server domain (or multiple domains) to a different type of security store. That’s where the term “reassociation” comes from.

The information presented here is a small subset, but complements and sometimes overlaps “Configuring OPSS Security Store” documentation (reading is strongly recommended).
Before going any further on reassociation, let me talk a bit about an important character: jps-config.xml.


This is the OPSS file that describes all its services. It is located through the – system property, which is set in script in a standard JRF (Java Required Files) domain. By the default, the property points to ${DOMAIN_HOME}/config/fmwconfig/jps-config.xml and it is defined in the variable EXTRA_JAVA_PROPERTIES. It is NOT a good idea to change it, since jps-config.xml holds several relative references to other files.

Whenever you create a BPM, SOA or WebCenter domain via the script, JRF template is automatically selected as a dependency.

That put, jps-config.xml is a domain-wide artifact. There’s no such thing as a server-level or application-level jps-config.xml. However, jps-config.xml provides the concept of contexts. They can be explored in case you want to hookup different applications to different services, but this would be a topic for another post.

When a reassociation operation is performed, configuration changes are made to jps-config.xml. In many cases, a corrupted jps-config.xml can bring your domain to a non startable state. Therefore, be very diligent and careful when making changes to it. Do NOT perform manual changes. Instead, use either Enterprise Manager or wlst.

The Policy Store

The policy store holds all security policies used by applications deployed on a Fusion Middleware instance. These include grants given to principals (users, groups, application roles) as well as to code.

For instance, if you look at the OOTB policy store of a BPM domain, you would see policies scoped into 4 applications (OracleBPMProcessRolesApp, OracleBPMComposerRolesApp, b2bui and soainfra) as well as a bunch of code-source policies (which are applicable to code in any application deployed in the domain).

In jps-config.xml, the “default” context defines the services that are by default :-) picked up by Fusion Middleware applications.

<jpsContext name="default">
 <serviceInstanceRef ref="credstore"/>
 <serviceInstanceRef ref="keystore"/>
 <serviceInstanceRef ref="policystore.xml"/>
 <serviceInstanceRef ref="audit"/>
 <serviceInstanceRef ref="idstore.ldap"/>
 <serviceInstanceRef ref="trust"/>
 <serviceInstanceRef ref="pdp.service"/>

If you follow policystore.xml up in the file, you should see a serviceInstance with that name:

<serviceInstance name="policystore.xml" 
   <description>File Based Policy Store Service Instance</description>

which brings us to the OOTB file-based policy store, system-jazn-data.xml. Down in this post, we’re going to change this.

The Credential Store

The credential store securely holds credentials to be used by Fusion Middleware applications when connecting to other systems. OWSM agents, for instance, use the credential store service when a WSS username token needs to be attached to an outgoing SOAP message. Another heavy user is ADF (Application Development Framework), who uses it to store credentials required to connect to external systems. OOTB, the credential store is materialized as the cwallet.sso file pointed by the credstore serviceInstance (notice the file name itself is not specified):

<serviceInstance location="./" provider="credstoressp" name="credstore">
   <description>File Based Credential Store Service Instance</description>

Notice this credential store is not the same as the bootstrap credential store, described in "Bootstrap cwallet.sso" section down below.

Reassociating to OID

There are two options to do reassociation: Enterprise Manager and wlst. They are very well covered in the OPSS documentation, but let me still explore them here a little bit more. I hope to add some details that are not very clear in the docs.

First and foremost, make sure to satisfy a requirement: create a root node in the LDAP server that is going to hold our security store tree. Forgetting this is a very common mistake.

Enterprise Manager

Navigate to the drop-down menu for the Weblogic Domain and choose Security –> Security Provider Configuration


Notice that both the policy store and the credential store are of the same store type: file. That tells us something: the OPSS Security store can only be persisted in one physical store type for production. Persisting credentials to a different store from keys or policies is not supported or recommended.

By clicking “Change Store Type” button, you get:


The LDAP Server Details properties are straightforward. Those under Root Node Details deserves some comments.

Root DN: the node you’ve manually created in OID before.

Create New Domain: no, this is not going to create a new Weblogic domain. It determines whether or not the new security store (OID) is going to be bootstrapped with data from the source security store (in this case, the file-based policy, credential and keystore). Unchecking the box is relevant when you want more than one Weblogic domain sharing the security store. This flag corresponds to the join parameter in the wlst command, shown below.

Domain Name: Enterprise Manager uses the Weblogic domain name as a convenience, but it can actually be any arbitrary string. This name is going to manifest itself as a container node, under which OPSS security data are migrated. Several Weblogic domains can bind to the same container, but a single Weblogic domain cannot bind to different containers. I am talking about this kind of deployment:


Within OID, cn=SecurityStore is the Root DN. cn=JPSContext is implicitly created upon the first reassociation.

Notice there’s no reference to a particular Weblogic server. That essentially means that you bind one domain as a whole to a container node. Within a container, specific applications bind to application  stripes (nodes cn=AppsA, cn=AppsB, cn=AppsC, cn=AppsD).  Although not shown in the picture, any number of applications can bind to a given stripe, but the same application can bind to one and only one stripe.


Now that we understand the behavior of those reassociation properties, the wlst command is straightforward. It is an online command, which means you must connect to the Admin Server to execute it.

> reassociateSecurityStore(domain="farm1", admin="cn=orcladmin", 
password="welcome1", ldapurl=ldap://localhost:3060, servertype="OID", 


domain corresponds to the policy container that the WebLogic domain will bind to and it does NOT need to be named as the Weblogic domain. Notice the format: you do NOT prepend “cn=” to the value.

servertype is the security store type. Supported values are “OID” and “DB_Oracle”.

jpsroot corresponds to the Root DN node in OID, that, again, has to be manually created upfront.

join is optional, but of uttermost importance. It corresponds to the “Create New Domain” checkbox in Enterprise Manager. The default value (if unspecified) is “false”, which means a container node specified by the domain parameter is going to be created in OID and the security artifacts (policies, credentials, keys) migrated. As a pre-requisite, OPSS first automatically seeds an LDAP schema in OID. If there’s a container with the same name already created, you’re likely going to be presented with an error. If join is set to “true”, any security artifacts in the Weblogic domain are left alone and not migrated to the security store. 

Details on admin and password parameters in the “Bootstrap cwallet.sso” section below.

Bootstrap cwallet.sso

When “reassociated” to OID, Weblogic needs to know which credentials to use when connecting to the server. Such credentials are, by default, kept in the location that is pointed by the following jpsContext in jps-config.xml:

<jpsContext name="bootstrap_credstore_context">
 <serviceInstanceRef ref="bootstrap.credstore"/>

Looking up the file, you find:

<serviceInstance location="./bootstrap" 
 <property value="./bootstrap" name="location"/>

That implicitly means cwallet.sso file in ./bootstrap folder. This is an encrypted file which only authorized code is allowed access.

Important note: make sure the used credential has permissions to write to OID if you expect to allow changes to your policies through the Weblogic domain. That’s true, I’d say, always, or in 99.99% of the cases.

What if you need to change these credentials later? wlst command to the rescue (in wlst, type help(‘opss’) for a list of OPSS-related commands. You will see modifyBootStrapCredential):

> modifyBootStrapCredential(jpsConfigFile='<filepath>',username='<username>', password='<password>')


jpsConfigFile = path of the valid jps config file from which the context is read
username = distinguished name of the user.
password = the password to be reset.


> modifyBootStrapCredential(jpsConfigFile='/opt/wls/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/jps-config.xml',username='cn=orcladmin', password='welcome1')

 Note: modifyBootStrapCredential is an offline command.

Reassociating to Oracle database

My colleague Kavitha Srinivasan already describes the process along with some benign error messages when reassociating to an Oracle database. I will add a link to her post once she makes it public. Here I simply want to mention that the same principles discussed in the OID section apply.

For the reader convenience, I repeat here the 2 pre-requisites for Oracle database reassociation:

1) An OPSS schema needs to be created in the database. This is done using the RCU (Repository Creation Utility) tool. Here’s the RCU screenshot (notice the Metadata Services scheme is automatically selected as a dependency once you select Oracle Platform Security Services):


2) A data source needs to be created in Weblogic. It must be non-XA with support for global transactions disabled. Thanks Kavitha for such an important detail.

Enterprise Manager

Follow the same path as in the OID reassociation up to this screen:


Notice the Root DN parameter value. Differently than the OID case, it does NOT need to be previously created and it also does NOT need to follow an LDAP DN format (starting with “cn=”). This is just a convention.

Here’s the typical output of a successful reassociation:



Please check reassociateSecurityStore command.

For your convenience, here’s an example:

> reassociateSecurityStore(domain="farm2", servertype="DB_ORACLE", datasourcename="jndi/OPSS_DS", jpsroot="cn=SecurityStore",[admin="adminAccnt"], [password="passWord"],[join="trueOrfalse"])

Inform admin and password parameters only if your data source itself is protected. Their values are going to be stored in the bootstrap credential store. Notice they do NOT correspond to the database credentials used by the data source (these are actually defined in the data source itself in Weblogic). I guess the join parameter is understood by now. If not, take a look at the “Reassociating to OID” section above. 

After Reassociation

This is how jps-config.xml looks like after reassociation to Oracle DB:

<jpsContext name="default">
<serviceInstanceRef ref="credstore.db"/>
<serviceInstanceRef ref="keystore.db"/>
<serviceInstanceRef ref="policystore.db"/>
<serviceInstanceRef ref="audit"/>
<serviceInstanceRef ref="idstore.ldap"/>
<serviceInstanceRef ref="trust"/>
<serviceInstanceRef ref="pdp.service"/>

Look up in jps-config.xml for credstore.db, policystore.db and keystore.db serviceInstances. They all refer to the database via the props.db.1 property. For example, policystore.db is like:

<serviceInstance provider="policystore.provider" name="policystore.db">
 <property value="DB_ORACLE" name="policystore.type"/>
 <propertySetRef ref="props.db.1"/>

 And props.db.1:

<propertySet name="props.db.1">
 <property value="cn=soa_domain" name=""/>
 <property value="DB_ORACLE" name="server.type"/>
 <property value="cn=policystore" name=""/>
 <property value="jndi/OPSS_DS" name=""/>

 Does the value of (jndi/OPSS_DS) look familiar?

Important Note: You may be tempted to get rid of some files after reassociation, like system-jazn-data.xml, for example. Not a good idea. It’s true that it is completely out of the runtime picture, but there are some implications to the JMX framework used by Enteprise Manager and you may want it later in case you need to revert your security store to file again.

Reassociating to file

As you may have noticed, “file” is not a supported value for the servertype parameter in the reassociateSecurityStore command. Indeed, the only way to switch back to file-based security store is changing jps-config.xml manually, which is NOT a good idea. Therefore, the best thing you can do is backing up jps-config.xml before running any reassociation, so you can revert it back later on if needed.

What if you made changes to the security store (creating new policies, for example) in OID or DB mode and want those policies in the file-based policy store? Check wlst’s migrateSecurityStore command (help(‘migrateSecurityStore’)), which is actually a good topic for a future post.

Enjoy your reassociations! :-)


  1. Hi,

    I am working on doing the re association to Oracle database.

    When I ran the wlst command and got an exception as below:

    Based on your work on this, can you please let me know if you have faced this issue and if you have a solution for the same.

    Starting policy store reassociation.
    The store and ServiceConfigurator setup done.
    Schema is seeded into the store
    Command FAILED, Reason:
    adyExistsException: javax.persistence.PersistenceException: Exception [EclipseLi
    nk-4002] (Eclipse Persistence Services - 2.1.3.v20110304-r9073): org.eclipse.per
    Internal Exception: java.sql.SQLIntegrityConstraintViolationException: ORA-00001
    : unique constraint (DEV_OPSS.IDX_JPS_RDN_PDN) violated

    Error Code: 1
    bind => [5303, cn=systempolicy, cn=securitystore,cn=jpscontext,cn=opss4_
    Query: InsertObjectQuery(EntryId=5303:rdn=cn=systempolicy:pdn=cn=securitystore,c
    n=jpscontext,cn=opss4_domain,: JpsStore Entry={[EntryId = 5303:Attribute RowId =
    dn = cn=SystemPolicy,cn=opss4_domain,cn=JPSContext,cn=SecurityStore, EntryId = 5
    303:Attribute RowId = 7542
    objectclass = top, EntryId = 5303:Attribute RowId = 7543
    objectclass = orclContainer, EntryId = 5303:Attribute RowId = 7544
    cn = SystemPolicy]})


    The RCU schema for opss was created on XE database on windows.


  2. Abhi,
    is this the first domain being reassociated to DB-policy store?
    If this is not the first domain being reassociated, which value are you giving to the join parameter?

  3. Hey Andre,

    Thanks for the response.
    I realized later that the changes to be done on Enterprise Manager (Change Security Store) was already done by my team mate and me running the wlst over the configuration resulted in this unique integrity issue.

    Once the configuration was complete and I started the managed server for the SOA server, I got the below exception:

    I am currently investigating the reason for this exception. Any help would be greatly appreciated.

    Again thanks a lot for this simplified blog. access denied (
    .credstore.CredentialAccessPermission context=SYSTEM,mapName=BPM-CRYPTO,keyName=
    BPM-CRYPTO read)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    at java.lang.reflect.Method.invoke(
    Error occurs while creating BPM-CRYPTO key in Credential Store


    1. Abhi, I am not sure how harmful this message is from the SOA perspective. From a security perspective, it means that, during Weblogic startup, some application is being denied access to reading the BPM-CRYPTO key in the credential store. The way to fix this problem is creating a code grant in the policy store for the application. You may want to explore this post:

  4. hi, after reassociation i have an error like jps 01055. It says "Could not create ldap credential store instance." This error occured when starting weblogic server. Do you have a workaround for this?

    1. The reassociation command has probably failed. If this is the first domain being reassociated, make sure join parameter is set to false.

  5. We have reassociated OID to Oracle Webcenter, but when trying to start the Weblogic server, it has an error: The credential store DN is missing in the LDAP store; the target DN must be pre-configured. Do you have a workaround for this?

    1. The reassociation command has probably failed. If this is the first domain being reassociated, make sure join parameter is set to false. Please, observe the outcome of the reassociation operation.

  6. Thanks this was helpful. I had more luck with the OPSS store. The LDAP reassociation didn't bring over the necessary code grants.

  7. I am working on upgrading OIM from to 11.1.2. My env has a "file" security store. The upgrade procedure has me creating the OPSS schema, extending the domain for OPSS (which I think just creates the DS), running upgradeOPSS(), and then it asks me to "configure the database Security Store". The upgrade doc isn't very specific as to what I should do here. They point me to the OPSS setup procedure for a new install, running: -d -c IAM -p -m create

    which I've tried, and results in a broken env, where adminserver throws lots of PolicyStoreExceptions in its log.

    Should I be doing reassociateSecurityStore() instead?

  8. Jeremy, reassociates the domain and migrates policy data to the new policy store repository, which in your case is a database. As far as I can tell, the difference between and reassociateSecurityStore is that the former can be used in offline mode.

    Your policies, once in a file, should be migrated under an structure like cn=IAM,cn=JPSContext,cn=jpsroot. Check if you can browse them through Enterprise Manager.

    This obviously depends on the OPSS database schema available and the OPSS data source in the Weblogic domain.


  9. Hi Andre - unfortunately, em is nonfunctional when I start adminserver. the app fails to start - victim of the PolicyStoreExceptions. I can browse the raw db tables, but it's hard to make sense of it.

    I have created a separate/new/non-upgraded oim 11.1.2 installation for comparison (separate schemas etc). using the official install procedure, i used configureSecurityStore with the " -m create". That installation works fine.

    It occurs to me that using "-m create" makes sense for a new install, where I don't have an existing security store, and the db store has to be fully seeded. But in the case of my upgrade, I obviously have an existing file-base security store and domain... so as you said, reassociating the domain with the new db store and migrating my existing store data to the db store would seem to be what i need. Maybe sing "-m create" is not the right switch?

    Alternatively, maybe I could switch to the db store at my baseline state - before even starting the OIM 11.1.2 upgrade?

    1. Jeremy,

      Did you get any exceptions while running -d -c IAM -p -m create ?


  10. Hi Andre - another theory... I tried another iteration of the upgrade, this time switching from file to DB store before starting the upgrade process:

    reassociateSecurityStore(domain="IAM", datasourcename="jdbc/OPSSDBDS", servertype="DB_ORACLE", jpsroot="cn=jpsRoot")

    I then skip that execution of configureSecurityStore during the 11.1.2 upgrade process... as I'm already using a DB store. This results in a working adminserver... and oim & soa start too - although I have some issues with oim. unsure if they are related to the upgrade approach I used. Moreover, I'm not comfortable applying this process, which deviates from the official process, to my prod env.

    1. Jeremy,

      from an OPSS perspective, this approach should be fine. I will ask around internally and get back to you. I Meanwhile, let me know which kind of issues you're getting now.


  11. Great article - it simplifies for me what is a confusing area in the product docs.

    Just one typo:

    'servertype is the security store type. Supported values are “OID” and “Oracle_DB”.'

    - should be "DB_ORACLE"

    Also, a link to Kavtiha's post would be useful - I found it in the end and it too is very helpful

  12. Andre,

    Thanks for the helpful post.

    Can you tell me at what point in the setup process I would have the opportunity to select what store to use (File or LDAP)? I am currently setup for LDAP and I don't recall selecting this store type.

    Since there is no supported way to change from LDAP to file what applications would need to be de-installed /reinstalled to utilize the system-jazn-data.xml file again?

    1. nmnative,

      it's not part of the setup process. You have to do it explicitly later on, either through wlst or through EM.
      If you want to switch back to file, you could theoretically redeploy all applications that use OPSS policy store (bunch of FMW applications, depending on your domain configuration). But that's nowhere near a supported option. It's a waste of time. Don't do it. The only option would be to back file-based policy store before going to ldap or db. But that also brings its own challenges, since you would have to transfer over all policies from ldap/db that were created after reassociation.


  13. Hi Andre,

    i have standalone Weblogic setup and I am trying to configure Authorization with Db based policy store. I have created opss schema and added datasourc in weblogic.
    I am trying to reassociateSecurity Store to Db, I always gets :
    eption:, please suggest if there is something missing.

    1. Tara,

      you need a JRF-based domain. OPSS is not available in standalone Weblogic.
      You need some FMW piece (SOA, WebCenter, IdM, BI, etc) to get JRF.


    2. Hi andre, thanks for the help, i am able to proceed with wslt scripts for reassociateSecurityStore,

      Whenever i run the Weblogic Server, i get this Exception : JPS-04028: Application with name "cn=test_domain,cn=JPSContext,cn=jpsTestNode" does not exist.

      since it doesnt need to be created like in case of OID, i did not create it. Please help.

    3. Tara, you're welcome.
      I think something went wrong when you ran reassociateSecurityStore. Did you notice any errors? Please let me know the exact command you've executed.


Note: Only a member of this blog may post a comment.