Friday, January 30, 2015

EM GC 12c OMS repository switch

If you have plans to scale your Oracle Enterprise Manager Cloud Control 12c installation. On of the ways to improve performance and increase availability is migrate repository form single instance database to Real Application Cluster.
Database reconfiguration is incredibly simple with rconfig utility. Instructions are simple and you may use documentation or a numerous posts about.

I'd like to share some tips about the very last steps in situation when you can't have both database up together (in my case - single instance became a cluster node). you still can use emctl utility even when old repository is not available..

1. Open additional terminal window and start AdminServer from command line  
It will throw a loot of error messages about JPS database availability, just keep it going  until server will not be in RUNNING state. From main terminal window run $OMS_HOME/bin/emctl to modify database connection as normal.

2. OMS interprets  database connection only as host:port:sid. It doesn't work for database services (unless you are going to use the connection to the single instance). For SCAN connection I used service descriptor. Sample command and parameters are below:

[oracle@host]$ $OMS_HOME/bin/emctl config oms -store_rpos_details -repos_conndesc \


3. Service Descriptor is standard SQL*Net format with some restrictions (at least on Linux platforms):
  • Descriptor is enclosed in double quotes. 
  • Descriptor doesn't contain spaces. 
After command execution just stop AdmiServer in the second terminal with Ctrl-Break and start Grid Control normally.

Monday, January 26, 2015

One with Developer Workstation

I have created a very simple applet code in JDeveloper. Here is how it looks in Applet Viewer.
I don’t care much about applet by itself, I want to make it running under Java 8, but if you need something to test project archive is right there.

Prepare applet manifest

One of the most important changes in security requirements – proper archive manifest. For JDeveloper it is really simple to implement.
  1. Create new text file in project directory with File –> New – File under General category.

  2. In the new file dialog window name it as manifest.txt and click "Ok" button.

  3. Create Manifest descriptors and save manifest.txt. Add lines as follow:
    Permissions: sandbox
    Codebase: * *
    Application-Name: Simple Applet Table

    Brief content description. “Permissions” property declares that applet code does not require any access to local resources at all. “Codebase” describes domains that we are going to use for. Please pay attention that I put two values, because for codebase and are different. The last one is quite self-explanatory.

  4. You already have JAR deployment profile, don’t you?  Open Project Properties, Navigate to deployments and open deployment profile for edit.
  5. Enable MANIFEST.MF in deployment profile and don’t forget to merge our text document.

  6. Compile and deploy your project to JAR file.
Applet prepared for signing and deployment.

Developer Certificate

Now we need a new certificate to sign our code. This time procedure a bit different, because jarsigner works with JKS storage and you should create one.

  1. Open new command window as Administrator. You shouldn't do it if you sign certificates against well known CA. In my case I have to add my private CA certificate into JRE keystore. 
  2. Create new key store and private key with keytool utility.

    \MyWork\jks>keytool -genkey -alias AppSign -keystore keysigner.jks -storepass welcome1 -keypass welcome1 -keysize 2048 -keyalg rsa -validity 1826

    What is your first and last name?
    [Unknown]: Michael M
    What is the name of your organizational unit?
    What is the name of your organization?
    [Unknown]: mmikhail
    What is the name of your City or Locality?
    [Unknown]: Naples
    What is the name of your State or Province?
    [Unknown]: Florida
    What is the two-letter country code for this unit?
    [Unknown]: US
    Is CN=Michael M, OU=Unknown, O=mmikhail, L=Naples, ST=Florida, C=US correct?
    [no]: y
  3. Create new certificate request

    \MyWork\jks>keytool -certreq -keystore keysigner.jks -alias AppSign -file michaelm.crq

    Enter keystore password:
  4. Transfer your request to the server and sign it with openSSL CA. I prefer WinSCP or MobaXTerm

    E:\MyWork\jks>"c:\Program Files (x86)\WinSCP\" /open root@rhas48

    Searching for host...
    Connecting to host...
    Using username "root".
    Starting the session...
    Reading remote directory...
    Session started.
    Active session: [1] root@rhas48
    winscp> put michaelm.crq
    michaelm.crq              |          1 KiB |    0.0 KiB/s | binary | 100%
  5. On server side sign it with OpenSSL

    [root@rhas48 ~]# openssl ca -in michaelm.crq -out michaelm.pem -days 1826 -md sha1

    Using configuration from /usr/local/openssl/openssl.cnf
    Enter pass phrase for /root/sslCA/private/cakey.pem:
    Check that the request matches the signature
    Signature ok
    Certificate Details:
            Serial Number: 4102 (0x1006)

    Certificate is to be certified until Jan 25 00:22:55 2020 GMT (1826 days)
    Sign the certificate? [y/n]:y

    1 out of 1 certificate requests certified, commit? [y/n]y
    Write out database with 1 new entries
    Data Base Updated
    [root@rhas48 ~]# openssl x509 -in michaelm.pem -out michaelm.crt -outform DER
    [root@rhas48 ~]#
  6. Return to the command window and get certificates back to workstation.

    winscp> get michaelm.crt

    michaelm.crt              |          4 KiB |    0.0 KiB/s | binary | 100%
    winscp> get sslCA/cacert.pem
    cacert.pem                |          1 KiB |    0.0 KiB/s | binary | 100%
    winscp> exit
  7. Install certificates to the keystore starting with my personal CA certificate first.

    \MyWork\jks>keytool -importcert -keystore keysigner.jks -trustcacerts  -alias mmikhail-ca -file cacert.pem

    Enter keystore password:
    Owner:, CN=Personal CA, O=mmikhail,


    Trust this certificate? [no]:  yes
    Certificate was added to keystore

    \MyWork\jks>keytool -importcert -keystore keysigner.jks  -alias AppSign -file michaelm.crt
    Enter keystore password:
    Certificate reply was installed in keystore
  8. I should make my own CA trusted for JRE on my computer.
    \MyWork\jks>keytool -importcert -keystore “C:\Program Files (x86)\Java\jre1.8.0_31\lib\security\cacerts” –storepass changeit -trustcacerts  -alias mmikhail-ca -file cacert.pem
    Trust this certificate? [no]:  yes
    Certificate was added to keystore
Now we are ready to sign our library and deploy it to Apache server.

Sign applet code

  1. sign applet library. Go to the project deployment directory and sign it with new key:

    \MyWork\OldSystemApps\SimpleApplet\deploy>jarsigner -keystore ..\..\..\jks\keysigner.jks simpleapplet.jar AppSign
    Enter Passphrase for keystore:

    Command jarsigner gets parameters as follow:

    • Key store and certificate location: -keystore <JKS file location>
    • Jar file name: simpleapplet.jar
    • Alias for private key to sign:  AppSign

  2. Transfer library to the server and make sure that it’s available to end users on new virtual host.

The last part is about Apex application .

Chasing Java 8

Last year brought multiple breaches in the security basement of middleware applications. While old systems still rely on weak protocols and compromised libraries modern versions of Java and browsers mushroom security restrictions. One of the latest – JRE 8 and Oracle Forms compatibility. Unlike JRE about Forms and code compatibility, it’s all about security enforcement. To minimize modifications on the user side you should meet the follow challenges:

  • JRE 8 has no “Medium” security level – All sources should be trusted and libraries signed. 
  • JRE 8 uses TLS 1.2 by default and does not download JAR libraries even from TLS 1.0 sources.
I have no Oracle Forms installed, however I use plain old Java Applet application, so configuration should be mostly the same. There are to many words for a single post so I had to split into parts:

I still have warnings mostly because I use non public Certificate Authority and have not implemented Online Certificate Status Protocol. You should be fine with any of big authorities.

Hope you it would be helpful.

One with System Administration

If you work for a company with existing certificate authority, or have service from any of big players – lucky you. Just look this through and get all necessary certificates with CLR and OCSP parts in it.
If you are not so lucky, you need to issue certificates:
  • Old system with SSLv3/TLSv1 support to expose our application
  • Codebase server with latest TLSv1.1 and TLSv1.2 support to publish our Forms/applets.
  • Personal certificate to sign Java libraries
Before we issue any new certificates, reconfigure CA settings.  Java 8 requires every verification for certificate with CRL (Certificate Revocation Lists) and with modern OCSP (Online Certificate Online Status) protocol.  I am going to skip OCSP server configuration (Latest OpenSSL versions allow implement it. Here is a good article).

Certificate Authority preparation

Locate OpenSSL default configuration file /usr/local/ and add information in section:
[ usr_cert ]
crlDistributionPoints = URI:

[ v3_ca ]
crlDistributionPoints = URI:

Of course, you should use your own server name. Save CA configuration and issue some certificates.

Server Certificates

I am going to create new virtual host with TLS 1.2 and of course it requires new certificate.
Create new certificate request for new virtual host name:
[root@rhas48 ~]# openssl req -new -nodes -newkey 2048 -days 1846 -sha1 -out sslCA/apps-demo-crq.pem -keyout sslCA/pps-demo-key.pem
Generating a 2048 bit RSA private key
writing new private key to 'apps-demo-key.pem'
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [US]:
State or Province Name (full name) [Florida]:
Locality Name (eg, city) [Naples]:
Organization Name (eg, company) [mmikhail]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:welcome1
An optional company name []:

Sign new request and issue certificate:
[root@rhas48 ~]# openssl ca -in sslCA/apps-demo-crq.pem -out sslCA/apps-demo-crt.pem -md sha1 -days 1826
Using configuration from /usr/local/openssl/openssl.cnf
Enter pass phrase for /root/sslCA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Certificate is to be certified until Jan 24 18:44:11 2020 GMT (1826 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@rhas48 ~]#

We already have certificate for server name, revoke the previous one and reissue certificate with additional fields.
[root@rhas48 ~]# openssl ca -revoke sslCA/rhas48-sha1-crt.pem
Using configuration from /usr/local/openssl/openssl.cnf
Enter pass phrase for /root/sslCA/private/cakey.pem:
Revoking Certificate 1004.
Data Base Updated
[root@rhas48 ~]#

I use the very same request to issue new certificate:
[root@rhas48 ~]# openssl ca -in apps-demo-crq.pem -out apps-demo-crt.pem -md sha1 -days 1826
Using configuration from /usr/local/openssl/openssl.cnf
Enter pass phrase for /root/sslCA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:

Certificate is to be certified until Jan 24 18:44:11 2020 GMT (1826 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@rhas48 ~]#

Copy new certificates and keys to Apache configuration directory (I use old configuration directory in /etc/httpd but you free to choose any).
[root@rhas48 ~]# cp  sslCA/apps-demo-crt.pem sslCA/apps-demo-key.pem /etc/httpd/conf/
[root@rhas48 ~]# cp  sslCA/rhs48-crt.pem sslCA/rhas48-key.pem /etc/httpd/conf/

Revocation List publication

Now we need to create certificate revocation list. It’s simple operation and for production purposes you need update revocation list. Local address for CRL file is /usr/local/apache2/htdocs/crl/crl.pem
[root@rhas48 ~]# mkdir –p /usr/local/apache2/htdocs/crl
[root@rhas48 ~]# openssl ca -keyfile sslCA/private/cakey.pem -cert sslCA/cacert.pem -gencrl -out /usr/local/apache2/htdocs/crl/crl.pem
Using configuration from /usr/local/openssl/openssl.cnf
Enter pass phrase for sslCA/private/cakey.pem:
[root@rhas48 ~]#

We are about to change Apache configuration.

Virtual Hosts configuration.

My goal is simulate the situation when you have one production server with SSLv3 and TLSv1 support and new virtual host to meet modern high security requirements. Here is my application configuration file /usr/local/apache2/config/apex/apex.conf below and quick explanation right here:

  • Virtual host on port 80 defines permanent redirect to protected server.
  • Virtual host on port 443 describes reverse proxy access to old business application (Oracle APEX 10g). To simulate old Web applications there is only SSLv3 and TLSv1 protocols allowed.
  • Listen port 445 to enable different TLS protocols on the same Apache instance and different server name. We need this port because with named virtual hosts you will have single SSL enabled port with common protocol versions. I am going to use this virtual host for CRL publications and as code base for applets in business applications.
<VirtualHost *:80>
 RewriteEngine On
 RewriteCond %{HTTPS} off
 RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
 Redirect permanent / https://%{HTTP_HOST}%{REQUEST_URI}
<IfModule ssl_module>

<VirtualHost *:443>
   SSLEngine on
DocumentRoot "/usr/local/apache2/htdocs"
SSLCertificateFile "/etc/httpd/conf/rhas48-sha1-cert.pem"
SSLCertificateKeyFile "/etc/httpd/conf/rhas48-key.pem"
       SSLProtocol  SSLv3 TLSv1
ErrorLog /var/log/httpd/apex-error_log
CustomLog /var/log/httpd/apex-access_log common
RewriteEngine On
RewriteRule ^/$ /apex/f?p=4500:1000 [R=301]
ProxyRequests On
<Proxy *>
Order allow,deny
Allow from all
ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

Listen 445
<VirtualHost *:445>
   SSLEngine on
   SSLProtocol TLSv1.2
SSLCertificateFile "/etc/httpd/conf/apps-crt.pem"
SSLCertificateKeyFile "/etc/httpd/conf/apps-key.pem"

DocumentRoot "/usr/local/apache2/htdocs"

Before apply new configuration, I also made some changes in hosts configuration. 
In etc/hosts file and add alias to server name:
# that require network functionality will fail.               localhost.localdomain localhost
XX.XX.XX.XX  rhas48

Create or modify configuration and restart your HTTP server:
[root@rhas48 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

Check new configuration and CRL accessibility:

Now it's time to get workstation ready.