Validating the server by its certificate

Any Open Client/ Open Server connection to an SSL-enabled server requires that the server have a certificate file, which consists of the server’s certificate and an encrypted private key. The certificate must also be digitally signed by a CA.

Open Client applications establish a socket connection to Adaptive Server similarly to the way that existing client connections are established. Before any user data is transmitted, an SSL handshake occurs on the socket when the network transport-level connect call completes on the client side and the accept call completes on the server side.

To make a successful connection to an SSL-enabled server:

NoteYou may choose to install SSL validation callback, which intercepts SSL handshakes and overrides SSL validation checks. SSL validation callback is installed with ct_callback using CS_SSLVALIDATE_CB.

When establishing a connection to an SSL-enabled Adaptive Server, Adaptive Server loads its own encoded certificates file at start-up from:

UNIX$SYBASE/$SYBASE_ASE/certificates/servername.crt

Windows – %SYBASE%\%SYBASE_ASE%\certificates\servername.crt

where servername is the name of the Adaptive Server as specified on the command line when starting the server with the -S flag or from the server’s environment variable $DSLISTEN.

Other types of servers may store their certificate in a different location. See the vendor-supplied documentation for the location of your server’s certificate.


Validation in an SDC environment

The default behavior for SSL validation in Open Client and Open Server is to compare the common name in the server certificate with the server name specified by ct_connect. However, in a Shared Disk Cluster (SDC) environment, a client may specify the SSL certificate common name independent of the server name or the SDC instance name. A client may connect to an SDC by its cluster name, which represents multiple server instances, or to a specific server instance.

Open Client and Open Server support common name validation in an SDC environment by allowing the client to use a transport address to specify the common name used in certificate validation. The ASE SSL certificate common name can therefore be different from the server or cluster name. The transport address can be specified in one of the directory services like the interfaces file, an LDAP or NT registry, or through the connection property CS_SERVERADDR.

The following is an example of an interfaces file for an SSL-enabled ASE and cluster for UNIX:

CLUSTERSSL
query tcp ether hostname1 5000 ssl="CN=name1"
query tcp ether hostname2 5000 ssl="CN=name2"
query tcp ether hostname3 5000 ssl="CN=name3"

ASESSL1
master tcp ether hostname1 5000 ssl="CN=name1"
query tcp ether hostname1 5000 ssl="CN=name1"

ASESSL2
master tcp ether hostname2 5000 ssl="CN=name2"
query tcp ether hostname2 5000 ssl="CN=name2"

ASESSL3
master tcp ether hostname3 5000 ssl="CN=name3"
query tcp ether hostname3 5000 ssl="CN=name3"

The following is an example of an interfaces file for an SSL-enabled ASE and cluster for Windows:

[CLUSTERSSL]
query=tcp,hostname1,5000, ssl="CN=name1"
query=tcp,hostname2,5000, ssl="CN=name2"
query=tcp,hostname3,5000, ssl="CN=name3"

[ASESSL1]
master=tcp,hostname1,5000, ssl="CN=name1"
query=tcp,hostname1,5000, ssl="CN=name1"

[ASESSL2]
master=tcp,hostname2,5000, ssl="CN=name2"
query=tcp,hostname2,5000, ssl="CN=name2"

[ASESSL3]
master=tcp,hostname3,5000, ssl="CN=name3"
query=tcp,hostname3,5000, ssl"CN=name3"