Defining an encryption callback

An encryption callback is defined as follows:

CS_RETCODE CS_PUBLIC 
 encrypt_cb(connection, pwd, pwdlen,
            key, keylen, buf, buflen, outlen)

CS_CONNECTION    *connection;
 CS_BYTE                  *pwd;
 CS_INT                      pwdlen;
 CS_BYTE                  *key;
 CS_INT                      keylen;
 CS_BYTE                  *buffer;
 CS_INT                      buflen;
 CS_INT                      *outlen;

where:

An encryption callback should return CS_SUCCEED to indicate that the password has been successfully encrypted. If the encryption callback returns a value other than CS_SUCCEED, Client-Library aborts the connection attempt, causing ct_connect to return CS_FAIL.


Password encryption in gateway applications

To handle encrypted passwords, a gateway application must:

When the gateway calls ct_connect to connect to the remote server:

  1. The remote server responds with an encryption key, causing Client-Library to trigger the encryption callback.

  2. The encryption callback passes the key on to the gateway’s client.

  3. The gateway’s client encrypts the password and returns it to the encryption callback.

  4. The encryption callback places the encrypted password into *buffer, sets *outlen, and returns a status code to Client-Library.

Client-Library calls the encryption once to encrypt the password defined by CS_PASSWORD, and one additional time for each remote server password defined by ct_remote_pwd.

A gateway to Adaptive Server or SQL Server must take special steps to make sure that encrypted remote passwords are handled correctly. The first time the encryption callback is called for a connect attempt, the gateway must perform the following actions:

  1. Clear the default remote password with ct_remote_pwd(CS_CLEAR).

    ct_connect creates a default remote password if the gateway has defined no remote passwords before calling ct_connect. The gateway must clear this default.

  2. Challenge the gateway’s client for encrypted local and remote passwords with srv_negotiate.

  3. Call ct_remote_pwd once for each encrypted remote password.

  4. Place the encrypted local password into *buffer and set *outlen to its length.

  5. Return CS_SUCCEED if no error occurred.

Each subsequent invocation of the callback should return one of the encrypted remote passwords read from the gateway’s client in response to the challenge.

A gateway forwards the encryption key and reads the client’s response with Server-Library calls. See srv_negotiate in the Open Server Server-Library/C Reference Manual.

See “Choosing a network security mechanism” for more information.