Defining an encryption callback

The encryption callback prototype for extended and normal password encryption are defined below.

Normal password encryption

CS_RETCODE CS_PUBLIC encrypt_cb(connection, pwd,
              pwdlen, key, keylen, buf, buflen, outlen)

CS_CONNECTION        *connection;
CS_BYTE              *pwd;
CS_INT               pwdlen;
CS_BYTE              *key;
CS_INT               keylen;
CS_BYTE              *buffer;
CS_INT               buflen;
CS_INT                *outlen;

where:

Extended password encryption

CS_RETCODE extended_encrypt_cb(
      CS_CONNECTION *connection,
      CS_BYTE       *pwd,
      CS_INT        pwdlen,
      CS_INT        *ciphersuite,
      CS_BYTE       *pubkey, 
      CS_INT        pubkeylen,
      CS_VOID       *buffer,
      CS_INT        buflen,
      CS_INT         *outlen)

where:

An encryption callback should return CS_SUCCEED to indicate that the password has been successfully encrypted. If the encryption callback returns a value other than CS_SUCCEED, Client-Library aborts the connection attempt, causing ct_connect to return CS_FAIL.


Password encryption in gateway applications

To handle encrypted passwords, a gateway application must:

When the gateway calls ct_connect to connect to the remote server:

  1. The remote server responds with an encryption key, causing Client-Library to trigger the encryption callback.

  2. The encryption callback passes the key on to the gateway’s client.

  3. The gateway’s client encrypts the password and returns it to the encryption callback.

  4. The encryption callback places the encrypted password into *buffer, sets *outlen, and returns a status code to Client-Library.

    • If the callback returns CS_SUCCEED, Client-Library sends the encrypted password to the remote server.

    • If the callback returns CS_FAIL, Client-Library aborts the connection process, causing ct_connect to return CS_FAIL.

Client-Library calls the encryption once to encrypt the password defined by CS_PASSWORD, and one additional time for each remote server password defined by ct_remote_pwd.

A gateway to Adaptive Server Enterprise must take special steps to make sure that encrypted remote passwords are handled correctly. The first time the encryption callback is called for a connect attempt, the gateway must perform the following actions:

  1. Clear the default remote password with ct_remote_pwd (CS_CLEAR).

    ct_connect creates a default remote password if the gateway has defined no remote passwords before calling ct_connect. The gateway must clear this default.

  2. Challenge the gateway’s client for encrypted local and remote passwords with srv_negotiate.

  3. Call ct_remote_pwd once for each encrypted remote password.

  4. Place the encrypted local password into *buffer and set *outlen to its length.

  5. Return CS_SUCCEED if no error occurred.

Each subsequent invocation of the callback should return one of the encrypted remote passwords read from the gateway’s client in response to the challenge.

A gateway forwards the encryption key and reads the client’s response with Server-Library calls. See srv_negotiate in the Open Server Server-Library/C Reference Manual.

See “Choosing a network security mechanism”.