The encryption callback prototype for extended and normal password encryption are defined below.
CS_RETCODE CS_PUBLIC encrypt_cb(connection, pwd, pwdlen, key, keylen, buf, buflen, outlen)
CS_CONNECTION *connection; CS_BYTE *pwd; CS_INT pwdlen; CS_BYTE *key; CS_INT keylen; CS_BYTE *buffer; CS_INT buflen; CS_INT *outlen;
where:
connection is a pointer to the CS_CONNECTION structure representing the connection that is logging in to the server.
pwd is a user password or a remote-server password to be encrypted. A user password matches the value of the CS_PASSWORD connection property. A remote-server password matches the string passed to ct_remote_pwd. The pwd string is not always null-terminated.
pwdlen is the length, in bytes, of the password.
key is the key that the encryption callback uses to encrypt the password. The encryption key is supplied by the remote server.
keylen is the length, in bytes, of the encryption key.
buffer is a pointer to a buffer. The encryption callback should place the encrypted password in this buffer. This buffer is allocated and freed by Client-Library. Its length is described by buflen.
buflen is the length, in bytes, of the *buffer data space.
outlen is a pointer to a CS_INT. The encryption callback must set *outlen to the length of the encrypted password in *buffer.
CS_RETCODE extended_encrypt_cb( CS_CONNECTION *connection, CS_BYTE *pwd, CS_INT pwdlen, CS_INT *ciphersuite, CS_BYTE *pubkey, CS_INT pubkeylen, CS_VOID *buffer, CS_INT buflen, CS_INT *outlen)
where:
connection is a pointer to the CS_CONNECTION structure representing the connection that is logging in to the server.
pwd is a user password or a remote-server password to be encrypted. A user password matches the value of the CS_PASSWORD connection property. A remote-server password matches the string passed to ct_remote_pwd. The pwd string is not always null-terminated.
pwdlen is the length, in bytes, of the password.
ciphersuite is a pointer to the ciphersuite used to encrypt the password. This parameter is not used by the default encryption.
pubkey is a pointer to the public key used to encrypt the password.
pubkeylen is the length, in bytes, of the public key.
buffer is a pointer to a buffer. The encryption callback should place the encrypted password in this buffer. This buffer is allocated and freed by Client-Library. Its length is described by buflen.
buflen is the length, in bytes, of the *buffer data space.
outlen is a pointer to a CS_INT to store the length of the newly accepted password. The encryption callback must set *outlen to the length of the encrypted password in *buffer.
An encryption callback should return CS_SUCCEED to indicate that the password has been successfully encrypted. If the encryption callback returns a value other than CS_SUCCEED, Client-Library aborts the connection attempt, causing ct_connect to return CS_FAIL.
To handle encrypted passwords, a gateway application must:
Supply an encryption callback routine.
Call ct_callback to install the encryption callback either at the context level or for a specific connection.
Call ct_con_props to set the CS_SEC_EXTENDED_ENCRYPTION or CS_SEC_ENCRYPTION property to CS_TRUE.
When the gateway calls ct_connect to connect to the remote server:
The remote server responds with an encryption key, causing Client-Library to trigger the encryption callback.
The encryption callback passes the key on to the gateway’s client.
The gateway’s client encrypts the password and returns it to the encryption callback.
The encryption callback places the encrypted password into *buffer, sets *outlen, and returns a status code to Client-Library.
If the callback returns CS_SUCCEED, Client-Library sends the encrypted password to the remote server.
If the callback returns CS_FAIL, Client-Library aborts the connection process, causing ct_connect to return CS_FAIL.
Client-Library calls the encryption once to encrypt the password defined by CS_PASSWORD, and one additional time for each remote server password defined by ct_remote_pwd.
A gateway to Adaptive Server Enterprise must take special steps to make sure that encrypted remote passwords are handled correctly. The first time the encryption callback is called for a connect attempt, the gateway must perform the following actions:
Clear the default remote password with ct_remote_pwd (CS_CLEAR).
ct_connect creates a default remote password if the gateway has defined no remote passwords before calling ct_connect. The gateway must clear this default.
Challenge the gateway’s client for encrypted local and remote passwords with srv_negotiate.
Call ct_remote_pwd once for each encrypted remote password.
Place the encrypted local password into *buffer and set *outlen to its length.
Return CS_SUCCEED if no error occurred.
Each subsequent invocation of the callback should return one of the encrypted remote passwords read from the gateway’s client in response to the challenge.
A gateway forwards the encryption key and reads the client’s response with Server-Library calls. See srv_negotiate in the Open Server Server-Library/C Reference Manual.
