Security handshaking: encrypted password

Sybase Servers use encrypted password handshakes if the client requests password encryption. Encrypted password security handshaking occurs while the connection to the server is being established.

When password encryption is enabled, the server receives the user passwords and remote-server passwords as follows:

1. Client-Library initially sends a dummy password to the server consisting of a zero-length string.

2. The server responds by asking the client for the encrypted passwords and sending an encryption key to the client.

   • If the client program has installed an encryption callback, Client-Library invokes the callback once for the local password and once for each remote-server password. Each time Client-Library invokes the encryption callback, it supplies the password to be encrypted and the encryption key as arguments.

   • If the client program has not installed an encryption callback, Client-Library performs the default encryption for all passwords.

Password encryption is disabled by default, so applications that need password encryption must set the CS_SEC_EXTENDED_ENCRYPTION or CS_SEC_ENCRYPTION property to CS_TRUE before calling ct_connect. Below are sample codes you can use to enable password encryption.

CS_BOOL boolval;
/* Enable password encryption for the connection attempt. */
boolval = CS_TRUE;

if (ct_con_props(conn, CS_SET, CS_SEC_ENCRYPTION, (CS_VOID *)&boolval,
    CS_UNUSED,(CS_INT *)NULL) != CS_SUCCEED)
{
    fprintf(stdout,"ct_con_props(SEC_ENCRYPTION) failed. Exiting\n");
    (CS_VOID)ct_con_drop(conn);
    (CS_VOID)ct_exit(ctx, CS_FORCE_EXIT);
    (CS_VOID)cs_ctx_drop(ctx);
    exit(1);
}

... 
CS_INT Ex_encryption = CS_TRUE;
CS_INT Ex_nonencryptionretry = CS_FALSE; 
...
main()
{
    ...
    /*
    ** This needs to be called before calling ct_connect()
    */
    ret = ct_con_props(connection, CS_SET, CS_SEC_EXTENDED_ENCRYPTION,
                       &Ex_encryption, CS_UNUSED, NULL);
    EXIT_ON_FAIL(context, ret, "Could not set extended encryption");

    ret = ct_con_props(connection, CS_SET, CS_SEC_NON_ENCRYPTION_RETRY,
                       &Ex_nonencryptionretry, CS_UNUSED, NULL);
    EXIT_ON_FAIL(context, ret, "Could not set non encryption retry");

    ....
}

Password encryption is performed either by Client-Library’s default encryption handler or by an application handler installed with ct_callback.

The default encryption handler performs the encryption expected by Adaptive Server Enterprise. Applications that connect to Adaptive Server Enterprise or an Open Server gateway to Adaptive Server Enterprise should rely on the default encryption. Most applications fall into this category.

Applications that require an encryption handler include the following:

• Open Server gateways that connect to an Adaptive Server Enterprise must support password encryption with an encryption callback that obtains encrypted passwords from the gateway’s client (through srv_negotiate) and forwards each password to the remote server (through the callback’s output parameters).

• Client applications that require a custom password encryption technique (for example, applications that connect to a custom Open Server) must install a custom encryption callback that performs the encryption expected by the server.