When using SySAM sub-capacity licensing, set the SYBASE_SAM_CAPACITY environment variable before installing, or, you can copy in the license key after installation.
Follow the instructions provided in Configuring SySAM Sub-capacity in the SySAM Users Guide. However, start your installer instead of starting the sub-capacity-enabled SAP product. The installer includes the sysamcap utility located in sysam_utilities/bin.