Formal security analysis of PKCS#11 and proprietary extensions

PKCS#11 defines an API for cryptographic devices that has been widely adoptedin industry. However, it has been shown to be vulnerable to a variety of attacks thatcould, for example, compromise the sensitive keys stored on the device. In this paper, weset out a formal model of the operation of the API, which differs from previous securityAPI models notably in that it accounts for non-monotonic mutable global state. We givedecidability results for our formalism, and describe an implementation of the resulting de-cision procedure using the model checker NuSMV. We report some new attacks and provethe safety of some configurations of the API in our model. We also analyse proprietaryextensions proposed by nCipher (Thales) and Eracom (Safenet), designed to address theshortcomings of PKCS#11.