On Monadic NP vs. Monadic co-NP

It is a well-known result of Fagin that the complexity class NP coincides with the class of problems expressible in existential second-order logic (1 ;). Monadic NP is the class of problems expressible in monadic Z;, i.e., 1: with the restriction that the second-order quantifiers range only over sets (as opposed to ranging over, say, binary relations). We prove that connectivity of finite graphs is not in monadic NP, even in the presence of arbitrary built-in relations of moderate degree (that is, degree (log n) " ")). This extends earlier results of Fagin and de Rouge-mont. Our proof uses a combination of three techniques: (1) an old technique of Hanf for showing that two (infinite) structures agree on all first-order sentences, under certain conditions, (2) a recent new approach to second-order Ehrenfeucht-Fraisse games by Ajtai and Fagin, and (3) playing Ehrenfeucht-Fraisse games over random structures (this was also used by Ajtai and Fagin). Regarding (1), we give a version of Hanf's result that is better suited for use as a tool in inexpressibility proofs for classes of finite structures. The power of these techniques is further demonstrated by using them (actually, using just the first two techniques) to give a very simple proof of the separation of monadic NP from monadic co-NP without the presence of built-in relations. 0 1995 Academic Press, Inc.