In this paper we introduce a notion of duality for matrix valued orthogonal polynomials with respect to measure supported on the nonnegative integers. We show that dual families are closely related certain difference operators acting polynomials. These belong so called Fourier algebras, which play key role in construction families. order illustrate duality, describe family Charlier type explici...