The remainder term $R(\lambda)$ for the spectral counting function $N(\lambda)$ likely encodes a great deal of dynamical information for the system at hand. For $\Omega \subset \mathbb{R}^n$, a piecewise smooth bounded domain, we prove an omega bound that depends on the dimension of the fixed point set of the billiard map; the approach taken is through boundary trace expansions. This is the first dynamical lower bound established in settings with boundary, at least to the knowledge of the authors. As a corollary, $R(\lambda)$ for the Bunimovich stadium is $\Omega(\lambda^{1/2})$, hence confirming a conjecture of Sarnak.