For the t-test each measurement comes from a protein in a single sample (so N = # of samples). Fisher's Exact test (FET) with spectrum counting, on the other hand, is applied to a 2x2 contingency table where each measurement is a spectrum and contributes to a count in exactly one cell of the table (so N = total count of measurements = sum of cells in 2x2 table = # of spectra). It is actually pretty misleading to present these two tests in such a similar way because they are answering very different questions, and really the t-test (or one of its nonparametric equivalents) is answering the question that researchers probably want to be asking. In the past when sample counts and spectrum counts were low, there was perhaps some value in using Fisher's Exact test the way it is implemented in Scaffold (protein vs background, category A vs category B), but these days we do not recommend it because it really isn't operating at the correct level and gives much lower p-values.
See our release notes for when we removed this test:
https://www.proteomesoftware.com/release-notes?product=scaffold-5
See this aricle for more general information on the FET:
https://en.wikipedia.org/wiki/Fisher's_exact_test