Observations in biology are always samples. Even in highly controlled experimental systems we do not know the complete set of biological processes determining the outcome of the interaction, or how they are affected by microscopic fluctuations in conditions. The situation in macro-scale observational studies affords even less access to the whole truth; we never know the true data-generating process and are constrained to observing stochastic realizations of it. Typically, it is also true that our observations are inexact adding measurement error to the noise in our data. In spite of these epistemic limitations science seems to be effective in revealing how the world works. Recently, however, scientific methodology has been questioned after several high-profile studies failed to replicate important results across a range of disciplines. An important step in all scientific methods is the inductive generalization from the particular conditions under which data are collected, to the claim of the probable validity of our findings in the wider world. In order for that warrant of induction to be valid we rely on an implicit assumption that the systems we study are ergodic; i.e. that the average values of their parameters over the long term are equal to the averages we would obtain if we had access to the statistical ensemble of instances of the system evolving separately at the same time. Strictly, the statistical ensemble is a purely abstract concept, not to be confused with a sample of realizations of a system collected over space; this subtlety is often ignored and space- and time- averages of parameter values are treated as equivalent. In relation to reproducibility, we are faced with a conundrum. If systems are not ergodic, we should not expect robust reproducibility. If systems areergodic measurements should be reproducible in principle, but observations may not sufficiently characterize the variability of the system to allow us to know what the reproducible behavior is. When observations do not sufficiently characterize variability, lack of reproducibility is to be expected, even if nothing about the system being measured changes between bouts of observation. We illustrate these points by sampling from the output of a stochastic, second order, linear, auto-regressive process model.