We present measurements of long-range angular correlations and the transverse momentum dependence of elliptic flow v(2) in high-multiplicity p + Au collisions at root S-NN = 200 GeV. A comparison of these results to previous measurements in high-multiplicity d + Au and He-3+Au collisions demonstrates a relation between v(2) and the initial collision eccentricity epsilon(2), suggesting that the observed momentum-space azimuthal anisotropies in these small systems have a collective origin and reflect the initial geometry. Good agreement is observed between the measured v(2) and hydrodynamic calculations for all systems, and an argument disfavoring theoretical explanations based on initial momentum-space domain correlations is presented. The set of measurements presented here allows us to leverage the distinct intrinsic geometry of each of these systems to distinguish between different theoretical descriptions of the long-range correlations observed in small collision systems.