We have found a remarkably large response of the transition temperature of CaFe2As2 single crystals grown from excess FeAs to annealing and quenching temperature. Whereas crystals that are annealed at 400ˆC exhibit a first-order phase transition from a high-temperature tetragonal to a low-temperature orthorhombic and antiferromagnetic state near 170 K, crystals that have been quenched from 960ˆC exhibit a transition from a high-temperature tetragonal phase to a low-temperature, nonmagnetic, collapsed tetragonal phase below 100 K. By use of temperature-dependent electrical resistivity, magnetic susceptibility, x-ray diffraction, Mössbauer spectroscopy, and nuclear magnetic resonance measurements we have been able to demonstrate that the transition temperature can be reduced in a monotonic fashion by varying the annealing or quenching temperature from 400ˆ to 850ˆC with the low-temperature state remaining antiferromagnetic for transition temperatures larger than 100 K and becoming collapsed tetragonal, nonmagnetic for transition temperatures below 90 K. This suppression of the orthorhombic-antiferromagnetic phase transition and its ultimate replacement with the collapsed tetragonal, nonmagnetic phase is similar to what has been observed for CaFe2As2 under hydrostatic pressure. Transmission electron microscopy studies indicate that there is a temperature-dependent width of formation of CaFe2As2 with a decreasing amount of excess Fe and As being soluble in the single crystal at lower annealing temperatures. For samples quenched from 960ˆC there is a fine (of order 10 nm) semiuniform distribution of precipitate that can be associated with an average strain field, whereas for samples annealed at 400ˆC the excess Fe and As form mesoscopic grains that induce little strain throughout the CaFe2As2 lattice.