A significant barrier to achieving the radial velocity (RV) measurement accuracy and precision required to characterize terrestrial mass exoplanets is the existence of time-variable features in the measured spectra, from both telluric absorption and stellar variability, which affect measured line shapes and can cause apparent RV shifts. Reaching the desired accuracy using traditional techniques often requires avoiding lines contaminated by stellar variability and/or changing tellurics, and thus discarding a large fraction of the spectrum, lowering precision. New data-driven methods can help achieve extremely precise and accurate RVs by enabling the use of a larger fraction of the available data. While there exist methods for modeling telluric features or the stellar variability individually, there is a need for additional tools that are capable of modeling them simultaneously at the spectral level. Here we present StellarSpectraObservationFitting.jl (SSOF), a Julia package for measuring Doppler shifts and creating data-driven models (with fast, physically-motivated Gaussian Process regularization) for the time-variable spectral features for both the telluric transmission and stellar spectrum, while accounting for the wavelength-dependent instrumental line-spread function. We demonstrate SSOF's state-of-the-art performance on data from the NEID RV spectrograph on the WIYN 3.5m Telescope for multiple stars. We show SSOF's, ability to accurately identify and characterize spectral variability and provide 2-6x smaller photon-limited errors over the NEID CCF-based pipeline and match the performance of SERVAL, a leading template-based pipeline, using only observed EPRV spectra.