From rag horn, describing the animals as non-prime trophy specimens, i.e. "rags". Another explanation is that it is descriptive of how the new antlers have a velvety covering that sloughs off each year and can hang like rags.
raghorn (plural raghorns)