There was a huge panic back in 1873 that caused the Stock market to collapsed. Because of this, the government couldn't afford reconstruction any longer.
and there was also some racist pseudo Science that said something about white superiority about blacks.
