A conservative Milwaukee financier and big backer of Gov. Scott Walker is walking away from a contract to manage $1 million in taxpayer money after a Journal Sentinel story raised a question about whether the arrangement violated federal conflict of interest rules.