Brown's wife was Matilda Frazier. They were married, but date and place are unknown. They had no known children.