A.J. Brown played college football at the University of Mississippi, also known as Ole Miss. He committed to play both college baseball and college football at Ole Miss in February 2016. However, he was drafted as an outfielder by the San Diego Padres in the 19th round of the 2016 Major League Baseball Draft and signed with the Padres, which prevented him from playing baseball at Ole Miss, but he was still eligible to play football. Brown majored in General Studies with an emphasis in Education, Journalism, and Legal Studies at Ole Miss.