Russel Crowe as General Maximus in the movie Gladiator
According to Roman terms, every person participating in gladiatorial fights was put under infamy, a loss of honour. This was associated with the deprivation of most public rights, limitation of legal capacity, deprivation of legal protection (e.g. torture could be used). In a legal sense, therefore, each gladiator (with very few exceptions) was on the margins of Roman society.
Even auctorati, so free ones who volunteered suffered the same handicap. It happened that people of the upper classes (equites, or more broadly the concepts of honestiores) appeared in the arena, but it always met with a shock to the environment. Ultimately, people from high society were banned from fighting during the Severan dynasty.