Former Wales and Lions loose forward Colin Charvis says there is not enough diversity at the top level of sport - and outside sport.