Luton and Huddersfield players had to be separated after tempers flared following Elijah Adebayo's penalty miss.