Brown refusing to panic