Swansea captain Ashley Williams admits his side's defensive frailties are putting their Premier League future in doubt.