Representatives from across politics have called for the government to do more to deliver on affordable housing, arguing that the current model is not working.