Mortgage rates aren't the only thing to blame.