In terms of Section 35 (1) of the new Children’s Act the father commits a criminal offence if he refuses to return the child after a contact weekend or holiday.
Typically, a deed of settlement that is made an order of court gives primary place of residence of a minor child to the mother. If the father (and indeed the child) thinks that the child would be better off living with him instead of the mother he can’t simply refuse to return the child after a contact weekend or holiday.
(1) Any person having care or custody of a child who, contrary to an order of any court or to a parental responsibilities and rights agreement that has taken effect as contemplated in section 22 (4), refuses another person who has access to that child or who holds parental responsibilities and rights in respect of that child in terms of that order or agreement to exercise such access or such responsibilities and rights or who prevents that person from exercising such access or such responsibilities and rights is guilty of an offence and liable on conviction to a fine or to imprisonment for a period not exceeding one year.
On a practical level, what happens is that your ex would complain to her local police station and the public prosecutor would write you a letter demanding that you return the child urgently. If you fail to do so the police will be instructed to arrest you.