Conservation agriculture (CA) can sustainably address soil degradation and improve crop yields. However, the success of CA amongst South African smallholder farmers is not known. The aims of the study were to find out: (1) the extent of CA practice by the Eastern Cape smallholder farmers; (2) perceptions towards CA amongst smallholder farmers with some history of practising the technology; and (3) the impact of practised CA components on soil quality indicators. Diagnostic survey techniques and soil sampling in farmers’ fields were employed to gather data from five districts of the Eastern Cape, South Africa. The most common CA principle adopted by surveyed farmers was no-till (34.81%), whilst crop rotation and residue retention were practised by only 25.93% and 22.22% of the farmers, respectively. Education level and CA training influenced the likelihood of a farmer to practise no-till farming, whilst the likelihood of farmers to retain residues was influenced by education level and access to grazing lands. Lack of appropriate equipment and costly herbicides were the major constraining factors to practising CA. Crop residue retention conflicted with the common practice of free-range livestock grazing. Cabbage, maize and beans were the most common crops of choice for the few farmers that practised crop rotations. No significant (p > 0.05) improvement on soil quality indicators was observed with CA compared to the conventional farming method. The noted high dependency on government grants by the smallholder farmers could be a disincentive towards the adoption of agricultural innovations such as CA. Identification of practical key CA entry points is recommended, bearing in mind the noted impediments to CA adoption.