Smalls remained involved with jazz and related vocal music throughout his career, but by the '50s and '60s could also be called something of a soul and rhythm and blues man, working closely as a bandleader for singers Clyde McPhatter, Smokey Robinson, and Brook Benton. Jazz buffs who look down their nose at what they regard as the simple nature of popular soul music need to be reminded, by facts if not a smack in the head, of the virtuosity of many of the instrumentalists involved. The brilliant, innovative bandleader and pianist Earl Hines was the man who put a halt to Smalls' aforementioned musical cotton-picking, utilizing this new hire in the unique doubling position of trombone and second piano.

Smalls worked with Hines in this capacity from 1942 through 1946, then accompanied singer Billy Eckstine for two years and enjoyed a brief stint with saxophonist Earl Bostic. In 1951 Smalls suffered a serious auto accident involving injuries which took several years to recover from. When he went back to work in the mid-'50s, Smalls continued playing jazz with the superb trombonist Bennie Green but also moved his way into accompanying McPhatter in the popular styles developing under names such as doo wop and rhythm and blues.

Benton hired him as pianist and musical director in the late '50s for more than seven years, a period in which Smalls also developed similar clients such as soul hit parade colossus Smokey Robinson. Smalls played with both Ella Fitzgerald and the Reuben Phillips Big Band during the '60s and in the '70s was heavily associated with a big band led by arranger and composer Sy Oliver. Resurging interest in classic jazz veterans led to some of Smalls' grandest recording opportunities in his later years, including a solo album entitled The Man I Love in 1979.