Allen was born May 25, 1924, in Louisville, KY, beginning clarinet lessons at the age of ten. After enlisting with the U.S. Army's renowned Buffalo Soldiers in 1942, he played clarinet and alto saxophone with the 17th Division Special Service Band, subsequently forming a trio with pianist Art Simmons and guitarist Don Byas while stationed in Paris. After spending the late '40s touring and recording behind James Moody, Allen studied at the Paris Conservatory of Music before returning stateside in 1951, settling in Chicago and forming his own dance band. He also began writing his first original compositions.

Circa 1956, Allen met Sun Ra, joining the pianist's legendary Arkestra two years later. He would go on to lead its reed section for more than four decades, over time earning renown as one of the most distinctive and original saxophonists of the postwar era. In tandem with tenorist Gilmore and baritone saxophonist Pat Patrick, Allen played on more than 200 Sun Ra recordings. He also developed his own reed instrument (dubbed the "morrow") by attaching a saxophone mouthpiece to an open-hole wooden body but failed to patent his creation, which is now commercially available under different names.

Whenever the Arkestra went on hiatus, Allen moonlighted with Olatunji and his Drums of Passion, even learning to build and play the West African multi-stringed instrument the kora; he also guested on live dates and recordings headlined by a multitude of next-generation musicians, including jam band Phish, avant rock combo Sonic Youth, and hip-hop upstarts Digable Planets. When Sun Ra died in 1993, the Arkestra's reins were handed to Gilmore, who himself passed away two years later; at that point Allen assumed control, leading the 18-piece ensemble well into the next millennium. In addition to giving master classes, lectures, and demonstrations of Sun Ra's enduring creative principles, Allen also founded the El Ra label, home to such latter-day Arkestra recordings as A Song for the Sun and Music for the 21st Century.