By 1959, Rollins had become frustrated with what he perceived as his own musical limitations and took the first – and most famous – of his musical sabbaticals. While living on the Lower East Side of Manhattan, he ventured to the pedestrian walkway of the Williamsburg Bridge to practice, in order to avoid disturbing a neighboring expectant mother.

Today, a fifteen-story apartment building named "The Rollins" stands on the Grand Street site where he lived. Almost every day from the summer of 1959 through the end of 1961, Rollins practiced on the bridge, next to the subway tracks. Rollins admitted that he would often practice for 15 or 16 hours a day, no matter what season.