Tar Baby is definitely taken as a slur - handily illustrated in this classic sketch from Saturday Night Live with Chevy Chase and Richard Pryor:
www.youtube.com/watch?v=A6341HeJDgU
I remember this one well. It appeared in audio form on the SNL lp record released sometime in the late 70s. I recall kids at my nearly-all-white school memorizing and performing it.