There are a couple "walk in" campgrounds on the valley floor but they are still pretty much in the thick of things. The most prominent is
Camp 4, but while the cars may be farther away the campers are dense enough that you won't be any more likely to feel particularly natural.
On the other hand, it is the primary campground for the mountain climbers so the people watching is pretty nice with very toned bodies and people scurrying around in place people shouldn't be.