What does yellow mean in this context?

In Catcher in the Rye, Holden says, “It's no fun to be yellow. Maybe I'm not all yellow. I don't know. I think maybe I'm just partly yellow and partly the type that doesn't give much of a damn if they lose their gloves.”