

I also have very good color vision.
For me it heavily depends on the context. For example I have spent some time working out a color palette for my wardrobe, so now the colors I use there are heavily associated for me.
Without any context, it’s similar to your grouping, except for the second group: oranges go nearly all to the reds, yellows go to the greens, browns go to black and greys.
Others here have already made a lot of good points, so I just want to add one I haven’t seen addressed yet.
Having only one friend or one group of friends can be “dangerous”, because what happens when you lose them? I know, this is not something we like to think about, but people have accidents or get sick or die suddenly or you have a huge argument or… And then you have no one to turn to to help you to deal with the situation.
If you have several “independent” friends or friend groups, you have a much stronger support network. You have someone else to turn to, even if it’s just for venting about problems with your other friends. For me personally this kind of social security – knowing I won’t ever have to deal with things on my own – is very important and reduces a lot of my anxiety about the future.
On a side note this reasoning also applies to maintaining friendships during a relationship.