Why does a good flat R/G/B go so well with black or white?

Because black and white are not colors by themselves, they're either a combination of all of them or none, it's difficult to mess up a color combination if either is dominant

Unless the color combo is gross. But is white being all color, and black being the absence of color the reason?

If either black or white are dominant it still difficult to mess up (unless the person knows next to nothing about color theory and design). Colors will clash with one another but won't clash with black and white because they're not colors, they're value, levels of light, so yes. That is talking about light, if we talk pigment (painting color theory) things get a bit more complicated.