I've been helping copy-edit a book on digital media, and a lot of the book is devoted to the subject of "color", so I think I can help here.
The problem, really, is that there are two definitions of "color" used in the English language.
One is that of chrominance, or hue -- i.e. red, orange, yellow, green, blue, indigo, violet -- which is considered in absence of luminance (where it falls on the black-white scale) and saturation (where it falls on the gray-neon scale). Under this definition, black, white, and perfect greys are not colors (the formula for converting from red-green-blue intensity to hue-saturation-luminance intensity includes a divide by zero in this case, and thus hue is undefined.)
The other definition is of the combined result of the three factors -- a certain point in the 3-dimensional space formed by the axis of luminance, with saturation going inwards and outwards from the axis of luminance, and hue being the circular third axis. By this definition, every color has a point, and so black, white, and greys in between are colors.