Colour in a lyophilised cake is worth noting for the less common compounds specifically, because some of them are genuinely not white and a member expecting white will report a problem that is not one.
The rule of thumb is fine; the edge cases are where it earns its keep.