Delete (minetestfolder)/debug.txt if it exists, then put debug_log_level=1 into your minetest.conf. Make the error happen again, then copy&paste the error messages near the bottom of debug.txt.
As for numbering your colors, you're on the right track. Maybe do it like so (yes, I think in 8-bit terms :-) )...
- There are 12 hues, so that fits into 4 bits. Just number them from 0, going clockwise around the color wheel: 0=red, 1=orange, 2=yellow, 3=lime, 4=green, etc.
- There are four shades, so that's 2 more bits (0=dark, 1=medium, 2=full, 3=light).
- There is are two saturation levels, for one more bit (0=50%, 1=100%).
- There are colors and there are greys, so you need a 1-bit flag for that (0=this is a shade of grey, 1=this is a color).
So your color index, mathematically, would be: Shade + (saturation * 4) + (Hue * 8) + (Colorflag * 128)
Two corner cases would be needed when you're encoding/decoding your color values using this method:
If the shade is light (3), you need to force saturation to 1, because light colors have only a single saturation level.
If Colorflag is 0, then re-purpose the first 3 bits as a 5-level scale of grey shades. Otherwise, treat them as shade + saturation as described above.
If you store this as a quoted string of 0/1's, e.g. of the form CHHHHsSS (e.g. "10110110", which would mean "color, #6 (cyan), 100% saturation, 'full' brightness" in this example), it'll help avoid screwing up sorting order or having to use bitwise operations (which Lua doesn't have anyway) to separate out your colors.
Hope that helps a little bit.