Be more careful about signed vs. unsigned char

Be more careful about signed vs. unsigned char

The buildfarm has reminded me that not all systems consider char to be
signed and we need to be explicit.  Adjust the various bits of mac8.c
for what we intend, mostly using casts to unsigned char as suggested by
Tom, and adjust the tests for valid input accordingly.  Explicitly make
the hexlookup table signed as it's useful to use -1 there to indicate an
invalid value.