Re: Key management with tests

On Fri, Jan  8, 2021 at 03:33:44PM -0500, Stephen Frost wrote:
> > No, I don't think so.  Stephen imported the entire NIST test suite.  It
> > was so comperhensive, it detected several OpenSSL bugs for zero-length
> > strings, which I already reported, but we would never be encrypting
> > zero-length strings, so there wasn't a lot of value to it.
> 
> I ran the entire test suite locally to ensure everything worked, but I
> didn't actually include all of it in the PR which you merged- I had
> already reduced it quite a bit by removing all 'additional
> authenticated data' test cases (which the tests will automatically skip
> and which we haven't implemented support for in the common library
> wrappers) and by removing the 192-bit cases.  This reduced the overall
> test set by about 2/3rd's or so, as I recall.

Wow, so that was reduced!

> > Anyway, I think we need to figure out how to trim.  The first part would
> > be to figure out whether we need 128 _and_ 256-bit tests, and then see
> > what items are really useful.  Stephen, do you have any ideas on that?
> > We currently have 10296 tests, and I think we could get away with 100.
> 
> Yeah, it's probably still too much, but I don't have any particularly
> justifiable suggestions as to exactly what we should remove or what we
> should keep.
> 
> Perhaps it'd make sense to try and cover the cases that are more likely
> to be issues between our wrapper functions and OpenSSL, and not stress
> too much about constantly testing cases that should really be up to
> OpenSSL.  As such, I'd propose:
> 
> - Add back in some 192-bit tests, so we cover all three bit lengths.
> - Add back in some additional authenticated test cases, just to make
>   sure that, until/unless we implement support, the test code properly
>   skips over those.
> - Keep tests for various length plaintext/ciphertext (including 0-byte
>   cases, so we make sure those work, since they really should).
> - Keep at least one test for each length of tag that's included in the
>   test suite.

Makes sense.  I did a simplistic trim-down to 90 tests but it still was
40% of the patch;  attached.  The hex strings are very long.

> I'm not sure how many tests we'd end up with from that, but my swag /
> gut feeling is that it'd probably be on the order of 100ish and a small
> enough set that it won't dwarf the rest of the patch.
> 
> Would be nice if we had a way for some buildfarm animal or something to
> pull in the entire suite and test it, imv..  If anyone wants to
> volunteer, I'd be happy to explain how to make that happen (it's not
> hard though- download/unzip the files, drop them in the directory,
> update the test script to add all the files into the array).

Yes, do we have a place to store more comprehensive tests outside of our
git tree?   Has this been done before?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee