Broken primes on Windows [vcpkg]

[enginelesscc]

Hi,

I've been trying to use libtomcrypt's rsa_make_key on windows and noticed it was never succeeding.
I traced it to rand_prime which seems to fail on mp_prime_is_prime
I'm assuming this is a libtommath problem, not libtomcrypt

vcpkg baseline: 9a718185e570b2366fda05eb65f1fc826fbf1cef
libtommath: 1.3.0#1
libtomcrypt: 1.18.2#3
VS 17.13.1

Code (taken from docs "2.4 Functions that need a PRNG"):

#define USE_LTM
#define LTM_DESC

#include <tomcrypt.h>
#include <tommath.h>

#pragma comment(lib, "bcrypt.lib")

int main() {
    rsa_key key;
    int err;
    ltc_mp = ltm_desc;
    register_prng(&sprng_desc);
    if ((err = rsa_make_key(NULL, find_prng("sprng"), 1024 / 8, 65537, &key)) != CRYPT_OK) {
        printf("make_key error: %s\n", error_to_string(err));
        return -1;
    }
    return 0;
}

Reduced to:

#define USE_LTM
#define LTM_DESC

#include <tomcrypt.h>
#include <tommath.h>

#pragma comment(lib, "bcrypt.lib")

int main() {
    int err;
    ltc_mp = ltm_desc;
    register_prng(&sprng_desc);
    void* p;
    ltc_mp.init(&p);
    if ((err = rand_prime(p, 64, NULL, find_prng("sprng"))) != CRYPT_OK) {
        printf("rand_prime error: %s\n", error_to_string(err));
        return -1;
    }
    return 0;
}

[enginelesscc]

Downgrading to baseline 6717592fcf83f2273bebf48f1e44aaa8a65ea435
Which is on 1.2.1, fixes the issue.

Ref: microsoft/vcpkg@6717592

Working code:

#define USE_LTM
#define LTM_DESC

#include <tomcrypt.h>
#include <tommath.h>

extern "C" {
    int s_read_arc4random(void* p, size_t n) { return 1; }
    int s_read_getrandom(void* p, size_t n) { return 1; }
    int s_read_urandom(void* p, size_t n) { return 1; }
    int s_read_ltm_rng(void* p, size_t n) { return 1; }
}

int main() {
    rsa_key key;
    int err;
    ltc_mp = ltm_desc;
    register_prng(&sprng_desc);
    if ((err = rsa_make_key(NULL, find_prng("sprng"), 1024 / 8, 65537, &key)) != CRYPT_OK) {
        printf("make_key error: %s\n", error_to_string(err));
        return -1;
    }
    return 0;
}

[sjaeckel]

That's super strange, because the major differences between libtommath 1.2.1 and 1.3.0 are only

1. the deprecation of some symbols
2. CMake support

After checking this vcpkg thing, I see that they introduced their own patches for s_read_wincsp(): microsoft/vcpkg@fa11d34

Maybe that's causing the problems?

[Edit] those patches were also upstreamed via #584 but changed in our review and not updated in vcpkg yet.
[Edit2] and I'm absolutely not sure whether they could have anything to do with the failure...

[enginelesscc]

I've ended up taking latest vcpkg' baseline libtomcrypt, but older libtommath using a vcpkg manifest override.
This also works.
So I'm guessing one of the later patches must be broken.

{ "name": "libtommath", "version": "1.2.1" }

[sjaeckel]

@paperchalice maybe you want to look into this?

[paperchalice]

I tested with vanilla sources in develop branch, the reproducer worked fine. The patch in vcpkg is not sync with the PR I made, need to update it. 😥

[sjaeckel]

I tested with vanilla sources in develop branch, the reproducer worked fine. The patch in vcpkg is not sync with the PR I made, need to update it. 😥

thanks for taking care of this! looking forward to your updated vcpkg version, so @enginelesscc can use the latest ltm release.

[sjaeckel]

@enginelesscc looks like vcpkg should be updated! Could you please test?

[enginelesscc]

@sjaeckel @paperchalice hi yes, its fixed :)
Thanks!