Support for custom scalars in graphql black-box testing

[jberryman]

Context

EvoMaster version: 3.4.0
Running with /opt/evomaster/bin/evomaster --blackBox true --maxTime 600s --ratePerMinute 60 --problemType GRAPHQL --bbTargetUrl http://localhost:3000/graphql

Issue

Our graphql API under test uses some custom scalars (e.g. int4), and it seems when these are in input position evomaster will use random strings (understandable), but then will generate a test case when we return the expected error. For example:

    @timeout_decorator.timeout(60)
    def test_2(self):
        
        headers = {}
        headers["content-type"] = "application/json"
        body = " { " + \
            " \"query\": \"  { EmployeeByID  (EmployeeId : \\\"QErAaRNm\\\")  {Address,Email,LastName,ReportsTo,Title}       } \" " + \
            " } "
        headers['Accept'] = "application/json"
        res_0 = requests \
                .post(self.baseUrlOfSut,
                    headers=headers, data=body)
        
        assert res_0.status_code == 200
        assert "application/json" in res_0.headers["content-type"]
        assert res_0.json()["data"]["EmployeeByID"] is None
        assert len(res_0.json()["errors"]) == 1
        assert res_0.json()["errors"][0]["message"] == "error from data source: error returned from database: invalid input syntax for type integer: \"QErAaRNm\""
        assert len(res_0.json()["errors"][0]["path"]) == 1
        assert res_0.json()["errors"][0]["path"][0] == "EmployeeByID"
        assert res_0.json()["errors"][0]["extensions"]["details"] is None

where in the schema we have EmployeeId int4!

So my issues/questions are:

• is there some way to give evomaster hints about custom scalars we use?
• the test case seems erroneous afaict as I mentioned; I probably need to read into what are the heuristics in use by evomaster

[arcuri82]

Hi @jberryman ,

thanks for reporting this. Currently, we have no support for custom scalar :( although it is something we should add in the future