Skip to content

Conversation

gramalingam
Copy link
Collaborator

Extend onnx to script converter to suppress initializers in generated script, and replace them with randomly generated weights. This allows a compact onnxscript representation of models with large initializers, especially when we don't care about the exact weights. This is useful for generating source-based test-cases from models.

Eg., this allows us to generate test-cases such as the one here.

Copy link

codecov bot commented Dec 27, 2024

❌ 15 Tests Failed:

Tests completed Failed Passed Skipped
16836 15 16821 3780
View the top 1 failed tests by shortest run time
::onnxscript.rewriter.onnxruntime.xformers._test_models
Stack Traces | 0s run time
No failure message available
View the full list of 2 ❄️ flaky tests
tests.eager_mode_test.TestEagerModeArguments_0_reference_runtime::test_function_input_and_attribute_by_kwargs_out_of_order

Flake rate in main: 39.28% (Passed 12587 times, Failed 8144 times)

Stack Traces | 0.002s run time
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:91: in run
    res = self._run(x, y)
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:139: in _run
    res = (convert_from_ml_dtypes(res[0]),)
..../test_onnx_weekly/lib/python3.11.../onnx/reference/custom_element_types.py:50: in convert_from_ml_dtypes
    return array.view(dtype=dtype)
E   ValueError: Changing the dtype of a 0d array is only supported if the itemsize is unchanged

The above exception was the direct cause of the following exception:
tests/eager_mode_test.py:115: in test_function_input_and_attribute_by_kwargs_out_of_order
    self.assertEqual(add_with_alpha(alpha=3.0, other=2.0, this=1.0), 7.0)
onnxscript/values.py:576: in __call__
    return evaluator.default().eval_function(self, args, kwargs)
onnxscript/evaluator.py:307: in eval_function
    result = function.function(*adapted_args, **adapted_kwargs)
tests/eager_mode_test.py:59: in add_with_alpha
    other = op.Mul(other, alpha)
.../onnx_opset/_impl/opset14.py:696: in Mul
    return op(*self._prepare_inputs(schema, A, B))
onnxscript/values.py:304: in __call__
    return evaluator.default().eval(schema, args, kwargs)
onnxscript/evaluator.py:194: in eval
    outputs = self._eval(schema, inputs, attributes, closure)
onnxscript/evaluator.py:524: in _eval
    result = session.run(None, session_run_input)
..../test_onnx_weekly/lib/python3.11.../onnx/reference/reference_evaluator.py:593: in run
    outputs = node.run(*inputs, **linked_attributes)
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:114: in run
    res = OpRunBinary.run(self, x, y)
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:93: in run
    raise TypeError(
E   TypeError: Issues with types <class 'numpy.ndarray'>, <class 'numpy.ndarray'> (binary operator 'Mul').
tests.eager_mode_test.TestEagerModeArguments_0_reference_runtime::test_function_some_input_by_kwargs

Flake rate in main: 39.28% (Passed 12587 times, Failed 8144 times)

Stack Traces | 0.002s run time
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:91: in run
    res = self._run(x, y)
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:139: in _run
    res = (convert_from_ml_dtypes(res[0]),)
..../test_onnx_weekly/lib/python3.11.../onnx/reference/custom_element_types.py:50: in convert_from_ml_dtypes
    return array.view(dtype=dtype)
E   ValueError: Changing the dtype of a 0d array is only supported if the itemsize is unchanged

The above exception was the direct cause of the following exception:
tests/eager_mode_test.py:106: in test_function_some_input_by_kwargs
    self.assertEqual(add_with_alpha(1.0, other=2.0), 3.0)
onnxscript/values.py:576: in __call__
    return evaluator.default().eval_function(self, args, kwargs)
onnxscript/evaluator.py:307: in eval_function
    result = function.function(*adapted_args, **adapted_kwargs)
tests/eager_mode_test.py:59: in add_with_alpha
    other = op.Mul(other, alpha)
.../onnx_opset/_impl/opset14.py:696: in Mul
    return op(*self._prepare_inputs(schema, A, B))
onnxscript/values.py:304: in __call__
    return evaluator.default().eval(schema, args, kwargs)
onnxscript/evaluator.py:194: in eval
    outputs = self._eval(schema, inputs, attributes, closure)
onnxscript/evaluator.py:524: in _eval
    result = session.run(None, session_run_input)
..../test_onnx_weekly/lib/python3.11.../onnx/reference/reference_evaluator.py:593: in run
    outputs = node.run(*inputs, **linked_attributes)
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:114: in run
    res = OpRunBinary.run(self, x, y)
..../test_onnx_weekly/lib/python3.11.../reference/ops/_op.py:93: in run
    raise TypeError(
E   TypeError: Issues with types <class 'numpy.ndarray'>, <class 'numpy.ndarray'> (binary operator 'Mul').

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

@gramalingam gramalingam enabled auto-merge (squash) December 30, 2024 20:15
@gramalingam gramalingam merged commit 9a4c4f5 into main Dec 30, 2024
21 of 39 checks passed
@gramalingam gramalingam deleted the rama/proto-to-os branch December 30, 2024 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

2 participants