chore: remove dependency on protoletariat and update protobuf

[nielspardon]

Protoletariat is using an outdated version of protobuf and does not seem
to be actively maintained.

By changing the way we generate the Python code from the proto files we can
remove the dependency on protoletariat and update to the latest protobuf
version.

Additionally, this adds the code formatting and lint fixing Make targets to the
codegen target and fixes some formatting issues that were introduced in a prior
PR.

[github-actions]

ACTION NEEDED

Substrait follows the Conventional Commits
specification for
release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[nielspardon]

one of the things that change by not using protoletariat is that we have absolute imports instead of relative imports

[nielspardon]

another thing that changes is that codegen no longer generates these __init__.pyi files which I guess is not an issue but let me know otherwise

[benbellick]

I can try and take a look at this later today, but just wanted to flag that there is an issue in upstream substrait being discussed that would eventually lead to removing this work.

[nielspardon]

I can try and take a look at this later today, but just wanted to flag that there is an issue in upstream substrait being discussed that would eventually lead to removing this work.

thanks for raising awareness (link substrait-io/substrait#925). this is more of a more short term solution since we are running into seg faults with substrait-python which look like they are coming from protobuf python and being able to upgrade protobuf short term would be useful.

[benbellick]

Thanks for doing this! Just a few comments but I believe they are relatively small. Just trying to understand all of the decisions being made.

Also, considering that @cpcloud is a PMC member + the author of proletariat, I thought it might make sense to @ him here in case there is some additional context he can provide.

Thanks again!

[benbellick]

Why is this necessary? Can we add a comment explaining why?

[benbellick]

I think the answer is that protoc doesn't automatically generated __init__.py files in all of the subdirectories, so we have to do it manually. Is that correct?

[nielspardon]

correct, protoc / buf don't generate those automatically and the previous ones get rm'ed

[benbellick]

Was altering the value of namespace and dest_dir necessary? If so, why?

[nielspardon]

it's necessary since protoc generates the Python imports following the directory structure. with the changed namespace it generates a directory structure with the namespace within ./buf_work_dir (./buf_work__dir/substrait/gen/proto) which then gets copied over into ./src.

[benbellick]

Seems reasonable to me!

[tokoko]

looks good. tbh, I'm not a fan of proto updates generally. python protobuf does an eager major version check and basically disqualifies any other library that still relies on protobuf 5 (which hasn't reached EOL yet). The ideal solution would be to figure out a way to support multiple major versions, but that's easier said than done.