Add a validator for src => transform => dst

[bkamins]

In order to improve readability of error messages

[bkamins]

@pdeffebach - can you give me please an example when we produce something that is not clear? I have checked and for incorrect selectors we produce:

julia> select(df, 'a')
ERROR: ArgumentError: Unrecognized column selector: a

[nathanrboyer]

I think the best suggestion for this was a function to help construct pairs.

make_pair(source = [:a, :b], fun = f, dest = AsTable)`
transform(df, make_pair(...))

There should also be functionality added to make typeof(source), typeof(f(source)), and typeof(dest) work.

This might also help with #2870 if the function syntax would make it easier to programatically create a list of args... to pass to transform and the like.

[bkamins]

Could you please explain what is the advantage of:

make_pair(source = [:a, :b], fun = f, dest = AsTable)

over

[:a, :b] => f => AsTable

?

For me it seems more verbose and less readable.

There should also be functionality added to make typeof(source), typeof(f(source)), and typeof(dest) work.

I am not fully clear what you mean here? Could you please provide an example of assumed input and expected output?

Thank you!

[nathanrboyer]

It is only advantageous if the new function (or package) can provide additional error checking, type checking, and validation. Ideally, it could tell you what part of your pair is invalid or needs broadcasted.

[bkamins]

Ideally, it could tell you what part of your pair is invalid or needs broadcasted.

OK - now I understand.

The problem is that it is extremely hard (if not impossible). The problem is that Julia is very flexible what you can define to work (e.g. consider https://docs.julialang.org/en/v1/manual/methods/#Function-like-objects-1 which are objects, but are callable - you cannot detect such things by checking types unfortunately)

But I will think what can be done.

[nathanrboyer]

I'll call on @pdeffebach and @CameronBieganek to add their thoughts too.

[nathanrboyer]

I'm afraid I don't know enough to be helpful with the implementation details. Could you start by supporting more basic types and just print a warning "Validation not yet supported" for functors and such? I imagine advanced users would construct the pairs themselves anyway.

[pdeffebach]

I think another problem is that because the rules are so complicated, having a validator be separate from the place where the error is would lead to code duplication and, at worst, divergent behavior.

One thing we could also do is export a prettier version of normalize_selection, which is where a lot of the pre-processing happens. This would, in effect, be a validator. I will check if this is feasible.

[bkamins]

I will check if this is feasible.

We would just need to allow passing a data frame, or grouped data frame instead of an index. This is fully feasible.
Plus we need to warn users that normalize_selection works on a single transform (because .=> stuff happens before).
This also means that when we soon allow Not(:x) .=> fun any errors in this would not be caught as it has to be resolved before being passed to normalize_selection.

In short - I agree that it is hard. I have added one answer on SO https://stackoverflow.com/questions/69194267/failing-to-execute-column-transformation-in-dataframes-jl discussing the problem that is impossible to solve but is pretty common.

[nathanrboyer]

Did normalize_selection get more features in another PR or is there really nothing that can be done to get better error checking on Pairs?