Allow `transform(df, [:x1, AsTable(:x2, :x4)] => fun)`

[pdeffebach]

df = DataFrame(rand(10, 4))
transform(df, [:x1, Between(:x2, :x4)] => fun)

where fun accepts a vector and a vector of vectors. i.e., when multi-column selectors are used inside the function input vector, default to a vector of vectors.

[bkamins]

In order to support this we would have to define a set of rules how nesting in general should be resolved. Currently nesting is not allowed anywhere except All.

In general the problem with this proposal, is that such a call in general would not be type stable, and currently the rules we have guarantee type stability of the call.

[pdeffebach]

Yes. I wonder if we should allow AsTable to be an exception in the Cols / All.

Then you could input a named tuple into the function.

[pdeffebach]

For other readers, this is a selfish issue referenced in this issue in DataFramesMeta.

[bkamins]

In general - before moving forward we need a grammar here. The current design has a precise grammar that things are allowed if and only if they are a valid index into a data frame, except for duplicates, which are allowed (and are disallowed in indexing).

[pdeffebach]

Could I write out documentation as a start? I think special-casing AsTable to be included as function arguments is feasible.

[bkamins]

Maybe please just first write down the rules you would like here as a comment so that we are clear what level of sophistication you would like to see.

In general in the long run we should refactor combine/select/transform to follow the pattern with common components:

1. common preprocessing
2. common transformation
3. common postprocessing

Then your proposal would go into the common preprocessing.
(other things that would also go in then are e.g. allowing returning tables from functions in select/transform)

The current design works but is very complex (this is a typical consequence of gradual adding of functionalities)

[pdeffebach]

Okay my specific proposal is to allow AsTable objects in the inputs to a function in select and transform.

function fun(x::Vector{Real}, y::NamedTupleofVectors)
    sum(y) ./ x
end

income_vars = [:non_farm_incom, :farm_income, :rent_income]

transform(df, [:price, AsTable(income_vars)] => fun => :adjusted_income)

Perhaps as @nalimilan mentioned in #2325 we could also include RowNumber() as something that could be inputted along with other variables into a function.

[bkamins]

Just to make the proposal more precise, if I understand what you propose correctly:

AsTable and RowNumber should be allowed as entries when a vector is passed as a first argument in src => fun => dst. Is this correct?

[pdeffebach]

Yes, exactly.