-
-
Notifications
You must be signed in to change notification settings - Fork 614
Description
It used to be that nested structs that contained models and other things on the gpu and other objects like DataFrames could be easily moved to the cpu with the cpu
function. This was quite useful, since a common workflow (for me at least) is to:
- train a model
- store it in the same struct as some training statistics by epoch
- move the whole thing to the cpu so that I can save it out in a checkpoint
However, it seems that since Functors switched to be opt in, this now errors. See below (works in a clean environment, on julia v.1.11.6).
In principle, I could opt out of this by tagging the DataFrame struct with @leaf
, but if I do this in my code, it would be a form of type piracy. I think that the correct way for this to be handled is with a package extension on Flux or Functors for DataFrames to do this, either just by tagging it as a leaf directly, or by explicitly handling the DataFrame constructor properly so that it can traverse it)
julia> using DataFrames, Flux
julia> df = DataFrame(a=[1,2,3], b=[2,3,4])
3×2 DataFrame
Row │ a b
│ Int64 Int64
─────┼──────────────
1 │ 1 2
2 │ 2 3
3 │ 3 4
julia> cpu(df)
ERROR: MethodError: no method matching DataFrame(::Vector{Vector{Int64}}, ::DataFrames.Index, ::Nothing, ::Nothing, ::Bool)
The type `DataFrame` exists, but no method is defined for this combination of argument types when trying to construct it.
Closest candidates are:
DataFrame(::Vector{<:AbstractVector})
@ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/dataframe/dataframe.jl:405
DataFrame(::Union{Vector{Any}, Vector{AbstractVector}}, ::DataFrames.Index; copycols)
@ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/dataframe/dataframe.jl:193
DataFrame(::AbstractVector, ::Symbol; copycols)
@ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/dataframe/dataframe.jl:367
...
Stacktrace:
[1] #3
@ ~/.julia/packages/Functors/LbNAu/src/functor.jl:22 [inlined]
[2] (::Functors.DefaultWalk)(::Function, ::DataFrame)
@ Functors ~/.julia/packages/Functors/LbNAu/src/walks.jl:73
[3] ExcludeWalk
@ ~/.julia/packages/Functors/LbNAu/src/walks.jl:126 [inlined]
[4] (::Functors.CachedWalk{Functors.ExcludeWalk{…}, Functors.NoKeyword, Functors.WalkCache{…}})(::Function, ::DataFrame)
@ Functors ~/.julia/packages/Functors/LbNAu/src/walks.jl:177
[5] execute(::Functors.CachedWalk{Functors.ExcludeWalk{…}, Functors.NoKeyword, Functors.WalkCache{…}}, ::DataFrame)
@ Functors ~/.julia/packages/Functors/LbNAu/src/walks.jl:55
[6] fmap(::Function, ::DataFrame; exclude::Function, walk::Functors.DefaultWalk, cache::IdDict{…}, prune::Functors.NoKeyword)
@ Functors ~/.julia/packages/Functors/LbNAu/src/maps.jl:11
[7] (::CPUDevice)(x::DataFrame)
@ MLDataDevices ~/.julia/packages/MLDataDevices/Si8Pa/src/public.jl:380
[8] cpu(x::DataFrame)
@ Flux ~/.julia/packages/Flux/uRn8o/src/functor.jl:85
[9] top-level scope
@ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.