Skip to content

Regression: cpu function is incompatible with DataFrames since Functors became opt-in #2617

@jacobadenbaum

Description

@jacobadenbaum

It used to be that nested structs that contained models and other things on the gpu and other objects like DataFrames could be easily moved to the cpu with the cpu function. This was quite useful, since a common workflow (for me at least) is to:

  1. train a model
  2. store it in the same struct as some training statistics by epoch
  3. move the whole thing to the cpu so that I can save it out in a checkpoint

However, it seems that since Functors switched to be opt in, this now errors. See below (works in a clean environment, on julia v.1.11.6).

In principle, I could opt out of this by tagging the DataFrame struct with @leaf, but if I do this in my code, it would be a form of type piracy. I think that the correct way for this to be handled is with a package extension on Flux or Functors for DataFrames to do this, either just by tagging it as a leaf directly, or by explicitly handling the DataFrame constructor properly so that it can traverse it)

julia> using DataFrames, Flux

julia> df = DataFrame(a=[1,2,3], b=[2,3,4])
3×2 DataFrame
 Row │ a      b     
     │ Int64  Int64 
─────┼──────────────
   11      2
   22      3
   33      4

julia> cpu(df)
ERROR: MethodError: no method matching DataFrame(::Vector{Vector{Int64}}, ::DataFrames.Index, ::Nothing, ::Nothing, ::Bool)
The type `DataFrame` exists, but no method is defined for this combination of argument types when trying to construct it.

Closest candidates are:
  DataFrame(::Vector{<:AbstractVector})
   @ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/dataframe/dataframe.jl:405
  DataFrame(::Union{Vector{Any}, Vector{AbstractVector}}, ::DataFrames.Index; copycols)
   @ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/dataframe/dataframe.jl:193
  DataFrame(::AbstractVector, ::Symbol; copycols)
   @ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/dataframe/dataframe.jl:367
  ...

Stacktrace:
 [1] #3
   @ ~/.julia/packages/Functors/LbNAu/src/functor.jl:22 [inlined]
 [2] (::Functors.DefaultWalk)(::Function, ::DataFrame)
   @ Functors ~/.julia/packages/Functors/LbNAu/src/walks.jl:73
 [3] ExcludeWalk
   @ ~/.julia/packages/Functors/LbNAu/src/walks.jl:126 [inlined]
 [4] (::Functors.CachedWalk{Functors.ExcludeWalk{…}, Functors.NoKeyword, Functors.WalkCache{…}})(::Function, ::DataFrame)
   @ Functors ~/.julia/packages/Functors/LbNAu/src/walks.jl:177
 [5] execute(::Functors.CachedWalk{Functors.ExcludeWalk{…}, Functors.NoKeyword, Functors.WalkCache{…}}, ::DataFrame)
   @ Functors ~/.julia/packages/Functors/LbNAu/src/walks.jl:55
 [6] fmap(::Function, ::DataFrame; exclude::Function, walk::Functors.DefaultWalk, cache::IdDict{…}, prune::Functors.NoKeyword)
   @ Functors ~/.julia/packages/Functors/LbNAu/src/maps.jl:11
 [7] (::CPUDevice)(x::DataFrame)
   @ MLDataDevices ~/.julia/packages/MLDataDevices/Si8Pa/src/public.jl:380
 [8] cpu(x::DataFrame)
   @ Flux ~/.julia/packages/Flux/uRn8o/src/functor.jl:85
 [9] top-level scope
   @ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions