Barebone scientifictypes (#84) (#85)

Author: tlienart

.travis.yml

@@ -15,14 +15,3 @@ notifications:
 after_success:
   # push coverage results to Codecov
   - julia -e 'using Pkg; pkg"add Coverage"; using Coverage; Codecov.submit(Codecov.process_folder())'
-
-jobs:
-  include:
-    - stage: "Documentation"
-      julia: 1.2
-      os: linux
-      script:
-        - julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd()));
-                                               Pkg.instantiate()'
-        - julia --project=docs/ docs/make.jl
-      after_success: skip

Project.toml

@@ -1,26 +1,14 @@
 name = "ScientificTypes"
 uuid = "321657f4-b219-11e9-178b-2701a2544e81"
 authors = ["Anthony D. Blaom <[email protected]>"]
-version = "0.5.1"
-
-[deps]
-CategoricalArrays = "324d7699-5711-5eae-9e2f-1d82baa6b597"
-ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
-PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
-Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
+version = "0.6.0"
 
 [compat]
-CategoricalArrays = "^0.7.3"
-ColorTypes = "^0.8"
-PrettyTables = "^0.6"
-Tables = "^0.2"
 julia = "1"
 
 [extras]
-CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
-DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
-Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Random", "Test", "CSV", "DataFrames"]
+test = ["Test", "Tables"]

README.md

@@ -1,12 +1,17 @@
 # ScientificTypes
 
-| [MacOS/Linux] | Coverage | Documentation |
-| :-----------: | :------: | :-----------: |
-| [![Build Status](https://travis-ci.org/alan-turing-institute/ScientificTypes.jl.svg?branch=master)](https://travis-ci.org/alan-turing-institute/ScientificTypes.jl) | [![codecov.io](http://codecov.io/github/alan-turing-institute/ScientificTypes.jl/coverage.svg?branch=master)](http://codecov.io/github/alan-turing-institute/ScientificTypes.jl?branch=master) | [![](https://img.shields.io/badge/docs-stable-blue.svg)](https://alan-turing-institute.github.io/ScientificTypes.jl/stable) |
+| [MacOS/Linux] | Coverage |
+| :-----------: | :------: |
+| [![Build Status](https://travis-ci.org/alan-turing-institute/ScientificTypes.jl.svg?branch=master)](https://travis-ci.org/alan-turing-institute/ScientificTypes.jl) | [![codecov.io](http://codecov.io/github/alan-turing-institute/ScientificTypes.jl/coverage.svg?branch=master)](http://codecov.io/github/alan-turing-institute/ScientificTypes.jl?branch=master) |
 
-A light-weight Julia interface for implementing conventions about the
-scientific interpretation of data, and for performing type coercions
-enforcing those conventions.
+A light-weight, dependency-free Julia interface for implementing conventions
+about the scientific interpretation of data.
+This package should only be used by developers who intend to define their own
+scientific type convention.
+The [MLJScientificTypes.jl](https://github.com/alan-turing-institute/MLJScientificTypes.jl) packages implements such a convention used in the [MLJ](https://github.com/alan-turing-institute/MLJ.jl)
+universe.
+
+## Purpose
 
 The package makes the distinction between between **machine type** and **scientific type**:
 
@@ -22,89 +27,169 @@ is used for product numbers (a factor) but also for a person's weight
 type is frequently represented by *different* machine types - both
 `Int` and `Float64` are used to represent weights, for example.
 
+### Type hierarchy
 
-## Very quick start
+The package provides a hierarchy of Julia types representing data types for use
+in method dispatch (e.g., for trait values). Instances of the types play no
+role.
 
-For more information and examples please refer to [the
-manual](https://alan-turing-institute.github.io/ScientificTypes.jl/dev).
+```
+Found
+├─ Known
+│  ├─ Finite
+│  │  ├─ Multiclass
+│  │  └─ OrderedFactor
+│  ├─ Infinite
+│  │  ├─ Continuous
+│  │  └─ Count
+│  ├─ Image
+│  │  ├─ ColorImage
+│  │  └─ GrayImage
+|  ├─ Textual
+│  └─ Table
+└─ Unknown
+```
 
-ScientificTypes.jl has three components:
+## Defining a new convention
 
-- An *interface*, for articulating a convention about the scientific
-  interpretation of data. This consists of a definition of a scientific
-  type hierarchy, and a single function `scitype` with scientific
-  types as values. Someone implementing a convention must add methods
-  to this function, while the general user just applies it to data, as
-  in `scitype(4.5)` (returning `Continuous` in the *MLJ* convention).
+If you want to implement your own convention, you can consider the [MLJScientificTypes.jl](https://github.com/alan-turing-institute/MLJScientificTypes.jl) as a blueprint.
 
-- A built-in convention, called *MLJ*, active by default.
+When defining a convention you may want to:
 
-- Convenience methods for working with scientific types, the most commonly used being
+* declare a new convention,
+* declare new traits,
+* implement custom `schema`, `show` and `info` functions,
+* add explicit `scitype` and `Scitype` definitions,
+* define a `coerce` function.
 
-    - `schema(X)`, which gives an extended schema of any Tables.jl
-       compatible table `X`, including the column scientific types
-       implied by the active convention.
-	   
-	- `coerce(X, ...)`, which coerces the machine types of `X` to
-       reflect a desired scientific type.
+We explain below how these steps may look like taking the MLJ convention as
+an example.
 
-For example,
+### Declaring a new convention
+
+In the module, define a
 
 ```julia
-using ScientificTypes, DataFrames
-X = DataFrame(
-    a = randn(5),
-    b = [-2.0, 1.0, 2.0, missing, 3.0],
-    c = [1, 2, 3, 4, 5],
-    d = [0, 1, 0, 1, 0],
-    e = ['M', 'F', missing, 'M', 'F'],
-    )
-sch = schema(X) # schema is overloaded in Scientifictypes
+struct MyConvention <: ScientificTypes.Convention end
 ```
 
-will print
+and add an init function with:
 
+```julia
+function __init__()
+  ScientificTypes.set_convention(MyConvention())
+end
 ```
-_.table = 
-┌─────────┬─────────────────────────┬────────────────────────────┐
-│ _.names │ _.types                 │ _.scitypes                 │
-├─────────┼─────────────────────────┼────────────────────────────┤
-│ a       │ Float64                 │ Continuous                 │
-│ b       │ Union{Missing, Float64} │ Union{Missing, Continuous} │
-│ c       │ Int64                   │ Count                      │
-│ d       │ Int64                   │ Count                      │
-│ e       │ Union{Missing, Char}    │ Union{Missing, Unknown}    │
-└─────────┴─────────────────────────┴────────────────────────────┘
-_.nrows = 5
+
+Subsequently you will have functions dispatching over `::MyConvention` for
+instance in the MLJ case:
+
+```julia
+ScientificTypes.scitype(::Integer, ::MLJ) = Count
 ```
 
-Here the default *MLJ* convention is being applied ((cf. [docs](https://alan-turing-institute.github.io/ScientificTypes.jl/dev/#The-MLJ-convention-1)). Detail is obtained in the obvious way; for example:
+### Declaring new traits
+
+It's useful to mark containers that meet explicit traits; by default everything
+is marked as `:other`. In the MLJ convention, we specifically consider all
+containers that meet the [`Tables.jl`](https://github.com/JuliaData/Tables.jl)
+interface. In order to declare this you have to add a key to the
+`TRAIT_FUNCTION_GIVEN_NAME` dictionary with a boolean function that verifies
+the trait. This must also be placed in your `__init__` function.
+In the case of the MLJ convention:
 
 ```julia
-julia> sch.names
-(:a, :b, :c, :d, :e)
+function __init__()
+    ScientificTypes.set_convention(MLJ())
+    ScientificTypes.TRAIT_FUNCTION_GIVEN_NAME[:table] = Tables.istable
+end
 ```
 
-Now you could want to specify that `b` is actually a `Count`, and that `d` and `e` are `Multiclass`; this is done with the `coerce` function:
+### Adding scientific types
+
+You may want to extend the type hierarchy defined above. In the case of the
+MLJ convention, we consider a *table* as a scientific type:
 
 ```julia
-Xc = coerce(X, :b=>Count, :d=>Multiclass, :e=>Multiclass)
-schema(Xc)
+struct Table{K} <: Known end
 ```
 
-which prints
+where `K` is a union over the scientific type of each of the columns.
 
+### Implementing custom `schema`, `show` and `info`
+
+If you have added new traits, you *may* want to extend the `schema` function
+for objects with that trait. Subsequently you may also want to extend  the
+`show` of such schemas  and  the `info` of such objects.
+
+The `Schema` constructor takes 4 tuples:
+- the *names* of the features
+- their *machine type*
+- their *scientific type*
+- the *number of rows*
+
+In the MLJ convention:
+
+```julia
+function ScientificTypes.schema(X, ::Val{:table}; kw...)
+    sch = Tables.schema(X)
+    # ...
+    return Schema(names, types, stypes, nrows)
+end
 ```
-_.table = 
-┌─────────┬──────────────────────────────────────────────┬───────────────────────────────┐
-│ _.names │ _.types                                      │ _.scitypes                    │
-├─────────┼──────────────────────────────────────────────┼───────────────────────────────┤
-│ a       │ Float64                                      │ Continuous                    │
-│ b       │ Union{Missing, Int64}                        │ Union{Missing, Count}         │
-│ c       │ Int64                                        │ Count                         │
-│ d       │ CategoricalValue{Int64,UInt8}                │ Multiclass{2}                 │
-│ e       │ Union{Missing, CategoricalValue{Char,UInt8}} │ Union{Missing, Multiclass{2}} │
-└─────────┴──────────────────────────────────────────────┴───────────────────────────────┘
-_.nrows = 5
 
+Extending the `show` or `info` is then straightforward.
+
+```julia
+ScientificTypes.info(X, ::Val{:table}) = schema(X)
+
+function Base.show(io::IO, ::MIME"text/plain", s::ScientificTypes.Schema)
+    # ...
+end
 ```
+
+### Adding explicit `scitype` and `Scitype` definitions
+
+The `scitype` functions indicate default mappings from *machine type* to a
+*scientific type*. For instance in the MLJ convention:
+
+```julia
+ScientificType.scitype(::Integer, ::MLJ) = Count
+```
+
+where `::MLJ` refers to the convention.
+
+The `Scitype` functions will typically match a few of your `scitype` functions
+to automatically obtain the scientific type of arrays of a type.
+For instance in the MLJ convention:
+
+```julia
+ST.Scitype(::Type{<:Integer}, ::MLJ) = Count
+```
+
+meaning that the scitype of an array such as `[1,2,3]` will directly be
+inferred as an array of `Count`.
+
+### Defining a `coerce` function
+
+It may be very useful to define a function allowing you to convert an object
+with one scitype to another scitype. In the MLJ convention, this is assumed by
+the `coerce` function.
+
+For instance consider the simplified:
+
+```julia
+function coerce(y::AbstractArray{T}, T2::Type{<:Union{Missing,Continuous}}
+                ) where T <: Union{Missing,Real}
+    return float(y)
+end
+```
+
+This maps an array of Real to an array of `AbstractFloat` (which are mapped to
+`Continuous` in the MLJ convention).
+
+Further, if you work with specific containers, you may want to define a
+`coerce` function that works on the container by applying `coerce` on each
+of the features. In the MLJ convention, we work with tabular objects and
+define a `coerce` function which applies specific coercion on each of the
+columns.