SE-ML · schmelczer · Sep 18, 2022 · Sep 18, 2022 · Sep 18, 2022 · Sep 18, 2022
diff --git a/Gemfile b/Gemfile
@@ -7,4 +7,7 @@ group :jekyll_plugins do
     gem 'jekyll-scholar'
     gem 'jemoji'
     gem 'unicode_utils'
+    gem 'jekyll-sitemap'
 end
+
+gem "webrick"
diff --git a/README.md b/README.md
@@ -1,34 +1,38 @@
 # se-ml.github.io
 
-Software Engineering for Machine Learning Homepage
+> Software Engineering for Machine Learning Homepage
 
+### Prerequisites
 
-### Prereq
 Install [jekyll](https://jekyllrb.com/docs/).
 
-### Development:
+### Development
 
-#### Install dependencies:
-```
+#### Install dependencies
+
+```sh
 $ git clone https://github.com/SE-ML/se-ml.github.io
 $ cd se-ml.github.io
 $ git fetch --all
 $ git checkout -b source origin/source
 $ bundle install
 ```
 
-#### Run dev server:
-```
+#### Run development server
+
+```sh
 $ bundle exec jekyll serve
 ```
+> The website will be available at [localhost:4000](http://localhost:4000)
 
-### Deploy to production:
-```
+### Deploy to production
+
+```sh
 $ ./bin/deploy -u se-ml
 ```
 
+#### Kudos
 
-#### Kudos:
 - this webpage uses the [al-folio](https://alshedivat.github.io/al-folio/) theme.
 - some icons are downloaded from [Freepik](https://www.freepik.com/).
 

diff --git a/_best_practices/01-parallel_feature_extraction.md b/_best_practices/01-parallel_feature_extraction.md
@@ -0,0 +1,31 @@
+---
+layout: practice
+author: András Schmelczer
+name: Parallelise Feature Extraction
+title: Parallelise Feature Extraction
+category: Data
+unique_id: data_parallel
+index: 9
+difficulty: na
+comments: True
+references: [RAY, PPP, GREATAI]
+description:
+image:
+photocredit:
+
+intent: Iterate quicker during feature engineering and be more predictable in production.
+motivation: Before processing any data, features have to be extracted. Sometimes, this step can be computationally expensive, so speeding it up, for example, by parallelising the workload, can result in less time wasted during experimentation and a more predictable production deployment.
+applicability: Parallelisation should be considered in machine learning applications where feature extraction is resource-intensive and efficiently parallelisable.
+related: [exp_parallel] #
+dependencies: #
+survey_question:
+
+labels: effectiveness
+
+---
+
+Depending on the maturity of the data engineering processes, in some cases, the researchers and engineers might end-up processing data solely using simple scripts. Correctness over performance might be reasonably prioritised in such a setup, leading to too much waiting around while the scripts run. This can slow down development and, if the same functions are deployed, even production inference.
+
+Using a database with good support for online analytical processing (OLAP) use-cases (such as <a href="https://cassandra.apache.org/_/index.html" target="_blank">Apache Cassandra</a>) and building a proper, distributed processing cluster (using, for example, <a href="https://spark.apache.org/" target="_blank">Apache Spark</a>) are reasonable first steps for mitigating the issue. Of course, when the dataset is much smaller, single-machine processing can also be appropriate. But even in that case, keeping parallelisation in mind (achievable with, for instance, <a href="https://joblib.readthedocs.io/en/latest/" target="_blank">Joblib</a>) during experimentation is vital for rapid prototyping and increasing developer experience.
+
+In some deployments where we have available unused resources, it can also make sense to parallelise the feature extraction of individual production inputs (given that it is efficiently doable). This can result in more predictable response times, which depend less on the input length.
diff --git a/_best_practices/02-archive_old_feature.md b/_best_practices/02-archive_old_feature.md
@@ -5,7 +5,7 @@ name: Actively Remove or Archive Features That are Not Used
 title: Actively Remove or Archive Features That are Not Used
 category: Training
 unique_id: exp_archive
-index: 13
+index: 14
 difficulty: "medium"
 comments: True
 references: [Rs4ML, MLTD]

diff --git a/_best_practices/02-auto_feat.md b/_best_practices/02-auto_feat.md
@@ -4,7 +4,7 @@ author: Koen van der Blom, Alex Serban, Joost Visser
 name: Automate Feature Generation and Selection
 category: Training #
 unique_id: exp_auto_feat
-index: 17
+index: 18
 difficulty: "advanced"
 references: #
 comments: True

diff --git a/_best_practices/02-auto_hyperparams.md b/_best_practices/02-auto_hyperparams.md
@@ -5,7 +5,7 @@ name: Automate Hyper-Parameter Optimisation
 title: Automate Hyper-Parameter Optimisation
 category: Training
 unique_id: exp_hyperparam
-index: 18
+index: 19
 difficulty: "medium"
 comments: True
 description:

diff --git a/_best_practices/02-auto_nas.md b/_best_practices/02-auto_nas.md
@@ -4,7 +4,7 @@ author: Koen van der Blom, Alex Serban, Joost Visser
 name: Automate Configuration of Algorithms or Model Structure
 category: Training
 unique_id: exp_auto_nas
-index: 19
+index: 20
 difficulty: "advanced"
 references: #
 comments: True

diff --git a/_best_practices/02-data_version.md b/_best_practices/02-data_version.md
@@ -5,7 +5,7 @@ name: Use Versioning for Data, Model, Configurations and Training Scripts
 title: Use Versioning for Data, Model, Configurations and Training Scripts
 category: Training
 unique_id: exp_versioning
-index: 22
+index: 23
 difficulty: "basic"
 comments: True
 references: [VML, MLPROD, MMLP, BPDL, MDLOPS, PMLPP]

diff --git a/_best_practices/02-doc_features.md b/_best_practices/02-doc_features.md
@@ -5,7 +5,7 @@ name: Assign an Owner to Each Feature and Document its Rationale
 title: Assign an Owner to Each Feature and Document its Rationale
 category: Training
 unique_id: exp_owner
-index: 12
+index: 13
 difficulty: "medium"
 comments: True
 references: [Rs4ML]

diff --git a/_best_practices/02-efficient-models.md b/_best_practices/02-efficient-models.md
@@ -3,7 +3,7 @@ layout: practice
 author: Alex Serban, Koen van der Blom, Joost Visser
 name: Use The Most Efficient Models
 category: Training
-index: 24
+index: 25
 unique_id: efficient_compression
 difficulty: na
 references: [DISTSV, ] #

diff --git a/_best_practices/02-experimentation.md b/_best_practices/02-experimentation.md
@@ -0,0 +1,31 @@
+---
+layout: practice
+author: András Schmelczer
+name: Allow Experimentation with the Inference Function
+title: Allow Experimentation with the Inference Function
+category: Training
+unique_id: training_experimentation
+index: 26
+difficulty: na
+references: [GRADIO, STREAMLIT, GREATAI]
+comments: True
+description:
+image:
+photocredit:
+
+intent: Gather feedback on the ML service before its production-ready deployment and allow efficiently iterating on it.
+motivation: Involving more colleagues, specifically ones with different (non-technical) perspectives, early in the development can help catch issues quicker.
+applicability: Allowing experimentation with the inference function early on should be a part of any mature ML lifecycle.
+related: [interpretable, concerns]
+dependencies: 
+survey_question:
+
+labels: [agility, effectiveness]
+
+---
+
+Next to proper documentation, models can also be evaluated qualitatively by interacting with them. Streamlining this process, especially for making it accessible to all stakeholders, can help in getting a shared understanding of the actual results and tightening the feedback loop.
+
+These early deployments can be simply created with solutions such as <a href="https://gradio.app" target="_blank">Gradio</a> and <a href="https://streamlit.io/gallery" target="_blank">Streamlit</a>.
+
+Additionally, the gathered insights should be easily fed back into the development. Thus, supporting rapid iterations should be part of implementing this best practice. A key component of good developer experience (DX) is _progressive evaluation_, through which development can become a highly iterative, experimental process. This is well-understood by popular data science tools, such as Jupyter Notebooks. Progressive evaluation can be implemented by making the inference functions code locally runnable and providing auto-reloading for their projects.
diff --git a/_best_practices/02-interpretable.md b/_best_practices/02-interpretable.md
@@ -3,7 +3,7 @@ layout: practice
 author: Alex Serban, Koen van der Blom, Joost Visser
 name: Employ Interpretable Models When Possible
 category: Training
-index: 14
+index: 15
 unique_id: interpretable
 difficulty: "advanced"
 references: [IMLG,TTAID,AIHLEG] #

diff --git a/_best_practices/02-measure_mdl_quality.md b/_best_practices/02-measure_mdl_quality.md
@@ -5,7 +5,7 @@ name: Continuously Measure Model Quality and Performance
 title: Continuously Measure Model Quality and Performance
 category: Training
 unique_id: exp_quality
-index: 20
+index: 21
 difficulty: "basic"
 references: [Rs4ML, TDBML]
 comments: True

diff --git a/_best_practices/02-parallel_training.md b/_best_practices/02-parallel_training.md
@@ -5,7 +5,7 @@ name: Enable Parallel Training Experiments
 title: Enable Parallel Training Experiments
 category: Training
 unique_id: exp_parallel
-index: 16
+index: 17
 difficulty: "basic"
 comments: True
 references: [CD4ML, MLPROD]

diff --git a/_best_practices/02-peer_review_mdl.md b/_best_practices/02-peer_review_mdl.md
@@ -5,7 +5,7 @@ name: Peer Review Training Scripts
 title: Peer Review Training Scripts
 category: Training
 unique_id: exp_peer
-index: 15
+index: 16
 difficulty: "medium"
 comments: True
 references: [MLTS]

diff --git a/_best_practices/02-share_exp_status.md b/_best_practices/02-share_exp_status.md
@@ -5,7 +5,7 @@ name: Share Status and Outcomes of Experiments Within the Team
 title: Share Status and Outcomes of Experiments Within the Team
 category: Training
 unique_id: exp_status
-index: 23
+index: 24
 difficulty: "basic"
 references: [BPDL, PMLPP]
 comments: True

diff --git a/_best_practices/02-subgroup_bias.md b/_best_practices/02-subgroup_bias.md
@@ -3,10 +3,10 @@ layout: practice
 author: Alex Serban, Koen van der Blom, Joost Visser
 name: Assess and Manage Subgroup Bias
 category: Training
-index: 21
+index: 22
 unique_id: subgroup_bias
 difficulty: "advanced"
-references: [PFG, MCCIM]#
+references: [PFG, MCCIM]
 comments: True
 description:
 image: #

diff --git a/_best_practices/02-test_feature_extractor.md b/_best_practices/02-test_feature_extractor.md
@@ -5,7 +5,7 @@ name: Test all Feature Extraction Code
 title: Test all Feature Extraction Code
 category: Training
 unique_id: exp_tstfeature
-index: 11
+index: 12
 difficulty: "medium"
 references: [CD4ML]
 comments: True

diff --git a/_best_practices/02-train_metric.md b/_best_practices/02-train_metric.md
@@ -5,7 +5,7 @@ name: Capture the Training Objective in a Metric that is Easy to Measure and Und
 title: Capture the Training Objective in a Metric that is Easy to Measure and Understand
 category: Training
 unique_id: exp_trainingmetric
-index: 10
+index: 11
 difficulty: "basic"
 references: [Rs4ML, OPML, DSTEAM, MLTEAM]
 comments: True

diff --git a/_best_practices/02-train_objective.md b/_best_practices/02-train_objective.md
@@ -5,7 +5,7 @@ name: Share a Clearly Defined Training Objective within the Team
 title: Share a Clearly Defined Training Objective within the Team
 category: Training
 unique_id: exp_trainingobjective
-index: 9
+index: 10
 difficulty: "basic"
 references: [Rs4ML, MMLP, MLTEAM]
 comments: True

diff --git a/_best_practices/03-cont-int.md b/_best_practices/03-cont-int.md
@@ -5,7 +5,7 @@ name: Use Continuous Integration
 title: Use Continuous Integration
 category: Coding
 unique_id: coding_build
-index: 25
+index: 28
 difficulty: "advanced"
 references: [CD4ML]
 comments: True

diff --git a/_best_practices/03-regr_test.md b/_best_practices/03-regr_test.md
@@ -5,7 +5,7 @@ name: Run Automated Regression Tests
 title: Run Automated Regression Tests
 category: Coding
 unique_id: coding_regr
-index: 24
+index: 27
 difficulty: "medium"
 references: [MLPROD]
 comments: True

diff --git a/_best_practices/03-security.md b/_best_practices/03-security.md
@@ -3,10 +3,10 @@ layout: practice
 author: Alex Serban, Koen van der Blom, Joost Visser
 name: Assure Application Security
 category: Coding
-index: 27
+index: 30
 unique_id: security
 difficulty: "advanced"
-references: [TTAID]#
+references: [TTAID]
 comments: True
 description:
 image: #

diff --git a/_best_practices/03-shemas.md b/_best_practices/03-shemas.md
@@ -0,0 +1,31 @@
+---
+layout: practice
+author: András Schmelczer
+name: Implement Standard Schemas for Common Prediction Tasks
+title: Implement Standard Schemas for Common Prediction Tasks
+category: Coding
+unique_id: coding_schemas
+index: 31
+difficulty: na
+references: [PYDANTIC, GREATAI]
+comments: True
+description:
+image:
+photocredit:
+
+intent: Increase interoperability within the organisation and facilitate code reuse.
+motivation: Developing tools, dashboards, and APIs for ML models can be made more efficient by allowing to reuse them for different models easily. However, this is only possible if these models share a similar interface.
+applicability: Machine learning teams should share a repository of standard schemas within every organisation.
+related: [data_reusable, team_communication, deployment_log]
+dependencies:
+survey_question:
+
+labels: [agility, quality]
+
+---
+
+Standard prediction response schemas allow interchanging models without friction. They also enable agile teams to reuse their existing software support (such as dashboards and tools) across many different models implementing similar tasks. For example, multiclass classification tasks always result in a prediction, probability, and, optionally, an explanation. A multi-label classification task's result is often similar to a list of multiclass classification task results, etc.
+
+Next to helping code reuse, quality can also be improved through this. For instance, having a required `explanation` field could coerce colleagues into considering more explainable approaches. 
+
+Additionally, the refined, mostly stable schemas can prevent the unnecessary overhead of deciding the appropriate interface for each new ML service. It can also help avoid mistakes coming from slight differences between inference function outputs, such as mistaking `log_probability` for `probability` or `odds`.
diff --git a/_best_practices/03-use_static_analysis.md b/_best_practices/03-use_static_analysis.md
@@ -5,7 +5,7 @@ name: Use Static Analysis to Check Code Quality
 title: Use Static Analysis to Check Code Quality
 category: Coding
 unique_id: coding_static
-index: 26
+index: 29
 difficulty: "medium"
 references: [BMS]
 comments: True

diff --git a/_best_practices/04-audit_trails.md b/_best_practices/04-audit_trails.md
@@ -3,7 +3,7 @@ layout: practice
 author: Alex Serban, Koen van der Blom, Joost Visser
 name: Provide Audit Trails
 category: Deployment
-index: 34
+index: 38
 unique_id: audit_trails
 difficulty: "advanced"
 references: [CAIAG, AIHLEG, TTAID] #

diff --git a/_best_practices/04-auto_model_packaging.md b/_best_practices/04-auto_model_packaging.md
@@ -5,7 +5,7 @@ name: Automate Model Deployment
 title: Automate Model Deployment
 category: Deployment
 unique_id: deployment_automate
-index: 28
+index: 32
 difficulty: "medium"
 references: [VML, MLArch, MLLG, OPML]
 comments: True

diff --git a/_best_practices/04-cache.md b/_best_practices/04-cache.md
@@ -0,0 +1,31 @@
+---
+layout: practice
+author: András Schmelczer
+name: Cache Production Predictions
+title: Cache Production Predictions
+category: Deployment
+unique_id: deployment_cache
+index: 39
+difficulty: na
+references: [SAASSA, GREEN, CMMD, GREATAI]
+comments: True
+description:
+image:
+photocredit:
+
+intent: Improve performance, allow more flexibility for the clients, and reduce the deployment's carbon footprint.
+motivation: Avoiding the expensive &mdash; especially in the case of Deep Learning models &mdash; recomputation of results can lead to lower latency, lower costs, and an overall more socially conscious deployment.
+applicability: Caching should be implemented in production-level ML applications where repetitive input values may occur.
+related: [deployment_log]
+dependencies:
+survey_question:
+
+labels: [quality]
+
+---
+
+Sustainability is an increasingly crucial concern of ethical AI, and avoiding wasting computing resources is a part of it. To this end, caching the results of expensive operations has to be considered in any ML deployment.
+
+Caching is a well-known technique for improving the latency of repeated responses. By using it, we can avoid recomputing results we have already calculated. However, extra care must be taken to avoid exposing private data to third parties; therefore, access control must be thoroughly considered.
+
+If the ML service's clients can rely on virtually instant responses to repeated queries, that can open up opportunities for different, new access patterns. This freedom can also result in a developer friendlier API and better developer experience.