GITBOOK-418: change request with no subject merged in GitBook

Author: fedorov

data/organization-of-data/clinical.md

@@ -1,7 +1,7 @@
 # Clinical data
 
 {% hint style="info" %}
-Check out our [IDC clinical data exploration Colab notebook](https://github.com/ImagingDataCommons/IDC-Examples/blob/master/notebooks/clinical\_data\_intro.ipynb) tutorial for a brief hands-on introduction into IDC clinical data! You can also see the high-level summary of the clinical data attributes accompanying IDC data in [this dashboard](https://lookerstudio.google.com/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p\_s7mk6eybqc).
+Check out our [IDC clinical data exploration Colab  notebook](https://github.com/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/getting_started/exploring_clinical_data.ipynb) tutorial for a brief hands-on introduction into IDC clinical data! You can also see the high-level summary of the clinical data attributes accompanying IDC data in [this dashboard](https://lookerstudio.google.com/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p_s7mk6eybqc).
 {% endhint %}
 
 ### Background
@@ -12,19 +12,19 @@ Clinical data is often critical in understanding imaging data, and is essential
 
 Not only are the terms used in the clinical data accompanying individual collection not harmonized, but the format of the spreadsheets is also collection-specific. In order to search and navigate clinical data, one has to parse those collection specific tables, and there is no interface to support searching across collections.
 
-With the release v11 of IDC, we make the attempt to lower the barriers for accessing clinical data accompanying IDC imaging collections. We parse collection-specific tables, and organize the underlying data into BigQuery tables that can be accessed using standard SQL queries. You can also see the summary of clinical data available for IDC collections in [this dashboard](https://datastudio.google.com/u/0/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p\_s7mk6eybqc).
+With the release v11 of IDC, we make the attempt to lower the barriers for accessing clinical data accompanying IDC imaging collections. We parse collection-specific tables, and organize the underlying data into BigQuery tables that can be accessed using standard SQL queries. You can also see the summary of clinical data available for IDC collections in [this dashboard](https://datastudio.google.com/u/0/reporting/04cf5976-4ea0-4fee-a749-8bfd162f2e87/page/p_s7mk6eybqc).
 
 ### Clinical data BigQuery tables
 
-As of Version 11 IDC provides a public [BigQuery dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc\_clinical\_current) with clinical data associated with several of its imaging collections. The clinical data tables associated with a particular version are in the dataset `bigquery-public-data.idc_<idc_version_number>_clinical`. In addition the dataset `bigquery-public-data.idc_current_clinical` has an identically named view for each table in the BQ clinical dataset corresponding to the current IDC release.
+As of Version 11 IDC provides a public [BigQuery dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current) with clinical data associated with several of its imaging collections. The clinical data tables associated with a particular version are in the dataset `bigquery-public-data.idc_<idc_version_number>_clinical`. In addition the dataset `bigquery-public-data.idc_current_clinical` has an identically named view for each table in the BQ clinical dataset corresponding to the current IDC release.
 
-There are currently 130 tables with clinical data representing 70 different collections. Most of this data was curated from Excel and CSV files downloaded from [The Cancer Imaging Archive (TCIA) wiki](https://wiki.cancerimagingarchive.net/). For most collections, data is placed in a single table named `<collection_id>_clinical`, where `<collection_id>` is the name of the collection in a standardized format (i.e. the `idc_webapp_collection_id` column in the `dicom_all` view in the [idc\_current dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc\_clinical\&page=dataset)).
+There are currently 130 tables with clinical data representing 70 different collections. Most of this data was curated from Excel and CSV files downloaded from [The Cancer Imaging Archive (TCIA) wiki](https://wiki.cancerimagingarchive.net/). For most collections, data is placed in a single table named `<collection_id>_clinical`, where `<collection_id>` is the name of the collection in a standardized format (i.e. the `idc_webapp_collection_id` column in the `dicom_all` view in the [idc\_current dataset](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical\&page=dataset)).
 
-Collections from the ACRIN project have different types of clinical data spread across CSV files, and so this data is represented by several BigQuery tables. The clinical data for collections in the [CPTAC program](https://proteomics.cancer.gov/programs/cptac) program is not curated from TCIA but instead is copied from a [BigQuery table](https://console.cloud.google.com/bigquery?p=isb-cgc-bq\&d=cptac\&t=clinical\_gdc\_current\&page=table) in the ISB-CGC project, which in turn was sourced from the [Genomics Data Commons (GDC) api](https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). Similarly clinical data for collections in the [TCGA program](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga) is copied from the table `tcga_clinical_rel9` in the `idc_current` dataset, which was also created using the [Genomics Data Commons (GDC) api](https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). Every clinical data table contains two fields we have introduced, `dicom_patient_id` and `source_batch`. `dicom_patient_id` is identical to the `PatientID` field in the DICOM files that correspond to the given patient. The `dicom_patient_id` value is determined by inspecting the patient column in the clinical data file. In some of the collections' clinical data, the patients are separated into different 'batches' i.e. different source files, or different sheets in the same Excel file. The `source_batch` field is an integer indicating the 'batch' for the given patient. For most collections, in which all patients data is found in the same location, the `source_batch` value is zero.
+Collections from the ACRIN project have different types of clinical data spread across CSV files, and so this data is represented by several BigQuery tables. The clinical data for collections in the [CPTAC program](https://proteomics.cancer.gov/programs/cptac) program is not curated from TCIA but instead is copied from a [BigQuery table](https://console.cloud.google.com/bigquery?p=isb-cgc-bq\&d=cptac\&t=clinical_gdc_current\&page=table) in the ISB-CGC project, which in turn was sourced from the [Genomics Data Commons (GDC) api](https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). Similarly clinical data for collections in the [TCGA program](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga) is copied from the table `tcga_clinical_rel9` in the `idc_current` dataset, which was also created using the [Genomics Data Commons (GDC) api](https://gdc.cancer.gov/developers/gdc-application-programming-interface-api). Every clinical data table contains two fields we have introduced, `dicom_patient_id` and `source_batch`. `dicom_patient_id` is identical to the `PatientID` field in the DICOM files that correspond to the given patient. The `dicom_patient_id` value is determined by inspecting the patient column in the clinical data file. In some of the collections' clinical data, the patients are separated into different 'batches' i.e. different source files, or different sheets in the same Excel file. The `source_batch` field is an integer indicating the 'batch' for the given patient. For most collections, in which all patients data is found in the same location, the `source_batch` value is zero.
 
-Most of the clinical tables are legible by themselves. Tables from the ACRIN collection are an exception as the column names and some of the column values are coded. To provide for clarity and ease of use of all clinical data, we have created two metadata tables, [`table_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc\_clinical\_current\&t=table\_metadata\&page=table) and [`column_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc\_clinical\_current\&t=column\_metadata\&page=table) that provide information about the structure and provenance of all data in this dataset. `table_metadata` has table-level metadata about each clinical collection, while `column_metadata` has column-level metadata.
+Most of the clinical tables are legible by themselves. Tables from the ACRIN collection are an exception as the column names and some of the column values are coded. To provide for clarity and ease of use of all clinical data, we have created two metadata tables, [`table_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=table_metadata\&page=table) and [`column_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=column_metadata\&page=table) that provide information about the structure and provenance of all data in this dataset. `table_metadata` has table-level metadata about each clinical collection, while `column_metadata` has column-level metadata.
 
-Structure of the[`table_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc\_clinical\_current\&t=table\_metadata\&page=table) table:
+Structure of the[`table_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=table_metadata\&page=table) table:
 
 * `collection_id` (STRING, NULLABLE) - the collection\_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '\_' allowed). It is equivalent to the `idc_webapp_id` field in the `dicom_all` view in the `idc_current` dataset.
 * `table_name` (STRING,NULLABLE) - name of the table
@@ -45,7 +45,7 @@ Structure of the[`table_metadata`](https://console.cloud.google.com/bigquery?p=b
 * `source_info.table_last_modified` (STRING, NULLABLE) - CPTAC and TCGA collections only. The date and time the source BigQuery table was most recently modified, as recorded when last copied
 * `source_info.table_size` (STRING, NULLABLE) - CPTAC and TCGA collections only. The size of the source BigQuery table as recorded when last copied
 
-Structure of [`column_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc\_clinical\_current\&t=column\_metadata\&page=table) table:
+Structure of [`column_metadata`](https://console.cloud.google.com/bigquery?p=bigquery-public-data\&d=idc_clinical_current\&t=column_metadata\&page=table) table:
 
 * `collection_id` (STRING,NULLABLE) - the collection\_id of the collection in the given table. The collection id is in a format used internally by the IDC Web App (with only lowercase letters, numbers and '\_' allowed). It is equivalent to the `idc_webapp_id` field in the `dicom_all` view in the `idc_current` dataset.
 * `case_col` (BOOLEAN, NULLABLE) - true if the BigQuery column contains the patient or case id, i.e. if this column is used to determine the value of the `dicom_patient_id` column