Skip to content

Commit ba10f98

Browse files
DallasBowdenDallasBowden
authored
Merge pull request #1039 from Teradata/telco_ETE
Telco ETE - efs files
2 parents 482ac2f + 5c0a2ac commit ba10f98

File tree

3 files changed

+775
-162
lines changed

3 files changed

+775
-162
lines changed

‎UseCases/Teradata_End_to_End_Demos/Telco_Customer_Churn/Telco_Customer_Churn_AutoML_Approach.ipynb‎

Lines changed: 135 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@
154154
"outputs": [],
155155
"source": [
156156
"%%capture\n",
157-
"execute_sql('''SET query_band='DEMO=EE_Telco_Customer_Churn_AutoML_Approach.ipynb;' UPDATE FOR SESSION; ''')"
157+
"execute_sql('''SET query_band='DEMO=Telco_Customer_Churn_AutoML_Approach.ipynb;' UPDATE FOR SESSION; ''')"
158158
]
159159
},
160160
{
@@ -700,15 +700,22 @@
700700
"<p style='font-size:16px;font-family:Arial;'>A feature store repository serves as the foundational environment for storing and managing your data features. The owner of the FeatureStore can grant/revoke read only, write only or read and write authorization to other user(s)</p>"
701701
]
702702
},
703+
{
704+
"cell_type": "markdown",
705+
"id": "cc24f86a-a5c5-47d7-86c2-62c18aed4e46",
706+
"metadata": {},
707+
"source": [
708+
"<p style='font-size:16px;font-family:Arial'> Create the Feature Store</p>"
709+
]
710+
},
703711
{
704712
"cell_type": "code",
705713
"execution_count": null,
706714
"id": "21a4a190-ec1e-4096-a971-8aaf0c88e1d2",
707715
"metadata": {},
708716
"outputs": [],
709717
"source": [
710-
"telco_fs = FeatureStore(repo='TelcoFS')\n",
711-
"telco_fs.setup(perm_size='10e8')"
718+
"fs = FeatureStore(repo=\"TelcoFS\", data_domain=\"customer_profile\")"
712719
]
713720
},
714721
{
@@ -719,24 +726,35 @@
719726
"outputs": [],
720727
"source": [
721728
"# List whether FeatureStore is setup or not.\n",
722-
"telco_fs.list_repos()"
729+
"fs.setup()"
723730
]
724731
},
725732
{
726-
"cell_type": "markdown",
727-
"id": "44bfdbc2-bc48-4656-a219-45375b66984b",
733+
"cell_type": "code",
734+
"execution_count": null,
735+
"id": "35bf969c-e438-47df-abc4-1c075182dffe",
728736
"metadata": {},
737+
"outputs": [],
729738
"source": [
730-
"<hr style=\"height:1px;border:none;\">\n",
731-
"<p style = 'font-size:18px;font-family:Arial;'><b>5.2 Create and Register Entity </b></p>"
739+
"fs = FeatureStore(repo=\"TelcoFS\", data_domain=\"customer_profile\")"
740+
]
741+
},
742+
{
743+
"cell_type": "code",
744+
"execution_count": null,
745+
"id": "83dcbdbe-4086-4723-bfa5-1f3563ead275",
746+
"metadata": {},
747+
"outputs": [],
748+
"source": [
749+
"fs.list_repos()"
732750
]
733751
},
734752
{
735753
"cell_type": "markdown",
736754
"id": "7a3f98cc-084f-4366-8e1d-24f58481172d",
737755
"metadata": {},
738756
"source": [
739-
"<p style = 'font-size:16px;font-family:Arial;'>Let us now start with feature engineering, for which we will create the required columns in the dataframe and than use those columns to register as features in the feature group of feature store created in the step above.</p>"
757+
"<p style = 'font-size:16px;font-family:Arial'>Let us now start with feature engineering, for which we will create the required columns in the dataframe and than use those columns to register as features in the feature group of feature store created in the step above.</p>"
740758
]
741759
},
742760
{
@@ -852,15 +870,32 @@
852870
"df = DataFrame('transformed_data_automl')"
853871
]
854872
},
873+
{
874+
"cell_type": "markdown",
875+
"id": "27771e3d-5c63-4349-b718-0455042ecffa",
876+
"metadata": {},
877+
"source": [
878+
"<hr style=\"height:1px;border:none;\">\n",
879+
"<b style='font-size:18px;font-family:Arial'>5.2 Creating FeatureProcess for Fraud Transactions</b>\n",
880+
"\n",
881+
"<p style='font-size:16px;font-family:Arial'>\n",
882+
"The <code>FeatureProcess</code> captures the transformation logic and metadata of features created from raw data. \n",
883+
"These transformations are version-controlled so that future updates to the same data source can reuse identical logic for consistency.\n",
884+
"</p>\n",
885+
"\n",
886+
"<p style='font-size:16px;font-family:Arial'>\n",
887+
"Here, we create a view named <b>telco_view</b>.\n",
888+
"</p>"
889+
]
890+
},
855891
{
856892
"cell_type": "code",
857893
"execution_count": null,
858894
"id": "e81de7ec-bdf5-46c0-b9c3-6e82e4d17e9a",
859895
"metadata": {},
860896
"outputs": [],
861897
"source": [
862-
"# Create entity for DataFrame 'patient_profile_df'\n",
863-
"entity=Entity(name='CustId', columns=df.CustomerID)"
898+
"telco_view = df.create_view(view_name=\"telco_view\")"
864899
]
865900
},
866901
{
@@ -870,8 +905,7 @@
870905
"metadata": {},
871906
"outputs": [],
872907
"source": [
873-
"# Register the Entity.\n",
874-
"telco_fs.apply(entity)"
908+
"fs_customer = FeatureStore(repo=\"TelcoFS\", data_domain=\"customer_profile\")"
875909
]
876910
},
877911
{
@@ -881,29 +915,14 @@
881915
"metadata": {},
882916
"outputs": [],
883917
"source": [
884-
"# Look at existing Entities after registering the Entity.\n",
885-
"telco_fs.list_entities()"
886-
]
887-
},
888-
{
889-
"cell_type": "markdown",
890-
"id": "f5672211-fdb8-40e8-b26d-e0c076232b89",
891-
"metadata": {},
892-
"source": [
893-
"<hr style=\"height:1px;border:none;\">\n",
894-
"<p style = 'font-size:18px;font-family:Arial;'><b>5.3 Create and Register FeatureGroup </b></p>\n",
895-
"<li style = 'font-size:16px;font-family:Arial;'>FeatureGroup can be created using Teradata DataFrame.</li>\n",
896-
"<li style = 'font-size:16px;font-family:Arial;'>FeatureGroup can be created using SQL Query. </li>\n",
897-
"<li style = 'font-size:16px;font-family:Arial;'>FeatureGroup can be created using objects of Feature, Entity, DataSource. </li>\n"
898-
]
899-
},
900-
{
901-
"cell_type": "markdown",
902-
"id": "eb3ad9ab-22b7-4c49-b2fa-50edd93ec6cd",
903-
"metadata": {},
904-
"source": [
905-
"<p style = 'font-size:16px;font-family:Arial;'><b>Creating a FeatureGroup from Teradata DataFrame\n",
906-
"</b></p>"
918+
"telco_fp = FeatureProcess(repo=\"TelcoFS\",\n",
919+
" data_domain=\"customer_profile\",\n",
920+
" object=telco_view, \n",
921+
" entity='CustomerID',\n",
922+
" features=df.columns,\n",
923+
" description='Ingesting features in profile DD')\n",
924+
"\n",
925+
"telco_fp.run()"
907926
]
908927
},
909928
{
@@ -913,71 +932,97 @@
913932
"metadata": {},
914933
"outputs": [],
915934
"source": [
916-
"telco_fg = FeatureGroup.from_DataFrame(\n",
917-
" name='TelcoFG', \n",
918-
" entity_columns='CustomerID', \n",
919-
" df=df\n",
920-
")"
935+
"# List the feature process.\n",
936+
"fs_customer.list_feature_processes()"
921937
]
922938
},
923939
{
924-
"cell_type": "code",
925-
"execution_count": null,
926-
"id": "57bdc2a5-b441-4ca3-937b-6604020508b0",
940+
"cell_type": "markdown",
941+
"id": "aafb502e-306c-4ee9-8010-293dcbd30a50",
927942
"metadata": {},
928-
"outputs": [],
929943
"source": [
930-
"# Let's look at Properties.\n",
931-
"telco_fg.features, telco_fg.entity, telco_fg.data_source, telco_fg.description"
944+
"<hr style=\"height:1px;border:none;\">\n",
945+
"<b style='font-size:18px;font-family:Arial'>5.3 Building a Curated Dataset</b>\n",
946+
"\n",
947+
"<p style='font-size:16px;font-family:Arial'>\n",
948+
"The <code>DatasetCatalog</code> allows us to build curated datasets by combining selected features from one or multiple feature processes. \n",
949+
"This dataset serves as the primary input for AutoML training.\n",
950+
"</p>\n",
951+
"\n",
952+
"<p style='font-size:16px;font-family:Arial'>\n",
953+
"We select relevant features from <b>telco_fp</b> and construct a unified dataset for downstream modeling.\n",
954+
"</p>"
932955
]
933956
},
934957
{
935958
"cell_type": "code",
936959
"execution_count": null,
937-
"id": "9069c2fc-6e47-4d44-b5bf-f1c3efa9ab00",
960+
"id": "57bdc2a5-b441-4ca3-937b-6604020508b0",
938961
"metadata": {},
939962
"outputs": [],
940963
"source": [
941-
"telco_fs.apply(telco_fg)"
964+
"dc_customer = fs_customer.get_dataset_catalog()\n",
965+
"\n",
966+
"selected_profile_features = {\n",
967+
" 'Gender': telco_fp.process_id,\n",
968+
" 'SeniorCitizen': telco_fp.process_id,\n",
969+
" 'Partner': telco_fp.process_id,\n",
970+
" 'Dependents': telco_fp.process_id,\n",
971+
" 'Tenure': telco_fp.process_id,\n",
972+
" 'PhoneService': telco_fp.process_id,\n",
973+
" 'MultipleLines': telco_fp.process_id,\n",
974+
" 'InternetService': telco_fp.process_id,\n",
975+
" 'OnlineSecurity': telco_fp.process_id,\n",
976+
" 'OnlineBackup': telco_fp.process_id,\n",
977+
" 'DeviceProtection': telco_fp.process_id,\n",
978+
" 'TechSupport': telco_fp.process_id,\n",
979+
" 'StreamingTV': telco_fp.process_id,\n",
980+
" 'StreamingMovies': telco_fp.process_id,\n",
981+
" 'Contract': telco_fp.process_id,\n",
982+
" 'PaperlessBilling': telco_fp.process_id,\n",
983+
" 'PaymentMethod': telco_fp.process_id,\n",
984+
" 'MonthlyCharges': telco_fp.process_id,\n",
985+
" 'TotalCharges': telco_fp.process_id,\n",
986+
" 'Churn': telco_fp.process_id\n",
987+
" }\n",
988+
"\n",
989+
"tdf = dc_customer.build_dataset(entity='CustomerID',\n",
990+
" selected_features=selected_profile_features,\n",
991+
" view_name=\"telco_churn_profile_dataset\",\n",
992+
" description=\"Curated dataset for telco customer profile\")"
942993
]
943994
},
944995
{
945996
"cell_type": "code",
946997
"execution_count": null,
947-
"id": "b8867efb-13ab-4cf7-9924-60b31f911d64",
998+
"id": "9069c2fc-6e47-4d44-b5bf-f1c3efa9ab00",
948999
"metadata": {},
9491000
"outputs": [],
9501001
"source": [
951-
"telco_fs.list_features()"
1002+
"fs_customer.list_dataset_catalogs()"
9521003
]
9531004
},
9541005
{
9551006
"cell_type": "markdown",
956-
"id": "29bf5904-1dcc-46c1-bd3d-5be5f1001ec0",
1007+
"id": "27b0d98e-b4dc-4250-bb2f-d7c14e079079",
9571008
"metadata": {},
9581009
"source": [
959-
"<hr style=\"height:1px;border:none;\">\n",
960-
"<b style = 'font-size:18px;font-family:Arial;'>5.4 Reuse features from Enterprise Feature Store with teradataml analytic functions for AutoML processing.</b>\n"
961-
]
962-
},
963-
{
964-
"cell_type": "markdown",
965-
"id": "3be7cff1-4b1b-466c-8836-5e35d61763cd",
966-
"metadata": {},
967-
"source": [
968-
"<p style = 'font-size:16px;font-family:Arial'>Since FeatureStore stores DataSource also, you can retrive Teradata DataFrame from FeatureStore. <br> `FeatureStore.get_dataset()` get's Teradata DataFrame from FeatureGroup.</p>"
1010+
"<p style='font-size:16px;font-family:Arial'>\n",
1011+
"The above <b>mind map</b> visually represents the relationship between the raw data, the feature process, and the final dataset.\n",
1012+
"It provides complete lineage for auditability and reproducibility, which is a key advantage of EFS integration.\n",
1013+
"</p>"
9691014
]
9701015
},
9711016
{
9721017
"cell_type": "code",
9731018
"execution_count": null,
974-
"id": "6a13015d-d3d1-4bc5-9d13-cf813e03dda2",
975-
"metadata": {},
1019+
"id": "b8867efb-13ab-4cf7-9924-60b31f911d64",
1020+
"metadata": {
1021+
"tags": []
1022+
},
9761023
"outputs": [],
9771024
"source": [
978-
"# Get DataSet for FeatureGroup TelcoFG. \n",
979-
"df = telco_fs.get_dataset('TelcoFG')\n",
980-
"df"
1025+
"fs_customer.mind_map()"
9811026
]
9821027
},
9831028
{
@@ -1007,7 +1052,7 @@
10071052
"outputs": [],
10081053
"source": [
10091054
"# Performing sampling to get 80% for trainning and 20% for testing\n",
1010-
"tdf_sample = df.sample(frac = [0.8, 0.2])\n",
1055+
"tdf_sample = tdf.sample(frac = [0.8, 0.2])\n",
10111056
"\n",
10121057
"# Fetching train and test data\n",
10131058
"tdf_train= tdf_sample[tdf_sample['sampleid'] == 1].drop('sampleid', axis=1)\n",
@@ -1084,7 +1129,7 @@
10841129
"\n",
10851130
"aml = AutoClassifier(\n",
10861131
" exclude = ['knn','svm'],\n",
1087-
" verbose = 2,\n",
1132+
" verbose = 1,\n",
10881133
" max_runtime_secs = 600\n",
10891134
")"
10901135
]
@@ -1212,7 +1257,9 @@
12121257
"cell_type": "code",
12131258
"execution_count": null,
12141259
"id": "d31562da-23fa-468c-97da-c8993fb70b0f",
1215-
"metadata": {},
1260+
"metadata": {
1261+
"tags": []
1262+
},
12161263
"outputs": [],
12171264
"source": [
12181265
"# Fetching prediction and metrics on test data\n",
@@ -1392,34 +1439,40 @@
13921439
"source": [
13931440
"<hr style=\"height:2px;border:none;\">\n",
13941441
"<b style = 'font-size:20px;font-family:Arial'>8. Cleanup</b></p>\n",
1395-
"<p style = 'font-size:18px;font-family:Arial'> <b>Work Tables </b></p>"
1442+
"<p style = 'font-size:18px;font-family:Arial'> <b>Work Tables and Views</b></p>"
13961443
]
13971444
},
13981445
{
13991446
"cell_type": "code",
14001447
"execution_count": null,
1401-
"id": "9ff3fb9b-4d13-4628-988d-f82463d96537",
1448+
"id": "abecc15b-269d-47d3-b727-c09cf3e4d8af",
14021449
"metadata": {},
14031450
"outputs": [],
14041451
"source": [
1405-
"tables = ['transformed_data']\n",
1406-
"\n",
1407-
"# Loop through the list of tables and execute the drop table command for each table\n",
1408-
"for table in tables:\n",
1452+
"views = ['telco_view']\n",
1453+
"# Loop through the list of views and execute the drop view command for each view\n",
1454+
"for view in views:\n",
14091455
" try:\n",
1410-
" db_drop_table(table_name=table)\n",
1456+
" db_drop_view(view_name=view)\n",
14111457
" except:\n",
14121458
" pass"
14131459
]
14141460
},
14151461
{
14161462
"cell_type": "code",
14171463
"execution_count": null,
1418-
"id": "1a6f56a5-e38b-45d7-b532-001de783705a",
1464+
"id": "9ff3fb9b-4d13-4628-988d-f82463d96537",
14191465
"metadata": {},
14201466
"outputs": [],
14211467
"source": [
1422-
"telco_fs.archive_feature_group(feature_group='TelcoFG')"
1468+
"tables = ['transformed_data_automl']\n",
1469+
"\n",
1470+
"# Loop through the list of tables and execute the drop table command for each table\n",
1471+
"for table in tables:\n",
1472+
" try:\n",
1473+
" db_drop_table(table_name=table)\n",
1474+
" except:\n",
1475+
" pass"
14231476
]
14241477
},
14251478
{
@@ -1429,7 +1482,8 @@
14291482
"metadata": {},
14301483
"outputs": [],
14311484
"source": [
1432-
"telco_fs.delete_feature_group(feature_group='TelcoFG')"
1485+
"fs = FeatureStore(repo=\"TelcoFS\")\n",
1486+
"fs.delete()"
14331487
]
14341488
},
14351489
{
@@ -1573,7 +1627,7 @@
15731627
"name": "python",
15741628
"nbconvert_exporter": "python",
15751629
"pygments_lexer": "ipython3",
1576-
"version": "3.9.10"
1630+
"version": "3.11.14"
15771631
}
15781632
},
15791633
"nbformat": 4,

0 commit comments

Comments
 (0)