{"id":419,"date":"2024-02-29T14:29:00","date_gmt":"2024-02-29T19:29:00","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/chapter\/cross-validation\/"},"modified":"2024-02-29T14:31:36","modified_gmt":"2024-02-29T19:31:36","slug":"cross-validation","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/chapter\/cross-validation\/","title":{"raw":"Cross Validation","rendered":"Cross Validation"},"content":{"raw":"
\r\n
\r\n
\r\n
\r\n

Cross Validation: Training more efficiently.\u00b6<\/a><\/h1>\r\nWe will be using a technique called cross-validation. Basically:\r\n
    \r\n \t
  1. Shuffle the training set and chop the training set into [latex]k[\/latex] parts.<\/li>\r\n \t
  2. Train on k-1 parts, and test on the last one - we call that the validation set. That's the score we are interested in.<\/li>\r\n \t
  3. Repeat - set aside a different part and repeat.<\/li>\r\n \t
  4. We will end up with [latex]k[\/latex] different models. We'll average those to find the best one.<\/li>\r\n \t
  5. Repeat the whole process on a new model.<\/li>\r\n<\/ol>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[1]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ## Load up our libraries:<\/span>\r\nfrom<\/span> sklearn.model_selection<\/span> import<\/span> cross_val_score<\/span>\r\nfrom<\/span> sklearn<\/span> import<\/span> tree<\/span>\r\nfrom<\/span> sklearn<\/span> import<\/span> neighbors<\/span>\r\nfrom<\/span> sklearn.metrics<\/span> import<\/span> accuracy_score<\/span>\r\nfrom<\/span> sklearn.model_selection<\/span> import<\/span> train_test_split<\/span>\r\nfrom<\/span> sklearn<\/span> import<\/span> datasets<\/span>\r\n\r\n## and our data:<\/span>\r\ndigits<\/span> =<\/span> datasets<\/span>.<\/span>load_digits<\/span>()<\/span>\r\n\r\nX<\/span> =<\/span> digits<\/span>[<\/span>'data'<\/span>]<\/span>   \r\nY<\/span> =<\/span> digits<\/span>[<\/span>'target'<\/span>]<\/span>\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[2]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ##and split it right away:<\/span>\r\n## Fixing the random seed for demonstration purposes:<\/span>\r\n\r\nX_train<\/span>,<\/span> X_test<\/span>,<\/span> Y_train<\/span>,<\/span> Y_test<\/span> =<\/span> train_test_split<\/span>(<\/span>X<\/span>,<\/span>Y<\/span>,<\/span> test_size<\/span>=<\/span>0.2<\/span>,<\/span>random_state<\/span> =<\/span> 2023<\/span>)<\/span>\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[3]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ## learn about cross-validation<\/span>\r\n?<\/span>cross_val_score\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[8]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ## and start right away:<\/span>\r\n\r\n##Here we have a tree model of depth 3:<\/span>\r\ntree_model<\/span>=<\/span>tree<\/span>.<\/span>DecisionTreeClassifier<\/span>(<\/span>max_depth<\/span> =<\/span> 6<\/span>,<\/span> random_state<\/span>=<\/span>2023<\/span>)<\/span>\r\n\r\n## Train on 4\/5 of the data, testing on the 5th.<\/span>\r\n## Do this 5 times,<\/span>\r\nCV_score<\/span> =<\/span>cross_val_score<\/span>(<\/span>tree_model<\/span>,<\/span> X_train<\/span>,<\/span> Y_train<\/span>,<\/span> cv<\/span> =<\/span>3<\/span>)<\/span>\r\n \r\n\r\nprint<\/span>(<\/span>\"We have 5 scores: \"<\/span>,<\/span> CV_score<\/span>)<\/span>\r\n\r\n## CV_Score is an array, so we can use things like .mean() to find the average<\/span>\r\nprint<\/span>(<\/span>\"Averaging them gives us an accuracy score of:\"<\/span>,<\/span> CV_score<\/span>.<\/span>mean<\/span>())<\/span>\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    <\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    We have 5 scores:  [0.80793319 0.76409186 0.78914405]\r\nAveraging them gives us an accuracy score of: 0.7870563674321502\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[5]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ## when we actually make the model we get:<\/span>\r\ntree_1<\/span> =<\/span> tree<\/span>.<\/span>DecisionTreeClassifier<\/span>(<\/span>max_depth<\/span> =<\/span> 6<\/span>,<\/span> random_state<\/span>=<\/span>2023<\/span>)<\/span> \r\ntree_1<\/span> =<\/span> tree_1<\/span>.<\/span>fit<\/span>(<\/span>X_train<\/span>,<\/span> Y_train<\/span>)<\/span>\r\n\r\nY_pred<\/span> =<\/span> tree_1<\/span>.<\/span>predict<\/span>(<\/span>X_test<\/span>)<\/span>\r\nY_pred_train<\/span> =<\/span> tree_1<\/span>.<\/span>predict<\/span>(<\/span>X_train<\/span>)<\/span>\r\n\r\nprint<\/span> (<\/span>\"Training Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_train<\/span>,<\/span>Y_pred_train<\/span>))<\/span>\r\nprint<\/span> (<\/span>\"Testing Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_test<\/span>,<\/span>Y_pred<\/span>))<\/span>\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    <\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    Training Accuracy is  0.8691718858733473\r\nTesting Accuracy is  0.7944444444444444\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    <\/div>\r\n
    \r\n\r\nNote that while there is still a difference between are training score and test score, the testing accuracy is similar to the average of the CV score above\r\n\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[6]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ## Let's try on a better model:<\/span>\r\n## Remember cupcake from earlier?<\/span>\r\n\r\ncupcake<\/span> =<\/span>neighbors<\/span>.<\/span>KNeighborsClassifier<\/span>(<\/span>n_neighbors<\/span>=<\/span>15<\/span>,<\/span> p<\/span> =<\/span>2<\/span> )<\/span>\r\nCV_score<\/span> =<\/span>cross_val_score<\/span>(<\/span>cupcake<\/span>,<\/span> X_train<\/span>,<\/span> Y_train<\/span>,<\/span> cv<\/span> =<\/span>5<\/span>)<\/span>\r\n \r\nprint<\/span>(<\/span>\"We have 5 scores: \"<\/span>,<\/span> CV_score<\/span>)<\/span>\r\nprint<\/span>(<\/span>\"Averaging them gives us an accuracy score of:\"<\/span>,<\/span> CV_score<\/span>.<\/span>mean<\/span>())<\/span>\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    <\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    We have 5 scores:  [0.97222222 0.96875    0.97212544 0.97909408 0.97560976]\r\nAveraging them gives us an accuracy score of: 0.9735602981029811\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    In\u00a0[7]:<\/div>\r\n
    \r\n
    \r\n
    \r\n
    ## and the old fashioned way:<\/span>\r\n\r\ncupcake<\/span> =<\/span>neighbors<\/span>.<\/span>KNeighborsClassifier<\/span>(<\/span>n_neighbors<\/span>=<\/span>15<\/span>,<\/span> p<\/span> =<\/span>2<\/span> )<\/span>\r\ncupcake<\/span>.<\/span>fit<\/span>(<\/span>X_train<\/span>,<\/span>Y_train<\/span>)<\/span>\r\n\r\nY_pred_train<\/span> =<\/span> cupcake<\/span>.<\/span>predict<\/span>(<\/span>X_train<\/span>)<\/span> \r\nY_pred<\/span> =<\/span> cupcake<\/span>.<\/span>predict<\/span>(<\/span>X_test<\/span>)<\/span>\r\n\r\nprint<\/span> (<\/span>\"Training Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_train<\/span>,<\/span>Y_pred_train<\/span>))<\/span>\r\nprint<\/span> (<\/span>\"Testing Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_test<\/span>,<\/span>Y_pred<\/span>))<\/span>\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n
    \r\n
    <\/div>\r\n
    \r\n
    \r\n
    <\/div>\r\n
    \r\n
    Training Accuracy is  0.9805149617258176\r\nTesting Accuracy is  0.9833333333333333\r\n<\/pre>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>","rendered":"
    \n
    \n
    \n
    \n
    Cross Validation: Training more efficiently.\u00b6<\/a><\/h1>\n
    We will be using a technique called cross-validation. Basically:<\/p>\n
      \n
    1. Shuffle the training set and chop the training set into [latex]k[\/latex] parts.<\/li>\n
    2. Train on k-1 parts, and test on the last one – we call that the validation set. That’s the score we are interested in.<\/li>\n
    3. Repeat – set aside a different part and repeat.<\/li>\n
    4. We will end up with [latex]k[\/latex] different models. We’ll average those to find the best one.<\/li>\n
    5. Repeat the whole process on a new model.<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[1]:<\/div>\n
      \n
      \n
      \n
      ## Load up our libraries:<\/span>\r\nfrom<\/span> sklearn.model_selection<\/span> import<\/span> cross_val_score<\/span>\r\nfrom<\/span> sklearn<\/span> import<\/span> tree<\/span>\r\nfrom<\/span> sklearn<\/span> import<\/span> neighbors<\/span>\r\nfrom<\/span> sklearn.metrics<\/span> import<\/span> accuracy_score<\/span>\r\nfrom<\/span> sklearn.model_selection<\/span> import<\/span> train_test_split<\/span>\r\nfrom<\/span> sklearn<\/span> import<\/span> datasets<\/span>\r\n\r\n## and our data:<\/span>\r\ndigits<\/span> =<\/span> datasets<\/span>.<\/span>load_digits<\/span>()<\/span>\r\n\r\nX<\/span> =<\/span> digits<\/span>[<\/span>'data'<\/span>]<\/span>   \r\nY<\/span> =<\/span> digits<\/span>[<\/span>'target'<\/span>]<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[2]:<\/div>\n
      \n
      \n
      \n
      ##and split it right away:<\/span>\r\n## Fixing the random seed for demonstration purposes:<\/span>\r\n\r\nX_train<\/span>,<\/span> X_test<\/span>,<\/span> Y_train<\/span>,<\/span> Y_test<\/span> =<\/span> train_test_split<\/span>(<\/span>X<\/span>,<\/span>Y<\/span>,<\/span> test_size<\/span>=<\/span>0.2<\/span>,<\/span>random_state<\/span> =<\/span> 2023<\/span>)<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[3]:<\/div>\n
      \n
      \n
      \n
      ## learn about cross-validation<\/span>\r\n?<\/span>cross_val_score\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[8]:<\/div>\n
      \n
      \n
      \n
      ## and start right away:<\/span>\r\n\r\n##Here we have a tree model of depth 3:<\/span>\r\ntree_model<\/span>=<\/span>tree<\/span>.<\/span>DecisionTreeClassifier<\/span>(<\/span>max_depth<\/span> =<\/span> 6<\/span>,<\/span> random_state<\/span>=<\/span>2023<\/span>)<\/span>\r\n\r\n## Train on 4\/5 of the data, testing on the 5th.<\/span>\r\n## Do this 5 times,<\/span>\r\nCV_score<\/span> =<\/span>cross_val_score<\/span>(<\/span>tree_model<\/span>,<\/span> X_train<\/span>,<\/span> Y_train<\/span>,<\/span> cv<\/span> =<\/span>3<\/span>)<\/span>\r\n \r\n\r\nprint<\/span>(<\/span>\"We have 5 scores: \"<\/span>,<\/span> CV_score<\/span>)<\/span>\r\n\r\n## CV_Score is an array, so we can use things like .mean() to find the average<\/span>\r\nprint<\/span>(<\/span>\"Averaging them gives us an accuracy score of:\"<\/span>,<\/span> CV_score<\/span>.<\/span>mean<\/span>())<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      <\/div>\n
      \n
      \n
      <\/div>\n
      \n
      We have 5 scores:  [0.80793319 0.76409186 0.78914405]\r\nAveraging them gives us an accuracy score of: 0.7870563674321502\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[5]:<\/div>\n
      \n
      \n
      \n
      ## when we actually make the model we get:<\/span>\r\ntree_1<\/span> =<\/span> tree<\/span>.<\/span>DecisionTreeClassifier<\/span>(<\/span>max_depth<\/span> =<\/span> 6<\/span>,<\/span> random_state<\/span>=<\/span>2023<\/span>)<\/span> \r\ntree_1<\/span> =<\/span> tree_1<\/span>.<\/span>fit<\/span>(<\/span>X_train<\/span>,<\/span> Y_train<\/span>)<\/span>\r\n\r\nY_pred<\/span> =<\/span> tree_1<\/span>.<\/span>predict<\/span>(<\/span>X_test<\/span>)<\/span>\r\nY_pred_train<\/span> =<\/span> tree_1<\/span>.<\/span>predict<\/span>(<\/span>X_train<\/span>)<\/span>\r\n\r\nprint<\/span> (<\/span>\"Training Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_train<\/span>,<\/span>Y_pred_train<\/span>))<\/span>\r\nprint<\/span> (<\/span>\"Testing Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_test<\/span>,<\/span>Y_pred<\/span>))<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      <\/div>\n
      \n
      \n
      <\/div>\n
      \n
      Training Accuracy is  0.8691718858733473\r\nTesting Accuracy is  0.7944444444444444\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      <\/div>\n
      \n
      Note that while there is still a difference between are training score and test score, the testing accuracy is similar to the average of the CV score above<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[6]:<\/div>\n
      \n
      \n
      \n
      ## Let's try on a better model:<\/span>\r\n## Remember cupcake from earlier?<\/span>\r\n\r\ncupcake<\/span> =<\/span>neighbors<\/span>.<\/span>KNeighborsClassifier<\/span>(<\/span>n_neighbors<\/span>=<\/span>15<\/span>,<\/span> p<\/span> =<\/span>2<\/span> )<\/span>\r\nCV_score<\/span> =<\/span>cross_val_score<\/span>(<\/span>cupcake<\/span>,<\/span> X_train<\/span>,<\/span> Y_train<\/span>,<\/span> cv<\/span> =<\/span>5<\/span>)<\/span>\r\n \r\nprint<\/span>(<\/span>\"We have 5 scores: \"<\/span>,<\/span> CV_score<\/span>)<\/span>\r\nprint<\/span>(<\/span>\"Averaging them gives us an accuracy score of:\"<\/span>,<\/span> CV_score<\/span>.<\/span>mean<\/span>())<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      <\/div>\n
      \n
      \n
      <\/div>\n
      \n
      We have 5 scores:  [0.97222222 0.96875    0.97212544 0.97909408 0.97560976]\r\nAveraging them gives us an accuracy score of: 0.9735602981029811\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      \n
      <\/div>\n
      \n
      In\u00a0[7]:<\/div>\n
      \n
      \n
      \n
      ## and the old fashioned way:<\/span>\r\n\r\ncupcake<\/span> =<\/span>neighbors<\/span>.<\/span>KNeighborsClassifier<\/span>(<\/span>n_neighbors<\/span>=<\/span>15<\/span>,<\/span> p<\/span> =<\/span>2<\/span> )<\/span>\r\ncupcake<\/span>.<\/span>fit<\/span>(<\/span>X_train<\/span>,<\/span>Y_train<\/span>)<\/span>\r\n\r\nY_pred_train<\/span> =<\/span> cupcake<\/span>.<\/span>predict<\/span>(<\/span>X_train<\/span>)<\/span> \r\nY_pred<\/span> =<\/span> cupcake<\/span>.<\/span>predict<\/span>(<\/span>X_test<\/span>)<\/span>\r\n\r\nprint<\/span> (<\/span>\"Training Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_train<\/span>,<\/span>Y_pred_train<\/span>))<\/span>\r\nprint<\/span> (<\/span>\"Testing Accuracy is \"<\/span>,<\/span> accuracy_score<\/span>(<\/span>Y_test<\/span>,<\/span>Y_pred<\/span>))<\/span>\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      \n
      <\/div>\n
      \n
      \n
      <\/div>\n
      \n
      Training Accuracy is  0.9805149617258176\r\nTesting Accuracy is  0.9833333333333333\r\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":883,"menu_order":11,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-419","chapter","type-chapter","status-publish","hentry"],"part":64,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/chapters\/419","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/wp\/v2\/users\/883"}],"version-history":[{"count":3,"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/chapters\/419\/revisions"}],"predecessor-version":[{"id":422,"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/chapters\/419\/revisions\/422"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/parts\/64"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/chapters\/419\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/wp\/v2\/media?parent=419"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/pressbooks\/v2\/chapter-type?post=419"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/wp\/v2\/contributor?post=419"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/businessanalytics\/wp-json\/wp\/v2\/license?post=419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}