weka overall results

is robust set the super set of the non robust set?

Yes it is.
the scrip is under the name: RobustLSSuperset.R

Classification with Clustering vs Regression

4. Classification Via Clustering:
on the subset of data for which p value < 0.1:



5. Classification Via Regression:
on the subset of data for which p value < 0.1:

2. Naiive Bayes classifier

2. Naiive Bayes classifier:

on all data:

Correctly Classified Instances          20               58.8235 %
Incorrectly Classified Instances        14               41.1765 %
Kappa statistic                         -0.1226
Mean absolute error                      0.4256
Root mean squared error                  0.6424
Relative absolute error                101.0706 %
Root relative squared error            140.787  %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.1       0.208      0.167     0.1       0.125      0.329    AR
                 0.792     0.9        0.679     0.792     0.731      0.333    NR
Weighted Avg.    0.588     0.697      0.528     0.588     0.553      0.332

=== Confusion Matrix ===

  a  b   <-- classified as
  1  9 |  a = AR
  5 19 |  b = NR

on the subset of peptides with p value <0.1:


Correctly Classified Instances          29               85.2941 %
Incorrectly Classified Instances         5               14.7059 %
Kappa statistic                          0.6559
Mean absolute error                      0.1657
Root mean squared error                  0.38  
Relative absolute error                 39.3546 %
Root relative squared error             83.2714 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.8       0.125      0.727     0.8       0.762      0.808    AR
                 0.875     0.2        0.913     0.875     0.894      0.808    NR
Weighted Avg.    0.853     0.178      0.858     0.853     0.855      0.808

=== Confusion Matrix ===

  a  b   <-- classified as
  8  2 |  a = AR
  3 21 |  b = NR

on the choice of the 4 peptides:


Correctly Classified Instances          27               79.4118 %
Incorrectly Classified Instances         7               20.5882 %
Kappa statistic                          0.5182
Mean absolute error                      0.2093
Root mean squared error                  0.3515
Relative absolute error                 49.6887 %
Root relative squared error             77.0359 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.167      0.636     0.7       0.667      0.883    AR
                 0.833     0.3        0.87      0.833     0.851      0.883    NR
Weighted Avg.    0.794     0.261      0.801     0.794     0.797      0.883

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  4 20 |  b = NR

3. Random Trees

3. Random Forest:

with 10 trees:
on all data:


Correctly Classified Instances          24               70.5882 %
Incorrectly Classified Instances        10               29.4118 %
Kappa statistic                          0.1981
Mean absolute error                      0.4095
Root mean squared error                  0.4614
Relative absolute error                 97.2302 %
Root relative squared error            101.111  %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.3       0.125      0.5       0.3       0.375      0.613    AR
                 0.875     0.7        0.75      0.875     0.808      0.613    NR
Weighted Avg.    0.706     0.531      0.676     0.706     0.68       0.613

=== Confusion Matrix ===

  a  b   <-- classified as
  3  7 |  a = AR
  3 21 |  b = NR

on the subset of peptides with p value < 0.1:

Correctly Classified Instances          28               82.3529 %
Incorrectly Classified Instances         6               17.6471 %
Kappa statistic                          0.5984
Mean absolute error                      0.3   
Root mean squared error                  0.3835
Relative absolute error                 71.2381 %
Root relative squared error             84.0423 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.8       0.167      0.667     0.8       0.727      0.871    AR
                 0.833     0.2        0.909     0.833     0.87       0.871    NR
Weighted Avg.    0.824     0.19       0.838     0.824     0.828      0.871

=== Confusion Matrix ===

  a  b   <-- classified as
  8  2 |  a = AR
  4 20 |  b = NR

on the choice of 4 peptides:

Correctly Classified Instances          25               73.5294 %
Incorrectly Classified Instances         9               26.4706 %
Kappa statistic                          0.4138
Mean absolute error                      0.3176
Root mean squared error                  0.4144
Relative absolute error                 75.4286 %
Root relative squared error             90.828  %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.25       0.538     0.7       0.609      0.8      AR
                 0.75      0.3        0.857     0.75      0.8        0.8      NR
Weighted Avg.    0.735     0.285      0.763     0.735     0.744      0.8  

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  6 18 |  b = NR

with 5 trees:
on all data:


Correctly Classified Instances          23               67.6471 %
Incorrectly Classified Instances        11               32.3529 %
Kappa statistic                          0.0878
Mean absolute error                      0.3804
Root mean squared error                  0.4571
Relative absolute error                 90.3293 %
Root relative squared error            100.1854 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.2       0.125      0.4       0.2       0.267      0.677    AR
                 0.875     0.8        0.724     0.875     0.792      0.677    NR
Weighted Avg.    0.676     0.601      0.629     0.676     0.638      0.677

=== Confusion Matrix ===

  a  b   <-- classified as
  2  8 |  a = AR
  3 21 |  b = NR

on the subset of data for which the p value is <0.1:


Correctly Classified Instances          29               85.2941 %
Incorrectly Classified Instances         5               14.7059 %
Kappa statistic                          0.6559
Mean absolute error                      0.2353
Root mean squared error                  0.3662
Relative absolute error                 55.873  %
Root relative squared error             80.2593 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.8       0.125      0.727     0.8       0.762      0.858    AR
                 0.875     0.2        0.913     0.875     0.894      0.858    NR
Weighted Avg.    0.853     0.178      0.858     0.853     0.855      0.858

=== Confusion Matrix ===

  a  b   <-- classified as
  8  2 |  a = AR
  3 21 |  b = NR


on the choice of 4 peptides:

Correctly Classified Instances          24               70.5882 %
Incorrectly Classified Instances        10               29.4118 %
Kappa statistic                          0.3657
Mean absolute error                      0.3059
Root mean squared error                  0.4366
Relative absolute error                 72.6349 %
Root relative squared error             95.6755 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.292      0.5       0.7       0.583      0.819    AR
                 0.708     0.3        0.85      0.708     0.773      0.819    NR
Weighted Avg.    0.706     0.298      0.747     0.706     0.717      0.819

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  7 17 |  b = NR

* playing with the number of features or the maximum depth of the tree did not help with increasing the accuracy.

1. Logistic Regression

Running Weka:

in the file responsible for fitting the lmFit, I added some extra lines so that a file would be created under the name : ExpMat_nominal.csv
This file contains all of the peptides with the following order. the first 10 patients are the AR patients and the next 24 patients are the NR patients. I added a column to this file indicating whether a patient is AR or not.

Next, I opened this file in Weka explorer page.
From the classify menu, I chose the following functions and ran them on this data to find out the accuracy of predicting the group (AR /NR) based on all the peptides.

1. Logistic Regression:
a 10 fold cross validation on the data with all the peptides gives 61.76% accuracy.

Correctly Classified Instances          21               61.7647 %
Incorrectly Classified Instances        13               38.2353 %
Kappa statistic                         -0.1571
Mean absolute error                      0.3564
Root mean squared error                  0.5758
Relative absolute error                 84.6275 %
Root relative squared error            126.1897 %
Total Number of Instances               34


=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0         0.125      0         0         0          0.579    AR
                 0.875     1          0.677     0.875     0.764      0.577    NR
Weighted Avg.    0.618     0.743      0.478     0.618     0.539      0.578

=== Confusion Matrix ===

  a  b   <-- classified as
  0 10 |  a = AR
  3 21 |  b = NR

with the file in which the subset of data for which the p value was less than 0.1 the following results were achieved
Correctly Classified Instances          22               64.7059 %
Incorrectly Classified Instances        12               35.2941 %
Kappa statistic                          0.15  
Mean absolute error                      0.3669
Root mean squared error                  0.5962
Relative absolute error                 87.1282 %
Root relative squared error            130.6554 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.4       0.25       0.4       0.4       0.4        0.683    AR
                 0.75      0.6        0.75      0.75      0.75       0.696    NR
Weighted Avg.    0.647     0.497      0.647     0.647     0.647      0.692

=== Confusion Matrix ===

  a  b   <-- classified as
  4  6 |  a = AR
  6 18 |  b = NR


a choice of 4 peptides as given below based on my visualization technique led to:
peptides chosen:
TLAFPLTIR
HGNTDSEGIVEVK
TPDVSSALDK
VLNQELR


Correctly Classified Instances          28               82.3529 %
Incorrectly Classified Instances         6               17.6471 %
Kappa statistic                          0.575 
Mean absolute error                      0.2488
Root mean squared error                  0.3881
Relative absolute error                 59.0782 %
Root relative squared error             85.0484 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.125      0.7       0.7       0.7        0.846    AR
                 0.875     0.3        0.875     0.875     0.875      0.846    NR
Weighted Avg.    0.824     0.249      0.824     0.824     0.824      0.846

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  3 21 |  b = NR

April 05 2013

To-Do-List:
run the script with different adjusting methods.