weka overall results

is robust set the super set of the non robust set?

Yes it is.
the scrip is under the name: RobustLSSuperset.R

Classification with Clustering vs Regression

4. Classification Via Clustering:
on the subset of data for which p value < 0.1:



5. Classification Via Regression:
on the subset of data for which p value < 0.1:

2. Naiive Bayes classifier

2. Naiive Bayes classifier:

on all data:

Correctly Classified Instances          20               58.8235 %
Incorrectly Classified Instances        14               41.1765 %
Kappa statistic                         -0.1226
Mean absolute error                      0.4256
Root mean squared error                  0.6424
Relative absolute error                101.0706 %
Root relative squared error            140.787  %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.1       0.208      0.167     0.1       0.125      0.329    AR
                 0.792     0.9        0.679     0.792     0.731      0.333    NR
Weighted Avg.    0.588     0.697      0.528     0.588     0.553      0.332

=== Confusion Matrix ===

  a  b   <-- classified as
  1  9 |  a = AR
  5 19 |  b = NR

on the subset of peptides with p value <0.1:


Correctly Classified Instances          29               85.2941 %
Incorrectly Classified Instances         5               14.7059 %
Kappa statistic                          0.6559
Mean absolute error                      0.1657
Root mean squared error                  0.38  
Relative absolute error                 39.3546 %
Root relative squared error             83.2714 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.8       0.125      0.727     0.8       0.762      0.808    AR
                 0.875     0.2        0.913     0.875     0.894      0.808    NR
Weighted Avg.    0.853     0.178      0.858     0.853     0.855      0.808

=== Confusion Matrix ===

  a  b   <-- classified as
  8  2 |  a = AR
  3 21 |  b = NR

on the choice of the 4 peptides:


Correctly Classified Instances          27               79.4118 %
Incorrectly Classified Instances         7               20.5882 %
Kappa statistic                          0.5182
Mean absolute error                      0.2093
Root mean squared error                  0.3515
Relative absolute error                 49.6887 %
Root relative squared error             77.0359 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.167      0.636     0.7       0.667      0.883    AR
                 0.833     0.3        0.87      0.833     0.851      0.883    NR
Weighted Avg.    0.794     0.261      0.801     0.794     0.797      0.883

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  4 20 |  b = NR

3. Random Trees

3. Random Forest:

with 10 trees:
on all data:


Correctly Classified Instances          24               70.5882 %
Incorrectly Classified Instances        10               29.4118 %
Kappa statistic                          0.1981
Mean absolute error                      0.4095
Root mean squared error                  0.4614
Relative absolute error                 97.2302 %
Root relative squared error            101.111  %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.3       0.125      0.5       0.3       0.375      0.613    AR
                 0.875     0.7        0.75      0.875     0.808      0.613    NR
Weighted Avg.    0.706     0.531      0.676     0.706     0.68       0.613

=== Confusion Matrix ===

  a  b   <-- classified as
  3  7 |  a = AR
  3 21 |  b = NR

on the subset of peptides with p value < 0.1:

Correctly Classified Instances          28               82.3529 %
Incorrectly Classified Instances         6               17.6471 %
Kappa statistic                          0.5984
Mean absolute error                      0.3   
Root mean squared error                  0.3835
Relative absolute error                 71.2381 %
Root relative squared error             84.0423 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.8       0.167      0.667     0.8       0.727      0.871    AR
                 0.833     0.2        0.909     0.833     0.87       0.871    NR
Weighted Avg.    0.824     0.19       0.838     0.824     0.828      0.871

=== Confusion Matrix ===

  a  b   <-- classified as
  8  2 |  a = AR
  4 20 |  b = NR

on the choice of 4 peptides:

Correctly Classified Instances          25               73.5294 %
Incorrectly Classified Instances         9               26.4706 %
Kappa statistic                          0.4138
Mean absolute error                      0.3176
Root mean squared error                  0.4144
Relative absolute error                 75.4286 %
Root relative squared error             90.828  %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.25       0.538     0.7       0.609      0.8      AR
                 0.75      0.3        0.857     0.75      0.8        0.8      NR
Weighted Avg.    0.735     0.285      0.763     0.735     0.744      0.8  

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  6 18 |  b = NR

with 5 trees:
on all data:


Correctly Classified Instances          23               67.6471 %
Incorrectly Classified Instances        11               32.3529 %
Kappa statistic                          0.0878
Mean absolute error                      0.3804
Root mean squared error                  0.4571
Relative absolute error                 90.3293 %
Root relative squared error            100.1854 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.2       0.125      0.4       0.2       0.267      0.677    AR
                 0.875     0.8        0.724     0.875     0.792      0.677    NR
Weighted Avg.    0.676     0.601      0.629     0.676     0.638      0.677

=== Confusion Matrix ===

  a  b   <-- classified as
  2  8 |  a = AR
  3 21 |  b = NR

on the subset of data for which the p value is <0.1:


Correctly Classified Instances          29               85.2941 %
Incorrectly Classified Instances         5               14.7059 %
Kappa statistic                          0.6559
Mean absolute error                      0.2353
Root mean squared error                  0.3662
Relative absolute error                 55.873  %
Root relative squared error             80.2593 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.8       0.125      0.727     0.8       0.762      0.858    AR
                 0.875     0.2        0.913     0.875     0.894      0.858    NR
Weighted Avg.    0.853     0.178      0.858     0.853     0.855      0.858

=== Confusion Matrix ===

  a  b   <-- classified as
  8  2 |  a = AR
  3 21 |  b = NR


on the choice of 4 peptides:

Correctly Classified Instances          24               70.5882 %
Incorrectly Classified Instances        10               29.4118 %
Kappa statistic                          0.3657
Mean absolute error                      0.3059
Root mean squared error                  0.4366
Relative absolute error                 72.6349 %
Root relative squared error             95.6755 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.292      0.5       0.7       0.583      0.819    AR
                 0.708     0.3        0.85      0.708     0.773      0.819    NR
Weighted Avg.    0.706     0.298      0.747     0.706     0.717      0.819

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  7 17 |  b = NR

* playing with the number of features or the maximum depth of the tree did not help with increasing the accuracy.

1. Logistic Regression

Running Weka:

in the file responsible for fitting the lmFit, I added some extra lines so that a file would be created under the name : ExpMat_nominal.csv
This file contains all of the peptides with the following order. the first 10 patients are the AR patients and the next 24 patients are the NR patients. I added a column to this file indicating whether a patient is AR or not.

Next, I opened this file in Weka explorer page.
From the classify menu, I chose the following functions and ran them on this data to find out the accuracy of predicting the group (AR /NR) based on all the peptides.

1. Logistic Regression:
a 10 fold cross validation on the data with all the peptides gives 61.76% accuracy.

Correctly Classified Instances          21               61.7647 %
Incorrectly Classified Instances        13               38.2353 %
Kappa statistic                         -0.1571
Mean absolute error                      0.3564
Root mean squared error                  0.5758
Relative absolute error                 84.6275 %
Root relative squared error            126.1897 %
Total Number of Instances               34


=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0         0.125      0         0         0          0.579    AR
                 0.875     1          0.677     0.875     0.764      0.577    NR
Weighted Avg.    0.618     0.743      0.478     0.618     0.539      0.578

=== Confusion Matrix ===

  a  b   <-- classified as
  0 10 |  a = AR
  3 21 |  b = NR

with the file in which the subset of data for which the p value was less than 0.1 the following results were achieved
Correctly Classified Instances          22               64.7059 %
Incorrectly Classified Instances        12               35.2941 %
Kappa statistic                          0.15  
Mean absolute error                      0.3669
Root mean squared error                  0.5962
Relative absolute error                 87.1282 %
Root relative squared error            130.6554 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.4       0.25       0.4       0.4       0.4        0.683    AR
                 0.75      0.6        0.75      0.75      0.75       0.696    NR
Weighted Avg.    0.647     0.497      0.647     0.647     0.647      0.692

=== Confusion Matrix ===

  a  b   <-- classified as
  4  6 |  a = AR
  6 18 |  b = NR


a choice of 4 peptides as given below based on my visualization technique led to:
peptides chosen:
TLAFPLTIR
HGNTDSEGIVEVK
TPDVSSALDK
VLNQELR


Correctly Classified Instances          28               82.3529 %
Incorrectly Classified Instances         6               17.6471 %
Kappa statistic                          0.575 
Mean absolute error                      0.2488
Root mean squared error                  0.3881
Relative absolute error                 59.0782 %
Root relative squared error             85.0484 %
Total Number of Instances               34     

=== Detailed Accuracy By Class ===

               TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
                 0.7       0.125      0.7       0.7       0.7        0.846    AR
                 0.875     0.3        0.875     0.875     0.875      0.846    NR
Weighted Avg.    0.824     0.249      0.824     0.824     0.824      0.846

=== Confusion Matrix ===

  a  b   <-- classified as
  7  3 |  a = AR
  3 21 |  b = NR

April 05 2013

To-Do-List:
run the script with different adjusting methods.

April 04 2013

To-Do-List:
1. order the Expression data with 1s first and 0s second.
2. see why it is not being sorted by the order specified
3. make sure the design is correct. Currently the 0s are put first and the 1s are put second.
4. run the fit with different adjusting methods.

Details:
1. see why it is not being sorted by the order specified
The order is not what we expect because the order of patients in the infoTable.txt is not the same as the order of patients in the ExpressionMatrix. Through line by line debugging, I found out that (1:ncol(ExpressionMatrix))[analysis.table$Group==1]
gives back an array that holds all the indices in the infoTable whose Group is 1 (all ARs). This gives back:
7  9 12 13 14 17 23 27 28 29
but since the 7th patient in the info table doesnt map to the correct 7th patient in the ExpressionMatrix table, a wrong patient will be returned as the AR group.

2. order the Expression data with 1s first and 0s second.
I fixed it by using subsets. First getting the all the data from infoTable for ARs and sub-setting the expression Matrix based on that and then doing the same thing for NRs and then "cbind"ing them.
This time, I got much better results.

3. make sure the design is correct. Currently the 0s are put first and the 1s are put second.
I ran the code for two different designs.
a. 24 zeros first and 10 ones after.
This gave me 18 good peptides.
b. 10 zeros first and 24 ones after.
This gave me 23 good peptides.

The adjusted P values gotten from the bonferroni method were mostly 1.

April 03 2013

To-Do-List:
1. Run the program with robust limma.
2. Run the program with different adjusting methods

Details:
1.   Run the program with robust limma.
The program was already being run with robust limma so I tried running it with normal non-robust limma. The result was much more limited compared to the robust limma.

2. Run the program with different adjusting methods
There are several adjusting methods that can be used in Limma to adjust the p- value, given at : http://rss.acs.unt.edu/Rdoc/library/stats/html/p.adjust.html
The methods consist of:
1. holm
2. hochberg
3. hommel
4. bonferroni
5. BH
6. BY
7. fdr
8. none

With the adjusting method set to bonferroni, I noticed that all the adjusted p-values are now 1 which sounded very unusual to me and so I started debugging the code line by line to see where things are not going in the right way.
while running the program with bonferroni method I realized that the expression Matrix didnt match the design specified. In our design vector we have 10 zeros at the top and 24 1s at the bottom, but we marked our ARs by 1 and NRs by 0 and the number of ARs was 10 and NRs 24.
In addition, although we are sorting the expression matrix in a way that the ARs are first and the NRs re second, this doesnt actually happen.
So my next task would be to figure out why the expression matrix is not sorted.