small fixes

README.md

@@ -1,6 +1,6 @@
 # Empirical Confidence Estimates for Classification
 
-This repo contains code for the paper [Empirical confidence estimates for classification by deep neural networks](https://arxiv.org/abs/1903.09215). The idea of the paper is to use auxillary variables (model entropy, gradient size, dropout variance etc) to improve the confidence of the model's predictions. 
+This repo contains code for the paper [Empirical confidence estimates for classification by deep neural networks](https://arxiv.org/abs/1903.09215). The idea of the paper is to use auxillary variables (model entropy, gradient size, dropout variance etc) to improve the confidence of the model's predictions.
 
 For example, model entropy (entropy of the model's softmaxed logits) is highly correlated with the probability that the model's classification is correct. Thus during model inference, although the correct labels are unknown, it may be possible to use model entropy as an extra piece of information to improve the estimate of the probability that the model is correct.
 
@@ -10,7 +10,7 @@ We measure the value of an auxillary variable using the *odds ratio*, which is a
 
 First, we generate a DataFrame of per-image variables for a pretrained model, using the script `eval_model.py`
 ```
-python eval_model.py /path/to/data/dir/ --model resnet152 --dataset imagenet
+python eval_model.py --data-dir /path/to/data/dir/ --model resnet152 --dataset imagenet
 ```
 This will save a pickled pandas DataFrame to `logs/imagenet/resnet152/eval.pkl`
 
@@ -24,7 +24,7 @@ X: rank, Y: model_entropy
 E[Bayes ratio, top1] = 13.619
 E[Bayes ratio, top5] = 8.184
 ```
-Here `rank` is the rank of the correct label in the sorted list of the model's softmax probabilities. This script outputs the Bayes ratio for model entropy for both Top1 and  Top5. This result says knowing the model entropy is worth about 8 times more valuable than knowing the model's average accuracy alone. 
+Here `rank` is the rank of the correct label in the sorted list of the model's softmax probabilities. This script outputs the Bayes ratio for model entropy for both Top1 and  Top5. This result says knowing the model entropy is worth about 8 times more valuable than knowing the model's average accuracy alone.
 
 To help visualize this a bit better, let's bin our images into 100 equal bins based on model entropy. In each bin, we will count the number of Top1 (green), Top5 (blue), and incorrect (orange) images. We then plot this histogram with `frequency.py`:
 ```

eval_model.py

@@ -14,8 +14,8 @@
 
 parser = argparse.ArgumentParser('Gathers statistics of an ImageNet model and writes these stats into a DataFrame.')
 
-parser.add_argument('data-dir', type=str,
-        metavar='DIR', 
+parser.add_argument('--data-dir', type=str,
+        metavar='DIR',
         help='Directory where ImageNet data is saved')
 parser.add_argument('--model', type=str, default='resnet152',
         choices=['resnet152'], help='Model')
@@ -51,7 +51,7 @@ def main():
             traindir = os.path.join(args.data_dir, 'train')
             valdir = os.path.join(args.data_dir, 'val')
 
-            
+
 
             loader = torch.utils.data.DataLoader(
                                 datasets.ImageFolder(valdir, transforms.Compose([
@@ -77,7 +77,7 @@ def main():
                 x = torch.from_numpy(x)
 
             test = torch.utils.data.TensorDataset(x,y)
-            loader = torch.utils.data.DataLoader(test, 
+            loader = torch.utils.data.DataLoader(test,
                     batch_size=args.batch_size,
                     num_workers=4,
                     shuffle=False,
@@ -123,7 +123,7 @@ def main():
     m.eval()
     for p in m.parameters():
         p.requires_grad_(False)
-        
+
     if torch.cuda.device_count()>1:
         m = nn.DataParallel(m)
 
@@ -144,7 +144,7 @@ def forward(self, l, x):
 
             dx, = grad(l, x, create_graph=self.create_graph, retain_graph=self.retain_graph)
             dx = dx.view(bsz, -1)
-            
+
             if self.norm in [2,'2']:
                 n = dx.norm(p=2,dim=-1)
             elif self.norm in [1,'1']:
@@ -204,7 +204,7 @@ def forward(self, l, x):
         log = pmax.log()
 
         p5 = p.topk(5,dim=-1)[0]
-        sump5 = p.sum(dim=-1)
+        sump5 = p5.sum(dim=-1)
 
         pnorm = p.norm(dim=-1)
         loss = criterion(yhat, y)