Figure 1

[nihaoxiaoli]

Thank you for your work and code！
I was wondering if you could provide the experimental setup and code for Figure 1?
Thank you.

[Eric-mingjie]

Thanks for your interest in our work. The code for the meta learning experiments can be found in cifar10_meta_train.py.

[nihaoxiaoli]

Thanks for your interest in our work. The code for the meta learning experiments can be found in cifar10_meta_train.py.

Thank you for your reply. However, this code is not consistent with Figure 1.
(1) The article mentions "training in 4 noises and testing in 15 corruptions"
(2) without illustrated code. “Fig 1 visualizes the learnt meta-loss over model predictions as we vary a single class prediction with the rest fixed.”
Therefore, I modified the code based on my own understanding
‘’‘

net_train = copy.deepcopy(net)
model = tent.configure_model(net_train)
params, param_names = tent.collect_params(model)
inner_opt = setup_optimizer(params, lr_test=None)
model_state, optimizer_state = copy_model_and_optimizer(model, inner_opt)

meta_opt = torch.optim.Adam(learnable_loss.parameters(), lr=1e-3, weight_decay=1e-5)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(meta_opt, T_max=NUM_EPOCHS, verbose=True, eta_min=1e-6)

for severity in cfg.CORRUPTION.SEVERITY:
for corruption_type in cfg.CORRUPTION.TYPE:
x_test, y_test = load_cifar10c(cfg.CORRUPTION.NUM_EX,severity, cfg.DATA_DIR, False,[corruption_type])

    x_test, y_test = x_test.cuda(), y_test.cuda()
    y_test = y_test.type(torch.cuda.LongTensor)

    num_examples_train, num_examples_test, num_inner_iter = 10000, 10000, 1

    best_err = 1.
    for epoch in range(NUM_EPOCHS):
        learnable_loss.train()
        load_model_and_optimizer(model, inner_opt, model_state, optimizer_state)
        learnable_loss = meta_train_one_epoch(model, meta_opt, inner_opt, learnable_loss, x_test, y_test, cfg.TEST.BATCH_SIZE, num_inner_iter)
        scheduler.step()

import matplotlib.pyplot as plt
inputs_range = torch.arange(-10, 41, 2).float().to(device)
inputs = torch.full((inputs_range.size(0), 10), 0.5).to(device)
inputs[:, 0] = inputs_range

outputs = learnable_loss(inputs)
print(inputs.shape, outputs.shape)

inputs_cpu = inputs.cpu().numpy()
outputs_cpu = outputs.detach().cpu().numpy()

plt.figure(figsize=(10, 6))
plt.plot(inputs_cpu[:,0], outputs_cpu[:,0], label='Model Output')
plt.title('Model Output for Inputs from -10 to 40 with Step 2')
plt.xlabel('Input')
plt.ylabel('Output')
plt.legend()
plt.grid(True)
plt.savefig('meta-loss.png')

’‘’
However, I don't know if there was a misunderstanding somewhere, and the result was not good.

[Eric-mingjie]

Can you share what the plot looks like in your case?

[nihaoxiaoli]

Can you share what the plot looks like in your case?

ce:

squared:

Is my process correct? I first use a certain loss to train a model on CIFAR10. Then, based on ‘CIFAR10_meta_train. py’, train the ‘meta_loss_transformer’ on four noises of CIFAR10-C. Then draw a picture. The input is a matrix of n * 10, with each value fixed, and then the first column is changed to [-10,40] (for squared loss, it is [-1,2]). Obtain the output of meta_loss_transformer. But the curve drawn so far is incorrect. In addition, there is a question as to the range of squared loss should be [0,1], but why is it [-1,2]. In addition, the output range of the CE loss model I trained is [-3, 18]. Is it my misunderstanding?

[Eric-mingjie]

First, the meta loss for the classifier trained with CE loss looks correct i think? The trend seems to match that in the paper. Second, for the squared loss, have you checked the performance of this meta learnt loss? It seems to be random. I recalled that the meta learning process could be unstable but make sure that the trained meta loss is a valid TTA loss.