diff --git a/index.html b/index.html index dd8432f..d475b53 100644 --- a/index.html +++ b/index.html @@ -31,35 +31,41 @@

Yiping Wang 王宜平

About me

I'm a first-year Ph.D. student in Paul G. Allen School of Computer Science & Engineering from University of Washington. I feel very fortunate to have worked under the guidance of Prof. Simon Shaolei Du since 2022 summer.

-

My main research interest lies in machine learning theory, especially the foundations of deep learning and representation learning. -I am also keen on developing practical machine learning algorithms with strong theoretical guarantees, and currently I'm working on designing data selection methods for training foundational models. -Furthermore, I always hold a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AI.

+

My main research interest broadly spread across machine learning theory and foundation models. +For the theortical part, I care about understanding the foundations of deep learning and representation learning, especially the training dynamics of the basic components like Transformer. +For the empirical part, I am keen on developing efficient algorithms with strong theoretical guarantees or insightful observations. In this aspect, currently I'm working on data selection/scheduling for multi-modal pretraining and improving model efficiency. +In addition, I always hold a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AGI, such as using LLM for mathematical proof.

Previously, I got my bachelor's degree in Computer Science & Technology from Zhejiang University in 2023, with an honor degree from Chu Kochen Honors College. I also minored in Mathematics at Zhejiang University. -During my undergraduate, I was very fortunate to work closely with Dr. Yuandong Tian, Prof. Huaxiu Yao, and Prof. Linjun Zhang on several research projects.

-

Publications and Preprint

+During my undergraduate, I was very fortunate to work closely with Dr. Yuandong Tian, Prof. Huaxiu Yao, and Prof. Linjun Zhang on several exciting research projects and learned a lot.

+

Selected Research

*: indicating equal contribution or alphabetic ordering.

    -

  1. Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning [Code]
    -*Yiping Wang, *Yifang Chen, Wendan Yan, Kevin Jamieson, Simon Du.
    -Preprint.

    +

  2. CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning [Code]
    +*Yiping Wang, *Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon Shaolei Du
    +Preprint.
    +tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can achieve a new SOTA in DataComp benchmark when combined with current best approaches.

  3. JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
    Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon Du.
    -International Conference on Learning Representations (ICLR) 2024

    +International Conference on Learning Representations (ICLR) 2024
    +tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models.

  4. Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
    Yuandong Tian, Yiping Wang, Beidi Chen, Simon Du.
    Conference on Neural Information Processing Systems (NeurIPS) 2023
    -Selected as Oral presentation at High-dimensional learning dynamics workshop at ICML 2023

    +Selected as Oral presentation at High-dimensional learning dynamics workshop at ICML 2023
    +tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias.

  5. Improved Active Multi-Task Representation Learning via Lasso
    Yiping Wang, Yifang Chen, Kevin Jamieson, Simon Du.
    -International Conference on Machine Learning (ICML) 2023

    +International Conference on Machine Learning (ICML) 2023
    +tl;dr: We improve the sample complexity of active multi-task representation learning by proposing a new LASSO-based strategy.

  6. C-Mixup: Improving Generalization in Regression [Code]
    *Huaxiu Yao, *Yiping Wang, Linjun Zhang, James Zou, Chelsea Finn.
    -Conference on Neural Information Processing Systems (NeurIPS) 2022

    +Conference on Neural Information Processing Systems (NeurIPS) 2022
    +tl;dr: We propose a simple yet effective data augmentation method to improve generalization on regression tasks.

diff --git a/index.jemdoc b/index.jemdoc index 377f2a8..60be258 100644 --- a/index.jemdoc +++ b/index.jemdoc @@ -15,32 +15,39 @@ Email: ypwang61 at cs dot washington dot edu \n I'm a first-year Ph.D. student in Paul G. Allen School of Computer Science & Engineering from University of Washington. I feel very fortunate to have worked under the guidance of [https://simonshaoleidu.com/index.html Prof. Simon Shaolei Du] since 2022 summer. -My main research interest lies in *machine learning theory*, especially the foundations of deep learning and representation learning. -I am also keen on developing practical machine learning algorithms with strong theoretical guarantees, and currently I'm working on designing *data selection methods for training foundational models*. -Furthermore, I always hold a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AI. +My main research interest broadly spread across *machine learning theory* and *foundation models*. +For the theortical part, I care about understanding the foundations of deep learning and representation learning, especially the *training dynamics of* the basic components like *Transformer*. +For the empirical part, I am keen on developing efficient algorithms with strong theoretical guarantees or insightful observations. In this aspect, currently I'm working on *data selection/scheduling for multi-modal pretraining* and improving model efficiency. +In addition, I always hold a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AGI, such as using LLM for mathematical proof. Previously, I got my bachelor's degree in [http://www.en.cs.zju.edu.cn/ Computer Science & Technology] from [https://www.zju.edu.cn/english/ Zhejiang University] in 2023, with an honor degree from [http://ckc.zju.edu.cn/ckcen/_t1906/main.psp Chu Kochen Honors College]. I also minored in [http://www.math.zju.edu.cn/mathen/main.psp Mathematics] at [https://www.zju.edu.cn/english/ Zhejiang University]. -During my undergraduate, I was very fortunate to work closely with [http://yuandong-tian.com/ Dr. Yuandong Tian], [https://www.huaxiuyao.io/ Prof. Huaxiu Yao], and [https://linjunz.github.io/ Prof. Linjun Zhang] on several research projects. +During my undergraduate, I was very fortunate to work closely with [http://yuandong-tian.com/ Dr. Yuandong Tian], [https://www.huaxiuyao.io/ Prof. Huaxiu Yao], and [https://linjunz.github.io/ Prof. Linjun Zhang] on several exciting research projects and learned a lot. -== Publications and Preprint +== Selected Research \*: indicating equal contribution or alphabetic ordering. -. [https://arxiv.org/abs/2402.02055 Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learning] [https://github.com/ypwang61/VAS \[Code\]] \n - \**Yiping Wang*, \*Yifang Chen, Wendan Yan, Kevin Jamieson, Simon Du. \n +. [https://arxiv.org/abs/2405.19547 CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning] [https://github.com/ypwang61/negCLIPLoss_NormSim \[Code\]] \n + \**Yiping Wang*, \*Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon Shaolei Du \n Preprint. \n + tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can achieve a new SOTA in [https://www.datacomp.ai/dcclip/leaderboard.html DataComp benchmark] when combined with current best approaches. . [https://arxiv.org/abs/2310.00535 JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention] \n Yuandong Tian, *Yiping Wang*, Zhenyu Zhang, Beidi Chen, Simon Du. \n International Conference on Learning Representations (ICLR) 2024 \n + tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models. . [https://arxiv.org/abs/2305.16380 Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer] \n Yuandong Tian, *Yiping Wang*, Beidi Chen, Simon Du. \n Conference on Neural Information Processing Systems (NeurIPS) 2023 \n - Selected as {{ Oral }} presentation at High-dimensional learning dynamics workshop at ICML 2023 + Selected as {{ Oral }} presentation at High-dimensional learning dynamics workshop at ICML 2023 \n + tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias. . [https://arxiv.org/abs/2306.02556 Improved Active Multi-Task Representation Learning via Lasso] \n *Yiping Wang*, Yifang Chen, Kevin Jamieson, Simon Du. \n - International Conference on Machine Learning (ICML) 2023 + International Conference on Machine Learning (ICML) 2023 \n + tl;dr: We improve the sample complexity of active multi-task representation learning by proposing a new LASSO-based strategy. . [https://arxiv.org/abs/2210.05775 C-Mixup: Improving Generalization in Regression] [https://github.com/huaxiuyao/C-Mixup \[Code\]] \n \*Huaxiu Yao, \**Yiping Wang*, Linjun Zhang, James Zou, Chelsea Finn. \n - Conference on Neural Information Processing Systems (NeurIPS) 2022 + Conference on Neural Information Processing Systems (NeurIPS) 2022 \n + tl;dr: We propose a simple yet effective data augmentation method to improve generalization on regression tasks. +