diff --git a/MENU b/MENU index 22b39f4..ca673fb 100644 --- a/MENU +++ b/MENU @@ -1,3 +1,6 @@ Yiping Wang - Home[index.html] + Home[index2.html] + Publications[pub.html] + Miscellaneous[miscellaneous.html] + Fun[fun.html] CV[CV_YipingWang_phd.pdf] \ No newline at end of file diff --git a/creed.jemdoc b/creed.jemdoc new file mode 100644 index 0000000..f5fa342 --- /dev/null +++ b/creed.jemdoc @@ -0,0 +1,2 @@ +# jemdoc: menu{MENU}{creed.html}, nofooter + diff --git a/fun.html b/fun.html new file mode 100644 index 0000000..eb5c57c --- /dev/null +++ b/fun.html @@ -0,0 +1,47 @@ + + + + + + +Fun + + + + + + + +
+
Yiping Wang
+
Home
+
Publications
+
Miscellaneous
+
Fun
+
CV
+		 +
+

Fun

+
+

Hobbies

+

I like playing pingpong very much :)

+

Some of my favorite thoughts

+
    +

  • Deep learning is based on the audacious conjecture that biological neurons and artificial neurons are not that different (from Ilya).

    +
    • +

  • The Bitter Lesson

    +
    • +

  • Useful or elegant, at least one.

    +
    • +

  • 若不披上这件衣裳，众生又怎知我尘缘已断、金海尽干

    +
    • +
+

Some of my thoughts

+
    +

  • 09/2024: Artificial intelligence can be very smart, but it still needs to follow the truth. If intelligence continues to evolve in the future, seeking the truth to survive in the universe may serve as a creed for a peaceful transition between generations.

    +
    • +
+
+ + diff --git a/fun.jemdoc b/fun.jemdoc new file mode 100644 index 0000000..c388c8b --- /dev/null +++ b/fun.jemdoc @@ -0,0 +1,18 @@ +# jemdoc: menu{MENU}{fun.html}, nofooter +==Fun + +== Hobbies + +I like playing pingpong very much :) + +== Some of my favorite thoughts +- Deep learning is based on the audacious conjecture that biological neurons and artificial neurons are not that different (from [https://twitter.com/ilyasut/status/1587478598809591808 Ilya]). +- [http://www.incompleteideas.net/IncIdeas/BitterLesson.html The Bitter Lesson] +- Useful or elegant, at least one. +- 若不披上这件衣裳，众生又怎知我尘缘已断、金海尽干 + +== Some of my thoughts +- 09\/2024: Artificial intelligence can be very smart, but it still needs to follow the truth. If intelligence continues to evolve in the future, seeking the truth to survive in the universe may serve as a creed for a peaceful transition between generations. + + + diff --git a/index.jemdoc b/index.jemdoc index 60be258..4487fe9 100644 --- a/index.jemdoc +++ b/index.jemdoc @@ -50,4 +50,3 @@ During my undergraduate, I was very fortunate to work closely with [http://yuand - diff --git a/index2.html b/index2.html new file mode 100644 index 0000000..267e8c5 --- /dev/null +++ b/index2.html @@ -0,0 +1,97 @@ + + + + + + +Yiping Wang 王宜平 + + + + + + + +
+
Yiping Wang
+
Home
+
Publications
+
Miscellaneous
+
Fun
+
CV
+		 +
+

Yiping Wang 王宜平

+
+ +
+alt text 

Yiping Wang
+Ph.D student
Paul G. Allen School of Computer Science & Engineering,
+University of Washington
+Email: ypwang61 at cs dot washington dot edu
+Google Scholar
+Twitter

+
+

About me

+

I'm a second-year Ph.D. student in Paul G. Allen School of Computer Science & Engineering from University of Washington. +I feel very fortunate to have worked under the guidance of Prof. Simon Shaolei Du since 2022 summer.

+

My main research interest broadly spread across machine learning theory and foundation models. +For the theortical part, I care about understanding the foundations of deep learning and representation learning, especially the training dynamics of the basic components like Transformer. +For the empirical part, I am keen on developing efficient algorithms with strong theoretical guarantees or insightful observations. Currently, in this aspect, I'm working on data selection/scheduling for multi-modal pretraining and improving inference efficiency of LLM. I'm also working on some projects related to video generation. +In addition, I have always held a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AGI, such as using LLMs for mathematical proof and seeking scientific truth.

+

I'm grateful to all my collaborators and mentors along the way. +I'm priviledged to be working closely with Dr. Yuandong Tian since 2023 spring. +Besides, I'm also having intern at Microsoft started from June 2024, fortunate to be advised by Yelong Shen and Shuohang Wang. +During my undergraduate, I was fortunate to work closely with Prof. Huaxiu Yao and Prof. Linjun Zhang.

+

Previously, I studied Computer Science and Mathematics in Zhejiang University, got an honors degree from Chu Kochen Honors College.

+

News

+
    +

  • 09/2024: Our negCLIPLoss paper is accepted by NeurIPS 2024 as spotlight!

    +
    • +

  • 06/2024: Started my internship at Microsoft!

    +
    • +

  • 01/2024: One paper (JoMA) is accepted by ICLR 2024!

    +
    • +

  • 12/2023: Attended NeurIPS 2023 at New Orleans.

    +
    • +

  • 09/2023: One paper (Scan&Snap) is accepted by NeurIPS 2024!

    +
    • +

  • 09/2023: Become a husky in UW!

    +
    • +
+

My Favourite Papers

+

(* denotes equal contribution or alphabetic ordering.)

+

Data Selection Algorithm

+

+We studied how to efficiently select data for multimodal pretraining tasks, drawing inspiration from both empirical observations and theoretical insights.

+ +
+alt text 

CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning [Code] [Poster] [Twitter] [Previous Versions]
+Yiping Wang*, Yifang Chen*, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon S. Du
+NeurIPS 2024 (Spotlight)

+tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can obtain a new SOTA in DataComp benchmark when combined with other approaches.

+
+

+

Training Dynamics of Transformer

+

+We attempted to analyze the training dynamics of transformers in a mathematical way.

+ +
+alt text 

Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer [Poster] [Twitter]
+Yuandong Tian, Yiping Wang, Beidi Chen, Simon S. Du
+NeurIPS 2023
+ Oral presentation at High-dimensional learning dynamics workshop @ ICML 2023

+tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias.

+
+ +
+alt text 

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention [Twitter]
+Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon S. Du
+ICLR 2024

+tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models.

+
+

+
+ + diff --git a/index2.jemdoc b/index2.jemdoc new file mode 100644 index 0000000..15b0c73 --- /dev/null +++ b/index2.jemdoc @@ -0,0 +1,82 @@ +# jemdoc: menu{MENU}{index.html}, nofooter +==Yiping Wang 王宜平 + +~~~ +{}{img_left}{photos/bio.jpg}{alt text}{146}{200} +Yiping Wang\n +Ph.D student\n [https://www.cs.washington.edu/ Paul G. Allen School of Computer Science & Engineering], \n +[https://www.washington.edu/ University of Washington]\n +Email: ypwang61 at cs dot washington dot edu \n +[https://scholar.google.com/citations?user=IuMFxFUAAAAJ&hl=en&oi=ao Google Scholar ]\n +[https://twitter.com/ypwang61 Twitter] +~~~ + +== About me +I'm a second-year Ph.D. student in Paul G. Allen School of Computer Science & Engineering from University of Washington. +I feel very fortunate to have worked under the guidance of [https://simonshaoleidu.com/index.html Prof. Simon Shaolei Du] since 2022 summer. + +My main research interest broadly spread across *machine learning theory* and *foundation models*. +For the theortical part, I care about understanding the foundations of deep learning and representation learning, especially the *training dynamics of* the basic components like *Transformer*. +For the empirical part, I am keen on developing efficient algorithms with strong theoretical guarantees or insightful observations. Currently, in this aspect, I'm working on *data selection/scheduling for multi-modal pretraining* and improving inference efficiency of LLM. I'm also working on some projects related to video generation. +In addition, I have always held a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AGI, such as using LLMs for mathematical proof and seeking scientific truth. + +I'm grateful to all my collaborators and mentors along the way. +I'm priviledged to be working closely with [http://yuandong-tian.com/ Dr. Yuandong Tian] since 2023 spring. +Besides, I'm also having intern at Microsoft started from June 2024, fortunate to be advised by [https://scholar.google.com/citations?user=S6OFEFEAAAAJ Yelong Shen] and [https://sites.google.com/site/shuohangsite/ Shuohang Wang]. +During my undergraduate, I was fortunate to work closely with [https://www.huaxiuyao.io/ Prof. Huaxiu Yao] and [https://linjunz.github.io/ Prof. Linjun Zhang]. + +Previously, I studied Computer Science and Mathematics in [https://www.zju.edu.cn/english/ Zhejiang University], got an honors degree from [http://ckc.zju.edu.cn/ckcen/_t1906/main.psp Chu Kochen Honors College]. + + +== News +- 09/2024: Our [https://arxiv.org/abs/2405.19547 negCLIPLoss] paper is accepted by NeurIPS 2024 as spotlight! +- 06/2024: Started my internship at Microsoft! +- 01/2024: One paper ([https://arxiv.org/abs/2310.00535 JoMA]) is accepted by ICLR 2024! +- 12/2023: Attended NeurIPS 2023 at New Orleans. +- 09/2023: One paper ([https://arxiv.org/abs/2305.16380 Scan&Snap]) is accepted by NeurIPS 2024! +- 09/2023: Become a husky in UW! + + +== My Favourite Papers +{{(* denotes equal contribution or alphabetic ordering.)}} \n\n + + +{{Data Selection Algorithm}} +{{
}} +We studied how to efficiently select data for multimodal pretraining tasks, drawing inspiration from both empirical observations and theoretical insights.\n + +~~~ +{}{img_left}{photos/negcliploss.png}{alt text}{400}{180} +*[https://arxiv.org/abs/2405.19547 CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning]* {{ }}[https://github.com/ypwang61/negCLIPLoss_NormSim \[Code\]] [./pdfs/Poster_negCLIPLoss_NormSim.pdf \[Poster\]] [https://twitter.com/ypwang61/status/1798396572516151612 \[Twitter\]] [https://arxiv.org/abs/2402.02055 \[Previous Versions\]] \n + *Yiping Wang*\*, Yifang Chen\*, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon S. Du \n + /NeurIPS 2024 ({{Spotlight}})/\n\n + tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can obtain a new SOTA in [https://www.datacomp.ai/dcclip/leaderboard.html DataComp benchmark] when combined with other approaches. +~~~ + +{{
}} + + +{{Training Dynamics of Transformer}} +{{
}} +We attempted to analyze the training dynamics of transformers in a mathematical way.\n + +~~~ +{}{img_left}{photos/scan.png}{alt text}{400}{130} +*[https://arxiv.org/abs/2305.16380 Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer]* {{ }} [./pdfs/poster_scan_snap.pdf \[Poster\]] [https://twitter.com/tydsh/status/1663611845603885056 \[Twitter\]]\n + Yuandong Tian, *Yiping Wang*, Beidi Chen, Simon S. Du \n + /NeurIPS 2023/\n + /{{ Oral }} presentation at High-dimensional learning dynamics workshop @ ICML 2023/ \n\n + tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias. +~~~ + +~~~ +{}{img_left}{photos/joma.png}{alt text}{400}{130} +*[https://arxiv.org/abs/2310.00535 JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention]* {{ }}[https://twitter.com/tydsh/status/1709785496056930654 \[Twitter\]]\n + Yuandong Tian, *Yiping Wang*, Zhenyu Zhang, Beidi Chen, Simon S. Du \n + /ICLR 2024/ \n\n + tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models. +~~~ + +{{
}} + + diff --git a/jemdoc.css b/jemdoc.css index 00e05f0..fe0511c 100644 --- a/jemdoc.css +++ b/jemdoc.css @@ -13,6 +13,7 @@ body { margin: 0; } + #layout-menu { background: #f6f6f6; border: 1px solid #dddddd; @@ -365,3 +366,28 @@ table.imgtable, table.imgtable td { border: none; text-align: left; } + + +.letter-spacing { + letter-spacing: 0.5em; +} + +.preserve-space { + white-space: pre; +} + + +.boxed { + border: 1px solid #dddddd; /* 灰色边框 */ + background-color: #ffffee; /* 浅黄色背景 */ + padding: 12px 12px 0px 12px; /* 内边距 */ + margin-bottom: 5px; /* 下方留一些空白 */ + border-radius: 5px; /* 可选：让边框有一些圆角 */ +} + +.topic-head { + font-size: 1.05em; + font-weight: bold; + color:#cf5732; + margin-bottom: -2em; +} diff --git a/miscellaneous.html b/miscellaneous.html new file mode 100644 index 0000000..8bd7ff2 --- /dev/null +++ b/miscellaneous.html @@ -0,0 +1,53 @@ + + + + + + +Miscellaneous + + + + + + + +
+
Yiping Wang
+
Home
+
Publications
+
Miscellaneous
+
Fun
+
CV
+		 +
+

Miscellaneous

+
+

Internship

+
    +

  • 06/2024 - Present: Research Intern @ Microsoft, Weizhu Chen's Group
    +Mentor: Yelong Shen and Shuohang Wang
    +Project: Video Generation

    +
    • +
+

Teaching

+ +

Services

+
    +

  • Paper Reviewer: NeurIPS 2023, ICLR 2024, ICML 2024, NeurIPS 2024

    +
    • +

  • UW CSE Ph.D. Admission Reviewer: 2024

    +
    • +
+

Honors and Awards

+
    +

  • 12/2022: Chu Kochen Scholarship in Zhejiang University

    +
    • +
+
+ + diff --git a/miscellaneous.jemdoc b/miscellaneous.jemdoc new file mode 100644 index 0000000..eb4c1d6 --- /dev/null +++ b/miscellaneous.jemdoc @@ -0,0 +1,18 @@ +# jemdoc: menu{MENU}{miscellaneous.html}, nofooter +==Miscellaneous + +== Internship +- 06\/2024 - Present: Research Intern @ Microsoft, Weizhu Chen's Group\n + Mentor: [https://scholar.google.com/citations?user=S6OFEFEAAAAJ Yelong Shen] and [https://sites.google.com/site/shuohangsite/ Shuohang Wang]\n + Project: Video Generation + +== Teaching +- TA in [https://courses.cs.washington.edu/courses/cse543/24au/ CSE 543 Deep Learning (24Au)] + +== Services +- Paper Reviewer: NeurIPS 2023, ICLR 2024, ICML 2024, NeurIPS 2024 +- UW CSE Ph.D. Admission Reviewer: 2024 + +== Honors and Awards +- 12\/2022: /Chu Kochen Scholarship/ in Zhejiang University + diff --git a/ori.html b/ori.html new file mode 100644 index 0000000..d475b53 --- /dev/null +++ b/ori.html @@ -0,0 +1,75 @@ + + + + + + +Yiping Wang 王宜平 + + + + + + + +
+
Yiping Wang
+
Home
+
CV
+		 +
+

Yiping Wang 王宜平

+
+ +
+alt text 

Yiping Wang
+Ph.D student
Paul G. Allen School of Computer Science & Engineering,
+University of Washington
+Email: ypwang61 at cs dot washington dot edu
+Google Scholar
+Twitter

+
+

About me

+

I'm a first-year Ph.D. student in Paul G. Allen School of Computer Science & Engineering from University of Washington. +I feel very fortunate to have worked under the guidance of Prof. Simon Shaolei Du since 2022 summer.

+

My main research interest broadly spread across machine learning theory and foundation models. +For the theortical part, I care about understanding the foundations of deep learning and representation learning, especially the training dynamics of the basic components like Transformer. +For the empirical part, I am keen on developing efficient algorithms with strong theoretical guarantees or insightful observations. In this aspect, currently I'm working on data selection/scheduling for multi-modal pretraining and improving model efficiency. +In addition, I always hold a strong enthusiasm for understanding the essence of intelligence and exploring the cross-cutting areas of mathematics, physics, and AGI, such as using LLM for mathematical proof.

+

Previously, I got my bachelor's degree in Computer Science & Technology from Zhejiang University in 2023, with an honor degree from Chu Kochen Honors College. +I also minored in Mathematics at Zhejiang University. +During my undergraduate, I was very fortunate to work closely with Dr. Yuandong Tian, Prof. Huaxiu Yao, and Prof. Linjun Zhang on several exciting research projects and learned a lot.

+

Selected Research

+

*: indicating equal contribution or alphabetic ordering.

+
    +

  1. CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning [Code]
    +*Yiping Wang, *Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon Shaolei Du
    +Preprint.
    +tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can achieve a new SOTA in DataComp benchmark when combined with current best approaches.

    +
    2. +

  2. JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
    +Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon Du.
    +International Conference on Learning Representations (ICLR) 2024
    +tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models.

    +
    3. +

  3. Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
    +Yuandong Tian, Yiping Wang, Beidi Chen, Simon Du.
    +Conference on Neural Information Processing Systems (NeurIPS) 2023
    +Selected as Oral presentation at High-dimensional learning dynamics workshop at ICML 2023
    +tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias.

    +
    4. +

  4. Improved Active Multi-Task Representation Learning via Lasso
    +Yiping Wang, Yifang Chen, Kevin Jamieson, Simon Du.
    +International Conference on Machine Learning (ICML) 2023
    +tl;dr: We improve the sample complexity of active multi-task representation learning by proposing a new LASSO-based strategy.

    +
    5. +

  5. C-Mixup: Improving Generalization in Regression [Code]
    +*Huaxiu Yao, *Yiping Wang, Linjun Zhang, James Zou, Chelsea Finn.
    +Conference on Neural Information Processing Systems (NeurIPS) 2022
    +tl;dr: We propose a simple yet effective data augmentation method to improve generalization on regression tasks.

    +
    6. +
+
+ + diff --git a/pdfs/Poster_negCLIPLoss_NormSim.pdf b/pdfs/Poster_negCLIPLoss_NormSim.pdf new file mode 100644 index 0000000..1d27fa8 Binary files /dev/null and b/pdfs/Poster_negCLIPLoss_NormSim.pdf differ diff --git a/pdfs/poster_scan_snap.pdf b/pdfs/poster_scan_snap.pdf new file mode 100644 index 0000000..4a0c49a Binary files /dev/null and b/pdfs/poster_scan_snap.pdf differ diff --git a/photos/L1_A_MTRL.png b/photos/L1_A_MTRL.png new file mode 100644 index 0000000..269633f Binary files /dev/null and b/photos/L1_A_MTRL.png differ diff --git a/photos/cmixup.png b/photos/cmixup.png new file mode 100644 index 0000000..dd40ecb Binary files /dev/null and b/photos/cmixup.png differ diff --git a/photos/joma.png b/photos/joma.png new file mode 100644 index 0000000..e6423d5 Binary files /dev/null and b/photos/joma.png differ diff --git a/photos/negcliploss.png b/photos/negcliploss.png new file mode 100644 index 0000000..d935d9c Binary files /dev/null and b/photos/negcliploss.png differ diff --git a/photos/scan.png b/photos/scan.png new file mode 100644 index 0000000..30766ef Binary files /dev/null and b/photos/scan.png differ diff --git a/pub.html b/pub.html new file mode 100644 index 0000000..7653ebb --- /dev/null +++ b/pub.html @@ -0,0 +1,67 @@ + + + + + + +Selected Publications + + + + + + + +
+
Yiping Wang
+
Home
+
Publications
+
Miscellaneous
+
Fun
+
CV
+		 +
+

Selected Publications

+
+

For the comprehensive list, check out my Google Scholar page.

+

(* denotes equal contribution or alphabetic ordering.)

+ +
+alt text 

CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning [Code] [Poster] [Twitter] [Previous Versions]
+Yiping Wang*, Yifang Chen*, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon S. Du
+NeurIPS 2024 (Spotlight)

+tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can obtain a new SOTA in DataComp benchmark when combined with other approaches.

+
+ +
+alt text 

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention [Twitter]
+Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, Simon S. Du
+ICLR 2024

+tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models.

+
+ +
+alt text 

Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer [Poster] [Twitter]
+Yuandong Tian, Yiping Wang, Beidi Chen, Simon S. Du
+NeurIPS 2023
+ Oral presentation at High-dimensional learning dynamics workshop @ ICML 2023

+tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias.

+
+ +
+alt text 

Improved Active Multi-Task Representation Learning via Lasso
+Yiping Wang, Yifang Chen, Kevin Jamieson, Simon S. Du
+ICML 2023

+tl;dr: We improve the sample complexity of active multi-task representation learning by proposing a new LASSO-based strategy.

+
+ +
+alt text 

C-Mixup: Improving Generalization in Regression [Code]
+Huaxiu Yao*, Yiping Wang*, Linjun Zhang, James Zou, Chelsea Finn
+NeurIPS 2022

+tl;dr: We propose a simple yet effective data augmentation method to improve generalization on regression tasks.

+
+
+ + diff --git a/pub.jemdoc b/pub.jemdoc new file mode 100644 index 0000000..f79a623 --- /dev/null +++ b/pub.jemdoc @@ -0,0 +1,46 @@ +# jemdoc: menu{MENU}{pub.html}, nofooter +==Selected Publications + +For the comprehensive list, check out my [https://scholar.google.com/citations?user=IuMFxFUAAAAJ&hl=en&oi=ao Google Scholar] page. \n +{{(* denotes equal contribution or alphabetic ordering.)}} \n\n + +~~~ +{}{img_left}{photos/negcliploss.png}{alt text}{400}{180} +*[https://arxiv.org/abs/2405.19547 CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning]* {{ }}[https://github.com/ypwang61/negCLIPLoss_NormSim \[Code\]] [./pdfs/Poster_negCLIPLoss_NormSim.pdf \[Poster\]] [https://twitter.com/ypwang61/status/1798396572516151612 \[Twitter\]] [https://arxiv.org/abs/2402.02055 \[Previous Versions\]] \n + *Yiping Wang*\*, Yifang Chen\*, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, Simon S. Du \n + /NeurIPS 2024 ({{Spotlight}})/\n\n + tl;dr: We design universal data selection methods for CLIP pretraining and achieve near SOTA results with less than 10% of preprocessing resources. It can obtain a new SOTA in [https://www.datacomp.ai/dcclip/leaderboard.html DataComp benchmark] when combined with other approaches. +~~~ + +~~~ +{}{img_left}{photos/joma.png}{alt text}{400}{150} +*[https://arxiv.org/abs/2310.00535 JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention]* {{ }}[https://twitter.com/tydsh/status/1709785496056930654 \[Twitter\]]\n + Yuandong Tian, *Yiping Wang*, Zhenyu Zhang, Beidi Chen, Simon S. Du \n + /ICLR 2024/ \n\n + tl;dr: We analyze the training dynamics of multilayer transformer, characterizing the role of self-attention, MLP nonlinearity, and the learning procedure of hierarchical structure, if the data follow hierarchical generative models.\n\n +~~~ + +~~~ +{}{img_left}{photos/scan.png}{alt text}{400}{140} +*[https://arxiv.org/abs/2305.16380 Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer]* {{ }} [./pdfs/poster_scan_snap.pdf \[Poster\]] [https://twitter.com/tydsh/status/1663611845603885056 \[Twitter\]]\n + Yuandong Tian, *Yiping Wang*, Beidi Chen, Simon S. Du \n + /NeurIPS 2023/\n + /{{ Oral }} presentation at High-dimensional learning dynamics workshop @ ICML 2023/ \n\n + tl;dr: We analyze the 1-layer transformer with next token prediction loss, and rigorously prove its training process and reveal how the token is combined via self-attention layer and the nature of its inductive bias. +~~~ + +~~~ +{}{img_left}{photos/L1_A_MTRL.png}{alt text}{400}{160} +*[https://arxiv.org/abs/2306.02556 Improved Active Multi-Task Representation Learning via Lasso]* \n + *Yiping Wang*, Yifang Chen, Kevin Jamieson, Simon S. Du \n + /ICML 2023/ \n\n + tl;dr: We improve the sample complexity of active multi-task representation learning by proposing a new LASSO-based strategy. +~~~ + +~~~ +{}{img_left}{photos/cmixup.png}{alt text}{400}{140} +*[https://arxiv.org/abs/2210.05775 C-Mixup: Improving Generalization in Regression]* {{ }}[https://github.com/huaxiuyao/C-Mixup \[Code\]] \n + Huaxiu Yao\*, *Yiping Wang*\*, Linjun Zhang, James Zou, Chelsea Finn \n + /NeurIPS 2022/ \n\n + tl;dr: We propose a simple yet effective data augmentation method to improve generalization on regression tasks. +~~~ \ No newline at end of file diff --git a/run.sh b/run.sh new file mode 100644 index 0000000..cd6db9c --- /dev/null +++ b/run.sh @@ -0,0 +1,2 @@ +python jemdoc.py index2.jemdoc pub.jemdoc miscellaneous.jemdoc fun.jemdoc +