Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
teaching:st23:master-seminar [2023/09/29 17:32] ipa [Literature] |
teaching:st23:master-seminar [2023/10/11 19:56] (current) ipa [Organization] |
||
|---|---|---|---|
| Line 15: | Line 15: | ||
| ==== Descripion of Seminar. ==== | ==== Descripion of Seminar. ==== | ||
| - | Score-based generative models has established state-of-the art performance in many applications for the past recent years. The key idea of these models is to inject successively noise to the training data and then learn to reverse this process for the generation of new samples. These models can be learned with noise-conditional score networks and samples can be obtained via Langevin Monte Carlo approaches, stochastic differential equations, ordinary differential equations and combinations. | + | Score-based generative models have demonstrated state-of-the-art performance in numerous applications in recent years. The central concept behind these models involves the gradual injection of noise into the training data, followed by learning the reverse process to generate new samples. The training and sampling procedures can be conducted independently. The learning phase is facilitated by noise-conditional score networks, while sampling can be accomplished through various methods, including Langevin Monte Carlo approaches, stochastic differential equations, ordinary differential equations, and various combinations. |
| - | + | ||
| - | + | ||
| - | + | ||
| + | In this seminar, we will begin by reviewing generative models and examining the most common architectures used in current research. Special attention will be given to comparing different objective functions needed for training, as well as the different sampling procedures. We will explore invertible neural networks, with a particular focus on normalizing flows and continuous normalizing flows. Subsequently, we will address score matching and Langevin dynamics for score-based generative models. This includes an explanation of how Langevin dynamics can be used to approximate scores, deriving noise-conditional score networks, and providing a detailed explanation of the training process. | ||
| + | Additionally, we will explore a broader generalization involving an infinite number of time steps for noise levels, studying the process using stochastic differential equations. This formulation, known as score SDEs, leverages SDEs for noise perturbation and sample generation. The seminar will conclude with a comparison to other possible diffusion models and a discussion of further enhancements in sample generation. | ||
| + | The seminar is scheduled for the second half of the winter term. Participants interested in reviewing concepts of stochastic differential equations have the option to attend a previous seminar titled [[teaching:st23:seminar| Stochastic Differential Equations and Generative Modelling (Proseminar/Seminar)]], which takes place in the first half of the winter term. | ||
| ==== Organization ==== | ==== Organization ==== | ||
| * **Prerequisites:** Basic knowledge in probability theory and statistics | * **Prerequisites:** Basic knowledge in probability theory and statistics | ||
| - | * **Registration:** Via Müsli. [[https://muesli.mathi.uni-heidelberg.de/lecture/view/1756|Link]] | + | * **Registration:** Via Müsli. [[https://muesli.mathi.uni-heidelberg.de/lecture/view/1757|Link]] |
| - | * **First (organizational) meeting:** Kalenderwoche 42. Specific day and time will be announced soon. | + | * **First (organizational) meeting:** Tuesday, 17 October at 14:00 c.t. |
| - | * **Time and Location:** Time and location will be announced soon. | + | * **Time and Location:** Tuesdays 14:00 c.t. in SR 6 |
| - | Further information on the seminar will be announced in the first organizational meeting. For any specific question you can contact [[:people | Daniel Gonzalez]]. | + | Further information on the seminar will be announced in the first organizational meeting. For any specific question you can contact [[:people | Daniel Gonzalez, Jonas Cassel]]. |
| Line 39: | Line 38: | ||
| * **A conceptual introduction to Markov chain Monte Carlo methods**, //Speagle, Joshua S//,arXiv preprint (2019) | * **A conceptual introduction to Markov chain Monte Carlo methods**, //Speagle, Joshua S//,arXiv preprint (2019) | ||
| * **Neural ordinary differential equations**, // Chen, Ricky TQ and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David K// NeurIPS (2018) | * **Neural ordinary differential equations**, // Chen, Ricky TQ and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David K// NeurIPS (2018) | ||
| - | * **Ffjord: Free-form continuous dynamics for scalable reversible generative models**, // Grathwohl, Will and Chen, Ricky TQ and Bettencourt, Jesse and Sutskever, Ilya and Duvenaud, David// arXiv preprint (2018) | + | * **Ffjord: Free-form continuous dynamics for scalable reversible generative models**, // Grathwohl, Will and Chen, Ricky TQ and Bettencourt, Jesse and Sutskever, Ilya and Duvenaud, David// arXiv preprint (2018) |
| + | * **Diffusion models: A comprehensive survey of methods and applications**,// Yang, Ling and Zhang, Zhilong and Song, Yang and Hong, Shenda and Xu, Runsheng and Zhao, Yue and Shao, Yingxia and Zhang, Wentao and Cui, Bin and Yang, Ming-Hsuan//, arXiv preprint (2022) | ||
| * **Applied stochastic differential equations**,// Särkkä, Simo and Solin, Arno//, Cambridge University Press (2019) | * **Applied stochastic differential equations**,// Särkkä, Simo and Solin, Arno//, Cambridge University Press (2019) | ||
| * **Generative modeling by estimating gradients of the data distribution**,// Song, Yang and Ermon, Stefano//, NeurIPS (2019) | * **Generative modeling by estimating gradients of the data distribution**,// Song, Yang and Ermon, Stefano//, NeurIPS (2019) | ||
| * **Improved techniques for training score-based generative models**,// Song, Yang and Ermon, Stefano//, NeurIPS (2020) | * **Improved techniques for training score-based generative models**,// Song, Yang and Ermon, Stefano//, NeurIPS (2020) | ||
| * **Score-based generative modeling through stochastic differential equations**,// Song, Yang and Sohl-Dickstein, Jascha and Kingma, Diederik P and Kumar, Abhishek and Ermon, Stefano and Poole, Ben//, ICLR (2021) | * **Score-based generative modeling through stochastic differential equations**,// Song, Yang and Sohl-Dickstein, Jascha and Kingma, Diederik P and Kumar, Abhishek and Ermon, Stefano and Poole, Ben//, ICLR (2021) | ||
| - | * **Diffusion models: A comprehensive survey of methods and applications**,// Yang, Ling and Zhang, Zhilong and Song, Yang and Hong, Shenda and Xu, Runsheng and Zhao, Yue and Shao, Yingxia and Zhang, Wentao and Cui, Bin and Yang, Ming-Hsuan//, arXiv preprint (2022) | + | * **Gotta go fast when generating data with score-based models**,// olicoeur-Martineau, Alexia and Li, Ke and Piché-Taillefer, Rémi and Kachman, Tal and Mitliagkas, Ioannis//, arXiv preprint (2021) |
| + | |||