Data Science Seminar: Seminar Machine Learning - Large Language Models @ Leuphana University Lüneburg
Instructor: Ricardo Usbeck ORCID Google Scholar
Language models, which are trained to predict text given other text, underlie many recent successes in natural language processing and artificial intelligence. Whether used for transfer learning (using language modelling as a pre-training objective before subsequent fine-tuning on a downstream task) or prompting (formulating an input sequence that induces a model to perform a desired task without any training), language modelling has proven to be an effective way of imbuing models with functional capabilities. These capabilities have been observed to consistently improve as the size of the language model increases, which has led to a focus on developing ever-larger language models. In this course, we will survey the history of language models and their future (agents and ethics included), as well as recent advances in building, analyzing, and using large LMs.
Students must have experience with machine learning (preferably deep learning) and the basics of modern natural language processing. Before taking the class, you should be able to read a recent machine learning or natural language processing conference paper and come away with a decent understanding of the basic concepts and ideas proposed in the paper (but not necessarily a deep, perfect knowledge of every last detail).
This class will use a role-playing seminar format where students take on different roles and present papers to one another. All grading will be based on these presentations and course participation, culminating in a final poster.
Each class will involve the presentation and discussion of two one papers.
- Before each class, everyone is required to have read the paper.
- Students will be assigned roles.
- This role defines the lens through which they read the paper and determines what they prepare for the in-class discussion.
- Students in the non-presenting groups are also required to read the paper, complete a quick exercise (described below), and come to class ready to discuss.
- All students will obtain a thorough understanding of the chosen papers and will develop their paper reading, literature review, and prototyping skills.
This seminar is organized around the different "roles" students play each week. The number of roles given out depends on the number of students in the class.
- Explainer (Theory): The explainer describes the underlying method and its theoretical foundation to the group. They can use the whiteboard to make a 1 sketch or bring a 1 handout or make 1 slide. Do not switch between tabs but stay on the same image. The goal is, that students understand the underlying theory in detail. The time-box for this role is 5 minutes max. The time-box for this role is 10 minutes max.
- Diagrammer: Create a diagram of one of the concepts or ideas from the paper, or remake one of the plots in the paper to make it more straightforward. Please select a topic that hasn't been diagrammed in a previous paper. The time-box for this role is 5 minutes max. The time-box for this role is 5 minutes max.
- Scientific Peer Reviewers: Complete a full, critical, but not necessarily negative, review of the paper. Follow the guidelines for NeurIPS reviewers (under "Review Form"). Please complete the "Strengths and Weaknesses" and "Questions" sections and assign an overall score; you can skip the rest of the review (including writing a summary, since all students should have read the paper). You can go to OpenReview to see actual reviews, e.g., search for my name. The time-box for this role is 5 minutes max.
- Archaeologist: Could you determine where this paper sits in the context of previous and subsequent work? Find and report on one prior paper that we are not reading in this class that substantially influenced the current paper or one newer paper that we are not reading in this class that was heavily influenced by the current paper. The time-box for this role is 5 minutes max.
- Academic Researcher: You’re a researcher who is working on a new project in this area. Propose an imaginary follow-up project not just based on the current, but only possible due to the existence and success of the current paper. The time-box for this role is 5 minutes max.
- Industry Practitioner: You work at a company or organization developing an application or product of your choice (that has not already been suggested in a prior session). I'd like you to bring a convincing pitch for why you should be paid to implement the method in the paper and discuss at least one positive and negative impact of this application. The time-box for this role is 5 minutes max.
- Private Investigator: You are a detective who needs to run a background check on one of the paper’s authors. Where have they worked? What did they study? What previous projects might have led to working on this one? What motivated them to work on this project? Feel free to contact the authors, but remember to be courteous, polite, and on-topic. Can you say something about the venue where this paper was published? Can you say something about the widespread interest in the paper, if there was any? Also, connect back to the authors of the earlier paper in this seminar! The time-box for this role is 5 minutes max.
- Hacker: Implement a small part of the paper on a small dataset or toy problem. Prepare to share the core code of the algorithm with the class. Do not simply download and run an existing implementation - you should implement at least a (toy version of a) method from the paper. However, you are welcome to use (and give credit to) an existing implementation for "backbone" code (e.g., model building, data loading, training loop, etc.). Pre-record a video as a backup. The time-box for this role is 10 minutes max.
- Blogger (not this semester): Write a paragraph about each of the two papers and an additional paragraph comparing and contrasting them. The summary of each paper should cover the motivation behind the paper, a description of any of the proposed methods, and an overview of the key findings. You should write a bit about how they are different and/or build on one another. The blogger will not be present during the class session.
- Social Impact and Sustainability Assessor (not this semester) Identify how this paper self-assesses its (likely positive) impact on the world. Have any additional positive social impacts left out? What are possible negative social impacts that were overlooked or omitted? The time-box for this role is 5 minutes max.
If you aren't in the presenting group during a given class period, please prepare the following and send it via email:
-
- Three questions about either paper - could be something you're confused about or something you'd like to hear discussed more. Also, try to form a hypothesis to answer your question and write it down.
-
- (not this semester) A new title for either one of the papers and/or a new name for an algorithm proposed in either paper
-
- (not this semester) One idea for a missing experiment.
- Presentations (up to 10 possibilities to be selected): For each class session where you are presenting, you will be graded out of 5 points (German university system). You will receive full credit if you do a thorough job of undertaking your role and present it in a clear and compelling way. The score will be averaged and normalized by the number of participating sessions.
- Discussion (up to 10 possibilities): For each class session where you aren't presenting, you'll be given 1 point for completing the non-presenter assignment and attending and participating in class. The score will be summed up and normalized by total sessions.
If you miss a class without completing the corresponding assignment, you'll get a zero for that session.
If you miss a class where you are in a "presenting" role for that session, you must still create the presentation (2-pager) for that role a day before the class.
If you miss a class where you'd be in a "non-presenting" role, to get credit for that session, you need to complete the non-presenting assignment and send it to me a day before the start of class.
There's really no way to accept late work for the readings since it's vital that we're all reading the same papers at the same time.
All students are expected to follow the guidelines of the Leuphana Campus Rules. In this class, it is essential that you cite the source of different ideas, facts, or methods and do not claim someone else's work as your own. If you are unsure about which actions violate that honour code, please let me know.
I ask that we all follow the NeurIPS Code of Conduct and the Recurse Center Social Rules. Since this is a discussion class, we must respect everyone's perspective and input. In particular, I value the perspectives of individuals from all backgrounds reflecting the diversity of our students. I broadly define diversity to include race, gender identity, national origin, ethnicity, religion, social class, age, sexual orientation, political background, and physical and learning ability. I will strive to make this classroom an inclusive space for all students. Please let me know if there is anything I can do to improve.
Acts of discrimination, harassment, interpersonal (relationship) violence, sexual violence, sexual exploitation, stalking, and related retaliation are prohibited. If you have experienced these types of conduct, you are encouraged to report the incident and seek resources on campus or in the community. Please contact the Ombudsperson at https://www.leuphana.de/en/university/organisation/ombudsperson.html to discuss your specific needs.
The professor reserves the right to make changes to the syllabus, including project due dates. These changes will be announced as early as possible.
The schedule below includes a preliminary list of the papers we will be reading. These papers are subject to change, though I will try to make changes only to papers that are at least two weeks away. If you have any suggested changes, please let me know.
Why are we not talking about the newest models such as Gemini, Mistral, DeepSeek, Qwen or GPT? There are also new model families, such as State Space Models and KAN: Kolmogorov-Arnold Networks, which we cannot explore in detail here. We know too little, but you can prove me wrong!
Thanks to Colin Raffel for providing this great idea for a seminar under https://github.com/craffel/llm-seminar/