Lecture 0: Verification and Validation (V&V) for Machine Learning (ML) Models
Verification and validation (V&V) are essential activities in product development. V&V ensures that the product meets the technical specifications as well as user requirements. Verification is the process of determining whether the product is built correctly, while validation is the process of determining whether the right product is built. In this lecture,
- we will review V&V definitions, scope differences, activities, and standards;
- we will discuss the importance of V&V as a critical step in product development process, especially for AI products that often consists of machine learning (ML) models;
- we will also highlight how V&V frameworks can help in risk management and technology readiness assessment;
- time permitting, we will discuss V&V techniques to verify input-output specification and find failure modes of ML models through class activities.
Schedule
- Duration: 45 minutes lectures and in-class activities + 15 minutes Q&A
- Date: 2024-09-05
- Time: 09:45 - 10:45 AM SST (Singapore Standard Time, UTC+8)
- Location: Teams Meeting (Link will be shared via email)
Instructor
- Name: Mansur M. Arief, PhD.
- Affiliation: Postdoctoral Scholar, Stanford Intelligent Systems Laboratory (SISL)
- Institution: Stanford University
- Email: mansur.arief@stanford.edu
- Website: mansurarief.github.io
Learning Objectives
- Students can identify V&V tasks in product developments, especially for AI products.
- Students can describe the importance of V&V in product development process.
- Students can illustrate V&V methods and activities and describe their outcomes.
- Students can give examples of falsification techniques to find failures of ML models.
Tools
- Microsoft Teams: Please make sure you have installed Teams on your computer or mobile device or use the web version.
- Google Slides: Mainly for presentation by the instructor.
- Poll Everywhere: For interactive quizzes and polls. You can use your smartphone or computer to participate and no account is required.
- HuggingFace Spaces: For class activity on adversarial attacks. You do NOT need a HuggingFace account to access the space.
(Optional) Reading Materials
- Concept Briefs:
- Verification and Validation: Concept Overview and Terminologies [highly recommend to skim through before the lecture]
- Intro to Supervised Machine Learning [recommended for those who are not familiar with ML or deep learning, or simply need a refresher]
- Machine Learning Verification Example and Adversarial Attacks [useful to checkout if you are interested in how to code adversarial attacks in Python]
- NASA Systems Engineering Handbook, Chapter 5.3 and 5.4 [one of the main references for V&V in aviation and aerospace industry from systems engineering perspective, though not necessarily specific to AI products]
- Algorithms for Verifying Deep Neural Nets (Changliu Liu, et al., 2021) [recommended for those interested in the theoretical aspect of ML verification – FYI, this level of rigor is beyond the scope of UG course, but might be useful for graduate students or those interested in pursuing research in this area]
Accommodations
- Students are expected to participate actively in discussions and class activities. If you have any special needs, please let me know in advance so that I can make necessary accommodations.
- For optimal learning experience, please make sure you have a stable internet connection, a quiet place to participate in the class, laptop or desktop computer, and a working microphone and camera.
- If English is not your first language and you need additional support, please let me know in advance so that I can adjust my speaking pace and provide additional explanations when needed.
AI Usage Policy
You may use AI tools to assist your learning process. Please be aware of responsible and ethical use of these tools. Never use them to cheat and do not share any private information when using them, especially those that are not of yours. Please also be aware of the university’s stance on generative AI usage in classrooms and policy on academic integrity and plagiarism. If you have any questions about this policy, please consult the official guidelines or ask the instructor.
Acknowledgement
Copilot has been used to generate text for this course website, but the final content has been reviewed and edited by the instructor.