REMINDER: QDC Meeting tomorrow, Tuesday, October 16th @ 11am EST

Dear QDCers,

This is a reminder of our QDC Meeting tomorrow (Tuesday, October 16th) at 11am EST. Local MSU folks can meet in the CREATE Collab room. Zoom info is as follows:

Here are the items I have on the Agenda:

--- Updates

--- Marisol's practice of her 5-minute presentation for the AAC&U meeting

Please let me know if you'd like to add any other items!

Thank you all, and have a great night!



Tuesday, October 16th, 2018

Tuesday, October 16th, 2018

QDC Meeting

Attendees: Marisol, Kevin, Kristen, John, Jennifer Kaplan, Alex, Juli, Mark, Matt, Jenifer Saldanha, Kamali, Keenan

--- Updates
--- Marisol’s practice demo for her AAC&U presentation
--- Jennifer and Kristen share student responses on their P-value question



Juli, Marisol, Lauren, and Jen did COPUS training with Anne-Marie last week.

--- Is on the 3rd iteration of ecology questions. Down to 4 from 15 initial ones. Has split one of the questions into a 2 part question (based on some other AACR question templates)
Is refining working on other questions. Will be sending it to folks he has been working with for feedback. One of the questions : was about Biomagnification of toxins up the food web (involving polar bears and PCBs). Isometric version involving mercury and top predator fish
--- Will be contacting folks soon to get some pilot data
--- Comment from Kevin on isometric versions of questions: important parts of the question have changed (such as chemical involved, parts of the food web etc). One solution is to collect responses for both items and put them in the same pool. The other measured approach - if one is getting richer responses, we could target that question as the initial round to get things up and running and then pursue the other

--- Is revising the analytical rubric for the stop codon (analytical).
Is working with Kamali to clarify the language whether students are talking only about the process OR also implying understanding of what the product is. This will enable consistent coding

--- Received feedback on IRB application, and is editing it again.
--- Is working on the Argulex Rubric .
--- Will be presenting today (slides she has prepared for the AACU conference)

--- LEAP UP data, COPUS, meeting Kevin and John about unexplored AACR questions

- has been uploading data sets to the cloud database. Kevin and John are looking at them.

--- Will be sending coauthors of the diet paper a draft (Kevin and Rosa's comments will be consolidated)
--- Working on IRB application with Matt - visualization project
--- Meeting with John and Hailey - talk about issues with diffusion and flip rubrics. Bit of trouble with the diffusion one - 3 of them will be scoring responses for those bins to see how they interpret student responses

--- Is working on revisions to Jenny and Scott’s non-coding mutations paper.
Matt has discovered that we can refer to a well known and well researched method called "Naive Bayes Optimal Classifier" in our papers (Matt can elaborate)
--- The LEAP UP group has become interested in the question of how many student responses need to be scored by hand to get models to work - to help them Matt is starting the effort of taking the work he did 2 in modeling the performance growth curves with the number of hand scored responses and packaging that in an app - so they can upload in data and let the machine model it Caveat: it is noise sensitive, so take with large grain of salt. Is also computationally intensive and can take several hours. His plan is to have it run only on the development server .
Minimal app - person uploads data, hits run, the app disconnects and runs in the background and sends an email when done

Gravity works: Mark and Marisol met with them and gave them all the feedback collected (meetings + surveys). He has taken the comments back and is working with his designer on that

Joel Michael visited MSU last week. AACR group met with him. Questions he raised were similar to ones received from other instructors who are introduced to the work done in the AACR group. He gave positive feedback.

Jennifer: Kristen is learning how long to make a rubric. THey’re working on refining the rubric for P-value interpretations; they disagree on whether or not some answers deserve credit, and need a tiebreaker, which will be Alex. They’ve been experiencing the need to develop a few categories, and are in the middle of resolving disagreements for the first 150 responses. They have 3 semesters’ worth of responses to two different stems on P-values: one is a proportion problem, and one is a means problem. The proportion problem came first, followed by the means problem. They expect differences between the two stems, not just because of stem differences, but also hopefully because students will learn more and do a better job. However, they might also make completely different errors even after learning more. They’re starting with the earlier responses to get as many “wrong ideas” in the rubric as possible. Once they’ve solidified the rubric, they’ll recruit other coders. John encourages Jennifer and Kristen to share the actual responses so we can see what they're struggling with.

Marisol’s practice demo for her AAC&U presentation
--- Marisol submitted a proposal to the American Association of Colleges and Universities, dealing with the web portal. Paula Lemons submitted a separate proposal for the FLCs. We believe this is a very high-level conference, with a lot of administrators and heads at colleges and universities. The intention is to get the word out that our tool exists, and to hopefully disseminate it to institutions in a broader way. She has 5 minutes for the presentation and 7 minutes for questions.
--- Marisol’s full presentation is from minute 13:24 to 17:53 of the recording
--- Marisol’s presentation clocked in at 4:40 seconds
--- Kevin notes that Marisol should make the point that instructors have to use our AACR questions: instructors can find a question they like and use that question/s in their class.
--- John had a thought about the bar graph: Marisol mentioned the bar graph can provide information to instructors. John suggests pointing out e.g., Mass to Energy, which is a consistent misconceptions. A lot of the audience will recognize that as a learning barrier. Mark follows up on John’s point: Recall that these are administrators, not biologists, and likely to have similar misconceptions. As Marisol moves through her demo, she could e.g., highlight nodes for correct (e.g., Correct Products) and incorrect (Mass Converted to Energy) ideas. Jennifer agrees that Marisol wants to focus on correct and non-normative reasoning, they won’t be biologists, so Jennifer suggests saying “this is a misconception” rather than trying to explain connections. Jennifer suggests making the point of “What do these data look like? Why can’t instructors read the responses themselves?”, and pointing out how messy the answers are, e.g., with handwriting. Matt echoes this; providing motivation for the audience is important. In order to provide time, Matt suggests leaving out the table of responses with codes and probabilities. Mark notes that Marisol will have time at the end for Q&A, so she could also revisit these concepts.
--- Mark also notes that Marisol’s slide about revising the web portal goes a little fast, and suggests that she provide motivation for the usability study (e.g., we got feedback from faculty that prompted us to revise the web portal). Mark also suggests reducing the content on the slide (e.g., focusing on the usability). Marisol added this slide since one reviewer suggested that the interesting point of this presentation might be the details of the usability study; Mark and Kevin note that it might be better to have a briefer slide and keep the current slide as a backup.
--- Kevin notes that on Marisol’s last slide, the first bullet point should contain the AACR website
--- Mark also suggests having the website up to either show the invitation for the pilot study, or to show participants where to find questions. The administrators are likely not to use the portal themselves, but to talk to other instructors, etc. who will be the users. Mark also notes that we have brochures for the web portal that Marisol should also take with her and handout.
--- Jennifer also points out the need for the NSF logo and the two required sentences in order to be compliant.
--- Mark notes that Marisol can also steer her Q&A in some sense; she can also use lulls in the Q&A to bring up other features of the web portal that she didn’t have a chance to discuss.

Jennifer and Kristen share student responses on their P-value question
-- Jennifer and Kristen are looking for three things: 1) the probability is .026 that 2) they would’ve observed ≥ 70% of people in their sample that prefer watching sports 3) if 60% of the population prefers watching sports.
--- Broader context: The P-value is the probability that 1) they would have observed the sample statistic, 2) or one that is more extreme, 3) if the null hypothesis is true.
These three features are necessary for Jennifer and Kristen’s criteria for a complete answer.
--- Jennifer and Kristen showed some responses where they disagreed on if the responses had all three criteria or not.
--- 0’s and 1’s signify if Jennifer and Kristen agree or disagree in their coding. Typically the responses have the statistic (points 1 and 2) but not the conditional (point 3).
--- The group discusses how important the context is for scoring a response as a complete answer.
--- Alex weighs in one edge case: The group discusses a student response that first gives a complete explanation, but then follows up with an incorrect thought. The group discusses how to treat responses like that? Jennifer and Alex discuss interpretations. Jennifer points out that sending this response to the model will cause it to choke, because the answer has all elements of a correct response. Mark points out that this is the positive of an analytical rubric: to be able to capture students’ correct and incorrect ideas. Mark also notes that a large part of this might be students’ tendencies to recite memorized definitions. Jennifer agrees.
--- On the second edge case, Alex points out that the response is missing a single word in order to be considered complete. Kristen points out an alternative interpretation for the response. Mark suggests that edge cases, if the human coders can’t agree, the model won’t be able to either, so Jennifer Kristen and Alex might consider leaving these out. John also suggests assessing if the edge case occurs frequently enough that an instructor can take action on it. What do they want instructors to see?
--- Mark clarifies that they are assessing completeness based on three criteria? Jennifer confirms, and elaborates the three points.