My institution is currently transitioning to Canvas from Blackboard (hallelujah), and I’ve been taking the opportunity to dig into the Canvas api. Fun times! After a lengthy process of figuring out how to use PostMan to get data, and then how to get that data into PowerBI, and then how to get the tables in PowerBI to relate to each other properly, and then how to begin to understand what I was seeing, I could get down to figuring out how to ask the questions I wanted to ask. My hope was that some initial exploration would then help direct me to a fuller understanding of the data, so I could imagine more fully how to get the data structured in the ways I wanted it to be structured. But first things first–what kinds of things does Canvas keep track of? What can we ask of it?
Canvas API Reference
The first place anyone interested in Canvas data should explore is the Canvas REST API documentation. We’re using v1, and you can check out the documentation on Instructure’s website.
In short, Canvas keeps track of almost everything, down to what specfic kind of device and browser on the device a user employs to interact with the LMS–this is an actual thing called a “user agent”, and it turns out it’s hella complicated. You can go down that rabbit hole here. Canvas keeps track of how long a user is active in any given session; how many assignments are turned in late, what kinds of quiz questions a faculty uses, how long it took to grade an assignment, what pronouns users use and when they were set. It keeps track of each click and action and timestamp of a single user’s activity over the course of the entire term. It keeps track of grades at any given point during the semester (hidden and visible to students), what module pages have been viewed, and when. It keeps track of IP addresses. It keeps track of all this stuff in ways that can be linked to programs and across colleges, by course modality, by instructor, and by each offering of a course (like EN100) over the life of the LMS.
A better question would be what doesn’t it keep track of. It doesn’t keep track of data gathered by third-party tools, like TurnItIn, McGraw-Hill Connect, or Qwickly.
I spent a good amount of time with this resource and came up with a preliminary list of things I wanted to know more about. I won’t go into them all, but I’ll dig into a few to give a sense of it.
My Initial Questions
As a former teacher myself, I was generally interested in things like course setup and design choices, how students interacted with Canvas (what did they do when they logged in?), and how these choices might (or might not) influence student success and shape our course design directions. Are courses that have to-do dates associated with pages more or less likely to have higher than average grades, for instance?
And I was especially interested how we could compare these topics across different colleges or programs, different levels of instruction, different modalities of instruction, and so on.
These questions are all very broad, of course, so I also thought about what smaller questions I could aggregate into some larger answers.
How do faculty structure their home pages and navigation options for students, and does there seem to be any impact from that on student behavior?
What navigation features are available vs hidden to students; how does this impact pageviews, participations. Any correlation to other data points, including grades? Student satisfaction? Student assessment of course organization? What differences exist in student behavior between courses that have announcements on the front page, vs not? With modules as home page, vs a welcome page as home page?
Syllabus use—how do faculty use the native Canvas syllabus page?
If faculty use the Canvas syllabus course schedule, are there dates associated with all assignments? To-do dates associated with pages? Are there any significant correlations between use of the course schedule feature and student participation/activity? How often do students access content from the schedule vs access content from modules?
What patterns or habits characterize student navigation of Canvas?
Where do they go, and how long do they remain in those places? In what order do they typically navigate options in Canvas? How do they access their assignments–through the navigation bar, or through modules? Discussion boards? Assignments and Discussions are two things I’d probably recommend turning off in the navigation bar.
What impact does frequent communication in Canvas have on student success?
How often do faculty send out announcements, and how often are students viewing them? Messaging in Canvas inbox? Do faculty use the native Chat feature of Canvas (an integration)? If so, what is student use like? Does more frequent communication lead to higher grades? Higher participations? Higher page views? What kind of communication is best for student success? What are the subjects of student/faculty communication through Canvas? How many levels of depth (replies)?
What kinds of assignments do faculty create? How do students engage with assignments?
How many require peer review? How is peer review done? How many require Turnitin? Is there any correlation between Turnitin use and peer review? Between Turnitin and student success? Between peer review and student success? Do faculty use rubrics? To grade? How many faculty who use rubrics also add overall comments? What about comments on the actual submission? What is the nature of faculty commentary?
What kinds of quizzes do faculty create, more specifically? How do students interact with quizzes?
How often do faculty use item banks and randomized questions from item banks? In what kinds of courses? What kinds of questions are used? Are there any patterns in question types and department and/or college/school? How often do faculty flag questions? How many quizzes use Honorlock? How often do faculty refer to Honorlock flags? What type of quiz questions routinely stump students? How do quiz grades relate to assignment grades? Are there differences in student outcomes between quizzes with and without Honorlock? With and without randomized item banks?
How many (and what) third-party tools do faculty use, and what patterns exist between tool use and student success?
Is there a sweet spot in terms of number of third-party tools used? Are these tools used for content delivery or for graded assessments? Which third-party tools are most used? Least used? Are there any correlations between third-party tool use and student success?
With this list of questions in hand, I went back to the data–by this time, I’d collected a bunch, and stuffed it all into a Power BI report with many, many related tables.
Tonya Howe 