-
Notifications
You must be signed in to change notification settings - Fork 0
/
criteria-development
140 lines (128 loc) · 10.4 KB
/
criteria-development
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
# Criteria collection and classification
Our criteria have evolved over time. Our initial set of criteria was drawn from a previous platform comparison set of use cases, our own use cases after implementing the Open edX platform, and additional new ideas for use cases based on potential or desired projects that had not yet been initiated. Our Duke Extend service team reviewed this list of criteria and gave each a +1/0/-1 vote. That list is in Google Drive: https://docs.google.com/spreadsheets/d/1Efgz2Ym4mH0VjPFme6sedj2eyDtl8nSCHFUb1xoYJ6s/edit?usp=sharing
Each use case begins with “We’re looking for a platform(s) that allows the university to...”
1. Deliver courses over a variety of modalities to mixed audiences at traditional and large scales.
2. Offer training and assessment in basic technical skills to Duke affiliates and the public.
3. Partner with other universities to deliver the same course to all students at once in one learning space.
4. Show student progress to individual learners.++
5. Allow experimentation and demonstration of the capabilities of the platform.
6. Add traditional and large scale, mixed audiences to specific modules/features/tools of a course.
7. Scale to incorporate up to 10,000 learners in a single course.
8. Add traditional and large scale, mixed audiences to an entire course regardless of original audience and without the need for prior student enrollment or account creation.
9. Facilitate hardware training and track and validate document learning in an always-available resource.
10. Provide sensible methods for authentication and access to traditional and large scale mixed audiences.
11. Offer for-credit Duke courses to remote or prospective Duke students over summer terms.
12. Facilitate engaging group work and community beyond the discussion board at large scale.
13. Add to alumni benefits, promote community among alumni, and increase connection between alumni and Duke.
14. Automate certificate generation upon completion for mixed audiences at traditional and large scale.
15. Create properly licensed, modular content that can be remixed by mixed audiences and delivered to students regardless of platform.
16. Satisfy accessibility requirements held by the university (such as video captioning).
17. Offer training in professional areas outside the scope of university coursework.
18. Provide seamless integration of external tools within the platform.
19. Provide learner analytics across students, sections, subjects, degrees, and across platforms.
20. Easily offer access to Duke for-credit courses to non-Duke or remote students from multiple international locations.
21. Manage student assessment of team contribution regardless of platform.
22. Allow staff depts to offer technical training to new employees.
23. Offer consistently high stability for course access and interaction.
24. Offer adaptive hints and feedback to students.
25. Participate in an active development community.
26. Add mixed access to student created content.
27. Offer free MOOC-type courses with customized start times and durations.
28. Audit an entire course or sections of a course against quality standards set by CIT and the University.
29. Show course-wide student progress to instructors (grades & responses to assessments)
30. Content created within the platform can be used simultaneously in other platforms via embed or other functionality.
31. Deliver content created outside the platform.
32. Delegated access and elevated permissions be granted to appropriate individuals via self-service regardless of platform.
33. Offer a well-designed, modern interface to students and instructors.
34. Gain wide visibility for its courses to eligible learners.
35. Provide an open online course alternative to Coursera.
36. Platform is Duke branded
37. Control options related to
1. time of release,
2. duration of release,
3. frequency of release,
4. certification,
5. eligible learner population, and
6. frequency and nature of platform updates.
38. Tailor flow control of student experience through the course (adaptive learning)
Based on our team conversation about these criteria, we identified that many of them are complex combinations of features that made evaluation difficult. So I deconstructed a number of the criteria into a longer list of simpler features.
1. control learning experience options related to the frequency and nature of platform updates
2. deliver content created outside of the platform
3. provide learning analytics on learning communities to individual learners
4. add learners to a learning experience without need for enrollment and/or account creation
5. offer assessment via instructor grading
6. provide learning analytics on individual learners to the individual learners
7. demonstrate capabilities of the platform to prospective authors (e.g. faculty/instructors)
8. make the platform Duke-branded
9. integrate with external platforms and tools
10. control learning experience options related to eligible learner population
11. control learning experience options related to frequency of release
12. offer access to learner-created content
13. tailor the flow of learner interaction throughout the learning experience (adaptive learning)
14. gain wide visibility among eligible learners for its learning experience
15. offer adaptive hints and feedback in assessments
16. create licensed content for learning experience
17. facilitate quality community building among learners
18. increase the connection between the university and Duke alumni
19. promote community among Duke alumni
20. experiment with the capabilities of the platform
21. actively participate in a platform development community
22. partner with other universities to offer the same learning experience to a joint learning community
23. add learners to specific components of a learning experience (modules, tools, etc...)
24. increase the value of alumni benefits
25. control learning experience options related to certification
26. provide learning analytics on learning communities to administrators
27. facilitate quality group work among learners
28. deliver learning experiences at massive (5,000+) scale
29. create modular content that can be reused and remixed in multiple learning experiences
30. provide learning analytics in a standard format that allows comparison across platforms
31. offer Duke credit for learning experiences
32. control learning experience options related to monetary payment
33. control learning experience options related to size and nature of the content
34. control learning experience options related to content visibility
35. offer a well-designed, modern interface to users
36. integrate external tools seamlessly within the platform
37. control learning experience options related to time of release (start date)
38. provide learning analytics on individual learners to instructors
39. maintain quality control of content developed for use on the platform
40. manage learner assessment of team members' contributions to group work
41. control learning experience options related to duration of availability
42. provide methods for access and authentication
43. provide good value for the cost of maintaining the platform
44. deliver learning experiences at large (501-5,000) scale
45. grant elevated admin privileges to appropriate individuals
46. provide learning analytics on learning communities to instructors
47. satisfy university accessibility requirements
48. offer text-based instruction tools
49. deliver learning experiences at traditional (1-500) scale
50. offer multimedia/video based instruction tools
51. offer assessment via automated grading
52. offer assessment via peer evaluation
After further review and discussion, this list was consolidated into new criteria with fewer overlaps between features, listed below. These criteria were written on cards, one criterion per card, and shared with several staff in Learning Innovation to gain their perspectives about the importance of each. Participants were asked to identify critical criteria or non-essential criteria; all remaining criteria were considered as 'differentiating'. After eight participants contributed their input, the criteria were ranked and classified. Upvotes were based on a participant identifying a criterion as 'critical' and downvotes were based on a participant identifying a criterion as 'unnecessary' or 'non-essential'. After the exercise, participants were each asked two questions. What guiding principles did you rely on to make your classifications? What, if any, additional criteria would you suggest we consider for this platform comparison?
## Critical
* +8/8 Satisfy university accessibility requirements
* +7/8 Deliver learning experiences at all scales of audiences
* +7/8 Integrate with Shibboleth and OneLink for account access and authentication
* +6/8 Offer a well-designed, modern interface to users
* +5/8 Control learning experience content options (e.g. course content size, multimedia options, course licensing, importing/exporting content, embedding content)
## Differentiating
* +4/8 Offer sophisticated assessment options beyond pure MCQs
* +3/8 (4 upvotes; 1 downvote) Provide detailed learning analytics in usable format(s)
* +3/8 Two-way integrate with external platforms and tools
* +2/8 (3 up; 1 down) Facilitate quality group/team work among learners
* +2/8 (3 up; 1 down) Offer a high quality mobile interface learning experience
## Non-essential
* 0/8 (2 up; 2 down) Control learning experience delivery options (e.g. course timing, payment & certification, course layout, user experience flow through course, gamification, project partnerships)
* 0/8 (3 up; 3 down) Make the platform Duke-branded (white label)
* 0/8 (1 up; 1 down) Facilitate quality community building among learners
* -1/8 (1 up; 2 down) Provide better value for cost (than Open edX)
* -1/8 (2 up; 3 down) Quickly prepare authors to create their content in the platform
* -2/8 (2 up; 4 down) Directly manage admin privileges
* -2/8 (1 up; 3 down) Control learning experience support options (maintenance, issue tracking, platform updates, available APIs, standard I/O formats)
* -2/8 (1 up; 3 down) Facilitate sharing of learner-created content in platform and on social media
## Additional suggested criteria
* Offer clean data exports;
* Use an established platform/vendor;
* Use a platform based on open source code;
* Offer a platform that is easy to use;
* Offer a platform that is distinct in application from Sakai (and Coursera)