Post on 18-Jan-2016
© Blackboard, Inc. All rights reserved.
CourseCopy NT (CoCoNuT)
Customized Course Copy operation
@ K.U.Leuven (Belgium)
2/38
Presentation overview
» Toledo team - KULeuven Association » Toledo in numbers» Blackboard implementation» Data integration» Blackboard Copy Operations» Archive – Restore» Customized Procedure – overview» Customized Procedure – pre processing» Customized Procedure – archive and restore» Customized Procedure – post processing
3/38
Toledo team - KULeuven Association
4/38
5/38
6/38
Toledo team - KULeuven Association
» KULeuven Association» 1 university +10 higher
education institutions
» K12» 140 k12 education
schools
2 production environments
» 1 project leader» 2 application managers (training and
support)» 3 software developers» 2 data managers (data integration)» 2 system administrators
7/38
Toledo in numbers
» KULeuven Association» Active users: 69.932 » Courses: 37.624
(+50% actively used)
» K12» Active users: 29.970 » Courses: 7.459
app server 2
app server 1
app server 3
Load balancer 1
Load balancer 2
Oracle DB server
Collab server
app server 1
Oracle DB server
NFS-cluster
8/38
Blackboard implementation
Users» Only two types of users: students and instructors » Primary institutionrole > reflects the Institution
users belong to (defines branding and tab pages)» Secondary institutionrole > reflects the users
permissions (defines modules on the tab pages)» 10 Blackboard system admins (Toledo Team)» 1 Local admin pro department, institution or k12 school» Staff » Student
9/38
Blackboard implementation
Courses
» 2 concurrent versions of one course» One copy of the previous academic year» One copy of the current academic year (set unavaileble @
copy)» Instructor decides when to set his course (un)available
Previous Current
CourseId a-courseid-0506 a-courseid-0607
Course Title course title [0506] course title
10/38
Toledo Server
IMS-XMLIMS-XML
Data integration
Institution a
Institution b
... Institution
k
Enterprise Information Systems
IMS-XML
perl
Nightly rsync
Toledo databa
seNightly parsed
Java BbJava/Bb-API
Java/php
5 min
Toledo applications•Enrollment module•Communities module...
Need extra storage of •organisational structure•User roles
Bb
11/38
Requirements for Course Copy @ K.U.Leuven
» The program has to be able to
» run in “batch” mode » be started from within blackboard for one or more courses,
as well as be started from bash» ignore “non-used” courses» add a “copied” flag to copied courses» check whether the target course already exists (created
with IMS-XML) and whether it ‘s not copied before (“copied” flag)
» unenroll student memberships, but keep staff memberships» set the target courses unavailable » change title of the source course (add [0506])» add an announcement in source and target courses» move the archive and its logfiles to a specific location» create symlinks at filesystem level (max #files/dir reached)
12/38
Blackboard Copy Operations
Control Panel
Batch Target course not required
Keeps users / interactions
Creates an archive
Copy Snapshot / batch copy
/ API -Clone() Export / import Archive / Restore
13/38
Archive
/usr/local/blackboard/apps/content-exchange/bin/batch_ImportExport.sh -f /path_to/A.dummyfile.txt -l 1 -t archive
a-courseid1-0506, /usr/local/blackboard/archives/
a-courseid2-0506, /usr/local/blackboard/archives/
a-courseid3-0506, /usr/local/blackboard/archives/
a-courseid4-0506, /usr/local/blackboard/archives/
a-courseid5-0506, /usr/local/blackboard/archives/
a-courseid6-0506, /usr/local/blackboard/archives/
A.dummyfile.txt
•-f path_to_file
•-l (delimiter) 1(,) 2(;) 3(tab)
•-t (type) import export archive restore
•-n virtual hostname
Options:
14/38
Restore
/usr/local/blackboard/apps/content-exchange/bin/batch_ImportExport.sh -f /path_to/R.dummyfile.txt -l 1 -t restore
a-courseid1-0607, /usr/local/blackboard/server/archives/ArchiveFile_a-courseid1-0506.zip
a-courseid1-0607, /usr/local/blackboard/server/archives/ArchiveFile_a-courseid2-0506.zip
a-courseid1-0607, /usr/local/blackboard/server/archives/ArchiveFile_a-courseid3-0506.zip
a-courseid1-0607, /usr/local/blackboard/server/archives/ArchiveFile_a-courseid4-0506.zip
a-courseid1-0607, /usr/local/blackboard/server/archives/ArchiveFile_a-courseid5-0506.zip
a-courseid1-0607, /usr/local/blackboard/server/archives/ArchiveFile_a-courseid6-0506.zip
R.dummyfile.txt
•-f path_to_file
•-l (delimiter) 1(,) 2(;) 3(tab)
•-t (type) import export archive restore
•-n virtual hostname
Options:
15/38
Customized Procedure – overviewif source has content if target existsif no copy flagremove targetcreate a/r filescreate symlinkarchiverestore
add annoucements add title suffixset target unavailableunenroll studentsadd copyflagmove logs and archive
Pre processing
Archive / Restore
Post processing
a-courseid1-0506, a-courseid1-0607
a-courseid2-0506, a-courseid2-0607
...
a-courseid1-0506, /path_to/archives/
a-courseid2-0506, /path_to/archives/
...a-courseid1-0607, /path_to/archives/ArchiveFile_a-courseid1-0506.zip
a-courseid2-0607, /path_to/archives/ArchiveFile_a-courseid2-0506.zip
...
mail (short) errorlog
16/38
Customized Procedure – overviewif source has content if target existsif no copy flagremove targetcreate a/r filescreate symlink
Pre processing
a-courseid1-0506, a-courseid1-0607
a-courseid2-0506, a-courseid2-0607
...
a-courseid1-0506, /path_to/archives/
a-courseid2-0506, /path_to/archives/
...a-courseid1-0607, /path_to/archives/ArchiveFile_a-courseid1-0506.zip
a-courseid2-0607, /path_to/archives/ArchiveFile_a-courseid2-0506.zip
...
17/38
Customized Procedure - pre processing» CourseCheck (GoF Strategy + GoF Composite Pattern)
ICourseCheck
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckAnnouncementString getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckCopyFlag
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckLink
String getMessage()
setMessage(String)
boolean hasitem(course)
Responsablility of each class: •checking for its own citerium•setting a MessageString if the criterium is met
CourseCheckDiscussionBoardString getMessage()
setMessage(String)
boolean hasitem(course)
18/38
BbList announcementList = AnnouncementDbLoader.Default.getInstance().loadByCourseId(course.getId());
if (announcementList.size() > 1) {
setMessage(course.getBatchUid()+ " has at least one Announcement");
return true;
}
setMessage(course.getBatchUid()+ " has one or less Announcements");
return false;
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
Get AnnouncementList from its loader
Check list size
19/38
If TOC has contentId, load its children with ContentDbLoader
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
ContentDbLoader contentLoader = ContentDbLoader.Default.getInstance();
BbList courseTocList = CourseTocDbLoader.Default.getInstance().loadByCourseId(course.getId());
for (Iterator i = courseTocList.iterator(); i.hasNext();) {
CourseToc courseToc = (CourseToc) i.next();
if (courseToc.getContentId().isSet()) {
courseToc.getContentId()).size());
if (contentLoader.loadChildren(courseToc.getContentId()).size() > 0) {
setMessage(course.getBatchUid()+ " has one or more Course Informations, Course Documents, Assignments or external links");
return true;
} } }
setMessage(course.getBatchUid()+ " has no Course Information, Course Documents, Assignments or external links");
return false;
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
Get the Table Of ContentsIterate Table Of Contents and check for valid ContendIds
20/38
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
BbList linkList = LinkDbLoader.Default.getInstance().loadByCourseId(course.getId());
if (linkList.size() > 0) {
setMessage(course.getBatchUid() + " has one or more links");
return true;
} else {
setMessage(course.getBatchUid() + " has no link");
return false; }
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckLink
String getMessage()
setMessage(String)
boolean hasitem(course)
Get LinkList from its loaderCheck list size
21/38
Iterate Conferences and check for fora
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
boolean hasDiscussionBoard=false;
BbList confManagers = ConferenceDbLoader.Default.getInstance().loadAllByCourseId(course.getId());
for (Iterator iter = confManagers.iterator(); iter.hasNext();) {Conference conf = (Conference) iter.next();BbList fora =
ForumDbLoader.Default.getInstance().loadByConferenceId(conf.getId()); if (fora.size()>0)hasDiscussionBoard=true;
}
if (hasDiscussionBoard) {setMessage(course.getBatchUid()+" has one or more DiscussionBoard");return true;
} else {setMessage(course.getBatchUid()+" has no DiscussionBoard");return false;
}
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckLink
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckDiscussionBoardString getMessage()
setMessage(String)
boolean hasitem(course)
BB6 Get ConferenceList from its loader
22/38
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
boolean hasDiscussionBoard=false;
List fManagers = DiscussionBoardManager.getForumManager().loadByCourseId(course.getId());
if (fManagers.size() > 0)hasDiscussionBoard=true;
if (hasDiscussionBoard) {
setMessage(course.getBatchUid()+" has one or more DiscussionBoard");
return true;
} else {
setMessage(course.getBatchUid()+" has no DiscussionBoard");
return false;
}
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckLink
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckDiscussionBoardString getMessage()
setMessage(String)
boolean hasitem(course)
BB7 Get ForumManagerlist from its
loaderCheck list size
23/38
Registry creg = CourseRegistryEntryDbLoader.Default.getInstance().loadRegistryByCourseId(course.getId());
Enumeration regenum = creg.entries();
while (regenum.hasMoreElements()) {
CourseRegistryEntry cregentry = (CourseRegistryEntry) regenum.nextElement();
if (cregentry.getKey().equals(copyFlag)) {setMessage(course.getBatchUid()+ " is already copied before");return true;}
}
setMessage(course.getBatchUid()+" is not copied before");return false;
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckLink
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckDiscussionBoardString getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckCopyFlag
String getMessage()
setMessage(String)
boolean hasitem(course)
Hidden API !Get CourseRegistry from its loaderIterate its entries and check for
key
24/38
Customized Procedure - pre processing» CourseCheck (GoF Strategy + GoF Composite Pattern)
ICourseCheck
String getMessage()
setMessage(String)
boolean hasitem(course)
AbstractCourseValidator
String getMessage()
setMessage(String)
boolean hasitem(course)
Add(ICourseCheck)
DestinationCourseValidatorboolean hasitem(course)
SourceCourseValidator
boolean hasitem(course)
Map<ICouseCheck>
Map with actual strategy in
implementing class
25/38
Customized Procedure - pre processingCourseCheckAnnounce
mentString getMessage()
setMessage(String)
boolean hasitem(course)
for (Iterator iter = validations.iterator(); iter.hasNext();) {
ICourseCheck cc = (ICourseCheck) iter.next();
if (cc.hasItem(c)) {
setMessage(cc.getMessage());
return true;
}
}
setMessage(c.getBatchUid()+" has no content items ");
return false;
CourseCheckContent
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckLink
String getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckDiscussionBoardString getMessage()
setMessage(String)
boolean hasitem(course)
CourseCheckCopyFlag
String getMessage()
setMessage(String)
boolean hasitem(course)
SourceCourseValidator
boolean hasitem(course)
public SourceCourseValidator() {
add(new CourseCheckAnnouncements());
add(new CourseCheckContent());
add(new CourseCheckDiscussionBoard());
add(new CourseCheckLink());
}
26/38
Customized Procedure - pre processingBufferedwriter awriter = new BufferedWriter(new FileWriter(CCIO.getArchiveFile()));
DestinationCourseValidator destCourseValidator = new DestinationCourseValidator(copyFlag);
SourceCourseValidator sourceCourseValidator = new SourceCourseValidator();
// if destination course has no copyflag
if (!(destCourseValidator.hasItem(trgCourse))) {
// if source course has contentif
((sourceCourseValidator.hasItem(srcCourse))) {
CourseDbPersister.Default.getInstance().deleteById(trgCourse.getId());
awriter.write(sourceCourseId + ", "+ archivePath + "\n");
}
}
if source has content if target existsif no copy flagremove targetcreate a/r filescreate symlink
CCIO : CourseCopy I/O
Bean (getters/setters) with all input, output and properties files
27/38
Customized Procedure – archive and restore
archiverestore
Archive / Restore
a-courseid1-0506, /path_to/archives/
a-courseid2-0506, /path_to/archives/
...a-courseid1-0607, /path_to/archives/ArchiveFile_a-courseid1-0506.zip
a-courseid2-0607, /path_to/archives/ArchiveFile_a-courseid2-0506.zip
...
28/38
Process archProcess = Runtime.getRuntime().exec( "/usr/local/blackboard/apps/content-exchange/bin/batch_ImportExport.sh -" + "f " + CCIO.getArchiveFile().getAbsolutePath()+ " -l 1 -t archive“);
InputStreamReader isr = new InputStreamReader(archProcess.getInputStream());BufferedReader br = new BufferedReader(isr);String line = null;
while ((line = br.readLine()) != null) {logger.info(line);
}
Customized Procedure – archive and restore
Execute the shell command with the Runtime classMonitor the process by reading its InputStreamManipulate logging here
29/38
Customized Procedure – post processing
add annoucements add title suffixset target unavailableunenroll studentsadd copyflagmove logs and archive
Post processing
a-courseid1-0506, /path_to/archives/
a-courseid2-0506, /path_to/archives/
...a-courseid1-0607, /path_to/archives/ArchiveFile_a-courseid1-0506.zip
a-courseid2-0607, /path_to/archives/ArchiveFile_a-courseid2-0506.zip
...
30/38
Post processing – add announcementAnnouncementDbPersister aPersister = (AnnouncementDbPersister) pManager.getPersister(AnnouncementDbPersister.TYPE);
CourseDbLoader cLoader = CourseDbLoader.Default.getInstance();UserDbLoader uLoader = UserDbLoader.Default.getInstance();idAboutCourse = cLoader.loadByBatchUid(sAboutCourse).getId();
String bodyText = MessageFormat.format(aBody, new Object[] { " <b><a href='" + bbPath+ idAboutCourse.toExternalString()+ "' target='blank'>" + sAboutCourse + "</a></b>" });
FormattedText body = new FormattedText(bodyText, FormattedText.Type.HTML);
Announcement a = new Announcement();
a.setIsPermanent(true);a.setRestrictionStartDate(calendar);
a.setType(Announcement.Type.COURSE);
a.setCourseId(cLoader.loadByBatchUid(sInCourse).getId());a.setTitle(aTitle);a.setBody(body);
a.setCreatorUserId(uLoader.loadByBatchUid(aPostedBy).getId());
aPersister.persist(a);
Get the appropriate loaders
Create HTML-formatted bodySet various annoucement properties
Save announcement
31/38
Post processing – add title suffix
if (copyProps.getProperty("doSetTitleSuffix").equalsIgnoreCase("Y")) {
if (!srcBBCourse.getTitle().contains(copyProps.getProperty("titleSuffix"))) {
srcBBCourse.setTitle(srcBBCourse.getTitle() + " " + copyProps.getProperty("titleSuffix"));
coursePersister.persist(srcBBCourse);
} else {logger.info("year suffix existed already in "+
srcBBCourse.getBatchUid());
}
} If specified in the properties, add titlesuffixIf the title doesn ‘t already contains suffix, add it
Save course
32/38
Set Enabled (admin apis only) Get appropriate loaders and persistersCheck whether the Course is an “Organization” or a “Course”
Use the correct loaderSave course
BbPersistenceManager pManager = BbServiceManager.getPersistenceService().getDbPersistenceManager();CourseSiteDbPersister cPersister = (CourseSiteDbPersister) pManager.getPersister(CourseSiteDbPersister.TYPE);CourseSiteDbLoader cLoader = (CourseSiteDbLoader)
if (trgBBCourse.getServiceLevelType().compareTo(ServiceLevel.DEFAULT) == 0) {
CourseSite trgDICourse = cLoader.load(trgBBCourse.getBatchUid());
if (copyProps.getProperty("doSetUnavailable").equalsIgnoreCase("Y")) {
trgDICourse.setIsAvailable(false);
trgDICourse.setRowStatus(IAdminObject.RowStatus.ENABLED);
cPersister.save(trgDICourse);}
....
Post processing – set course unavailalbe
Set unavailable (!= Disabled !!)
33/38
.....
pManager.getLoader(CourseSiteDbLoader.TYPE);EnrollmentLoader eLoader = EnrollmentLoader.Default.getInstance();EnrollmentPersister ePersister = EnrollmentPersister.Default.getInstance();
if (copyProps.getProperty("doUnenrollStudents").equalsIgnoreCase("Y")) {
BbList enrollmentList = new BbList();Enrollment tempEnrollment = new Enrollment();
tempEnrollment.setCourseSiteBatchUid(trgDICourse.getBatchUid());tempEnrollment.setRole(Enrollment.Role.STUDENT);
enrollmentList = eLoader.load(tempEnrollment);
for (Iterator i = enrollmentList.iterator(); i.hasNext();) {
Enrollment enrollment = (Enrollment) i.next();ePersister.remove(enrollment);
}}}
Post processing – unenroll students
Get appropriate loaders and persistersCreate an example enrollment with role == STUDENT and current courseid
Load all enrollments “by example”Remove selected enrollments
34/38
CourseRegistryEntry courseRegEntry = new CourseRegistryEntry(copyFlag, copyFlagValue);
courseRegEntry.setCourseId(trgBBCourse.getId());
CourseRegistryEntryDbPersister.Default.getInstance().persist(courseRegEntry);
Post processing – add copyflag
Create a new CoursRegistryEntry (String key, String value)Set the CourseIdSave the CourseRegistryEntry
35/38
ExtendedFile archivedFile = new ExtendedFile(srcPath);ExtendedFile reposFile = new ExtendedFile(reposPath+ archivedFile.getName());
if (!reposFile.exists()) reposFile.createNewFile();
if (archivedFile.renameTo(reposFile)) {logger.info("MOVE : " + archivedFile.getAbsolutePath() +
" --> "+reposFile.getAbsolutePath());} else {
logger.error("MOVE FAILED : "+ archivedFile.getAbsolutePath()+ " --> "+reposFile.getAbsolutePath());}
if (logFile.moveTo(reposLogFile)) {logger.info("MOVE : " + logFile.getAbsolutePath() + " --> "+ reposLogFile.getAbsolutePath());}
else {logger.error("MOVE FAILED : "+ logFile.getAbsolutePath() + " --> "+reposLogFile.getAbsolutePath());}
Post processing – move logs and archive
ExtendedFile
boolean copyTo(File)
boolean moveTo(File)
md5()
renameTo > ok for same filesystem
moveTo > bytestream copy for different filesystems
36/38
Current issues > future developments
Current issues: » In case of failure lots of manual setup work (due
to sequential batch structure) > use blackboard.platform.ApplicationLauncher.main ?
» Archive/restore phase relatively slow > Clone() ?» Fileserver crashes at large numbers > ?
37/38
Source Code
» http://perswww.kuleuven.be/~u0034877/coursecopy/coconut.zip
38/38
Questions – remarks – suggestions ?if source has content if target existsif no copy flagremove targetcreate a/r filescreate symlinkarchiverestore
add annoucements add title suffixset target unavailableunenroll studentsadd copyflagmove logs and archive
a-courseid1-0506, a-courseid1-0607
a-courseid2-0506, a-courseid2-0607
...
a-courseid1-0506, /path_to/archives/
a-courseid2-0506, /path_to/archives/
...a-courseid1-0607, /path_to/archives/ArchiveFile_a-courseid1-0506.zip
a-courseid2-0607, /path_to/archives/ArchiveFile_a-courseid2-0506.zip
...
mail (short) errorlog