This topic is for Blackboard Learn administrators who have implemented and maintain an automated batch integration using the Snapshot Tool in the Blackboard Learn product. The intent of this topic is to provide direction and guidance for updating the integration from using the command-line interface (CLI) Snapshot tools to the new SIS Integration Framework. Thus the primary focus of this document is on replicating as closely as possible existing CLI integration strategies using the SIS Integration framework.
The content of the topic is based on some important assumptions:
- The information only applies to Blackboard Learn 9.1 SP 8 or later.
- The information assumes you are moderately familiar and experienced with the CLI Snapshot tools and data management in Learn.
- The information assumes assumes you are using text-delimited, “flat” files for your Snapshot integration.
- Deployment assumptions are based on typical implementations facilitated by Blackboard Consulting and include certain effective practices, which may not be exactly the same for your deployment.
What’s not covered in this document?
- Information about implementing Real-Time Integration
- Detailed or extensive deployment guidance
- Training in the complete use of the SIS Integration Framework
- Details about new or extended features like:
How to manage course availability through “Terms.”
How to setup and use Institutional Hierarchy Nodes.
How to build and use custom field mapping scripts
A Note on Deprecation
In September 2012, Blackboard officially announced that the Command-Line Interface (CLI) Snapshot Tools had been deprecated and started encouraging customers to consider migrating existing integrations to the new SIS Integration Framework, introduced in the summer of 2011.
It is important to note that deprecation doesn’t mean unsupported or removed. The CLI Snapshot Tools continue to be supported interfaces for integration with Learn in 9.1 SP 10 and the foreseeable future. The CLI Snapshot tools will also receive ‘critical’ maintenance until they are removed from the product. As such Deprecation should be considered an announcement to begin the process of preparation for the end of something not an end in itself.
Specifically, deprecation is the announcement that a product or capability has been replaced or is no longer necessary and will be removed at some point in the future. This allows planning for change management. Additionally, announcing deprecation effectively ‘freezes’ development of the effected product – in the case of the snapshot command-line tools this means they will remain at release 9.1 SP 7 capabilities – and that the product will receive full support and only ‘critical’ maintenance until it is removed from the product.
Given that CLI Snapshot Tools are deprecated and ‘frozen’ at 9.1 SP 7 capabilities it is important to understand that this means that new platform features (for example, Terms, Course Merge, and Hierarchy) have not and will not be added to the capabilities of the CLI Snapshot.
When Blackboard determines the release in which Snapshot will be fully removed from the product, the customer community will be given extensive advanced notice so that you may plan your transition accordingly.
Official information and FAQ are presented on the Blackboard Support site in KB Article 43468.
How will deprecation affect my Snapshot integration?
Currently, not at all. However, the best practice of announcing deprecation allows you to begin to proactively assess your integration approach moving forward and identify the necessary changes you will need to make to support the new Framework Snapshot functionality. Note that the Framework Snapshot XML provides a path for those who currently use the XML file format in their Snapshot integration. It does not and will not support new capabilities post 9.1 SP 7.
Snapshot Integration History
For years, thousands of customers have used the command-line interface (CLI) called Snapshot to implement data integration with external systems and Blackboard Learn. Snapshot in the command-line interface context collectively refers to the command line tools called snapshot, snapshot_override, and dsm. These tools are to be replaced by the SIS Integration Framework Snapshot integration type.
With 9.1 SP 6, Blackboard enhanced the integration capabilities of the Learn platform substantially. The Student Information System (SIS) Integration Framework we’ve built is designed to provide, “out-of-the-box” tools and an extensible infrastructure to support robust, standards-based, flexible integrations with a variety of Student Information Systems. In summary, with the SIS Framework, administrators have GUI access to everything necessary to create, configure and monitor several simultaneous integrations, including:
- Custom mapping of fields from the SIS data to attributes in Learn
- GUI interface for manually uploading files on demand
- Web interface for automating file and event processing
- Access to detailed logs with variable logging verbosity settings
In SP 6, we included with the new framework three “integration types” based on IMS standard data formats. In SP 8 we added two more types based on the formats and capabilities of the old CLI Snapshot Tool. They are:
- Snapshot XML: Loosely based on IMS 1.0 with several Blackboard specific extensions.
- Snapshot Flat File: Character delimited text files. (This is the only type discussed below)
9.1 SP 6 and SP 8 also expanded Learn product features that have data integration implications. These features include the Institutional Hierarchy, Course Enrollment Merging, and Terms.
Given the complexities of working cooperatively in standards organizations like IMS, newer Learn product features may take extra time to fully implement using such open standards. Since Blackboard defines the Snapshot Flat File format, new Learn features can be immediately implemented using this well-known data format. The Snapshot Flat File format, being a simple text file, also provides the added advantage of being the easiest to create.
Four significant changes occurred between the new SIS Framework Snapshot Flat File and CLI Snapshot. Specifically:
- GUI-based Implementation Configuration and Management
- File Delivery
- Data Source Keys and Data Ownership
- Field Mapping and Scripting
Each of these must be taken into consideration as you move from CLI Snapshot to the SIS Framework Snapshot Flat File and will be covered in depth in the below Details section.
GUI-based Implementation Configuration and Management
Installing, configuring and managing SIS integrations are significantly easier when using the SIS Framework.
Past: With CLI Snapshot, all the configuration files, logs and the tools were only accessible at the command line – either on the application server or through the Snapshot client install. With the Framework, all interfaces for deployment, configuration and troubleshooting are now GUI based.
Present: Now the people who administer the integration don’t need command line access to the Learn servers to implement, execute or maintain the integration. This new configuration overcomes many constraints related to security, hosting server location, and the end-user skills that limited customer’s adoption of automated integration.
Providing data to Learn when using Snapshot is done via files. How this is done is different than with previous snapshot based integrations and will require some adjustment to existing processes.
Blackboard Consulting Controller: For customers who have been using the Blackboard Consulting Snapshot Controller, moving to the Framework will make that software obsolete. We will discuss how you replace it and the implications for ICM (maintenance services) in a later section.
Past: When using the CLI Snapshot, there were generally two steps involved in moving and processing the data files from the SIS. First, the files had to be moved to the Learn server - usually by FTP - then the Snapshot tool had to be called to process the files using JDBC or SOAP to submit the data to the Learn database.
Present: With the Framework, the data is introduced directly to Learn by POSTing files to defined, secure HTTPs “end points”. Files POSTed to Learn are immediately processed by the integration, so all automation of the integration process is external to the Learn server.
Monitoring of processing is possible via returned HTTP status codes indicating the success or failure of POSTing files, dataSetUID as provided in the response, and the use of the dataSetStatus endpoint for accessing detailed processing status based on the passed dataSetUID. See the Snapshot Flat File Automation knowledge path for more information.
Also important to note, the SIS Framework supports manual uploading of files too. Whereas with snapshot you had to run the command line, now a Learn administrator can simply upload a batch file in the GUI. Again, this gives control to GUI administrators who, due to security rules or system skills, may not have been able to execute one-off command-line runs.
Data Source Keys and Data Ownership
For more information and history about Data Source Keys, follow the Data Source Keys Overview knowledge path.
Past: The CLI Snapshot tools used the Data Source Key (DSK) attribute of records to group them together for certain bulk operations. In particular, when running CLI Snapshot in the SNPSHT mode, the DSK allowed you to implicitly disable records associated with a specific Data Source Key without disabling all the records in the system.
Present: The SIS Framework version of the SNPSHT mode is called “Complete Refresh”, and rather than using the Data Source Key to identify the group, it uses the integration deployment membership. This can be a difficult concept to understand, even if you’ve completely internalized how the DSKs work with the CLI Snapshot.
SP 12 adds the "Complete Refresh by Data Source" endpoint. This new endpoint more closely replicates the SNPSHT mode of the Command Line Snapshot. Complete Refresh by Data Source uses the data source assigned to the integration via configuration rather than using the integration deployment membership.
Field Mapping and Scripting
Previous snapshot integrations required that all data be prepared before creating the files used with the CLI. This required adjustment to SIS systems, middle-tier software, or pre-processing of the data to properly generate the snapshot files.
Past: With CLI Snapshot, virtually all of the business logic resided in the “generator” software, which extracted the data from the SIS. In order to make even small changes to the integration, Learn administrators would need to go back to the SIS programmers and request changes to the underlying queries.
Present: The SIS Framework provides the Blackboard administrator in-system tools to take control of the logic by enabling remapping fields and/or using simple, custom scripts through the admin GUI to provide the proper data to Learn.
When you transition from CLI Snapshot to Framework, you may not require this feature immediately, but you should build familiarity over time to make the most use of the capabilities later.
Review the version specific known issues documentation for up-to-date notes regarding product issues.
Summary of Significant Changes
SIS Framework is 100% GUI. No need for back-end access to implement or manage.
All configuration is completed via the GUI – the snapshot.properties file is no longer used.
Snapshot has not supported data elements and capabilities introduced in the Learn platform since 9.1 SP 6 (such as Terms, Course Merge, and Institutional Hierarchy).
HTTP POST delivers files to URLs on Learn secured with a username and password defined when setting up the Integration.
All scheduling and automation is external to Learn. (No more Bb Consulting Controller)
Posting of Files: cURL is one popular option for scripting procedures to POST files, but any technology that supports HTTPS POST of text files can be used.
Snapshot modes (SNAPSHT, MANUAL, REMOVE) have been replaced with Framework operations (“Full Replace”, “Store”, and “Delete”).
CRS_COPY_INTO operations have been incorporated into the standard processing of course data in the Framework. (Applies to Organizations too.)
(Complex) Whereas Snapshot used the Data Source Key to determine on which records to work, the Framework uses “membership” in the particular Integration deployment or the DSK depending on the selected operation. You may perform data update operations cross integration using Flat File and integration ownership will be conditionally updated - this is applicable to bi-laterally when using Flat File or LIS. Records owned by IMS Enterprise integration types may be updated to LIS or Flat File, but once changed may no longer be updated by IMS Enterprise integration types.
Field “Ownership” which was determined by the snapshot.properties file is now determined by a checkbox that controls whether the field should be subject to “Change on Update?”
File field headers are no longer hard-coded. You can name fields anything you want and map them in the GUI to the standard Learn fields. You can even add extra fields for use with custom scripting – giving you control of the business logic of the integration without (always) going back to the file generating scripts.
File footer is not utilized in the Snapshot Flat File format and will cause an error if included in the submitted data file. Integrity of the file transfer is based on file size as indicated in the HTTP header for the POST request.
You can run the Framework integration in “testing” mode, which will evaluate the data for errors, but not commit data/changes to Learn.
Making the Move
It is necessary to understand the differences between the CLI Snapshot and SIS Framework Snapshot in order to address changes required for a successful transition from CLI to Framework-based Snapshot.
Object Types and Files
All of the CLI Snapshot object types (files) are supported in the Snapshot Flat File type in the SIS Framework in Learn 9.1 SP 8 and later. Also, certain new features in Learn have added new data object types that are not supported by the CLI Snapshot but are supported by the Framework.
Not all the naming conventions are the same as CLI Snapshot nor are they entirely consistent within the SIS Framework interfaces. Most everything is obvious, but we’ve included some notes in this section to help navigate the differences.
For the Course Delivery license the supported Object Types are:
- Users (aka Person)
- Enrollments (aka Course Membership or just Membership)
The feature “Standards, Objectives and Goals” was only available to K-12 licenses before SP 8, but is now available to all customers. In the CLI Snapshot these were called Learning Standard Alignments.
The following object types have been added to Course Delivery and are not available to the CLI Snapshot tool:
With the Community Engagement license:
- Secondary Institution Role Associations (aka User Secondary Institution Role)
- Organization Enrollments
As you review the SIS Integration Framework interface, you should explore the features under Advanced Configuration to understand what the Header values should be and which attributes are required. Since this is all displayed in the GUI, the online product documentation doesn’t describe these items explicitly. (This is in contrast to the old Advanced Data Integration Guides that listed out all the fields for CLI Snapshot.)
For more detailed information on Headers and data see the Examples and Data Format knowledge paths.
Figure 1: Custom Field Headers and Field Mapping
By selecting the Field Mapping interface you can see the fields that are required by looking at the “Required for Insert” column. By selecting Custom Headers you can see the full list of default header values.
If you are very familiar with the CLI Snapshot formats you may realize that we have exposed even more attributes for the objects you can manage through the Framework. Some, but not all of them are related to the new features (such as Terms).
While it is also unlikely that you will use this feature immediately, the SIS Framework allows you do define the data header names to suit your needs. Not only can you rename an existing header attribute, but also you can add additional headers. You will be able to select these headers from the Source Field drop downs on the Field Mapping page and in custom scripting. For more detailed information on Field Mapping and Custom Scripting see the Examples knowledge path.
Setup of Snapshot Flat File
Detailed instructions and options for setting up the Snapshot Flat File integration type may be found in the Snapshot Flat File Configuration Knowledge Path.
Past: The CLI Snapshot uses the configuration file snapshot.properties to get a large amount of information about how it should operate.
Present: With the SIS Framework, virtually all of that information is now implemented through the GUI.
Figure 2: Portion of the Snapshot Flat File properties page. Accessible when creating a new integration or by selecting 'edit' from the integration's contextual menu.
As you can see in Figure 2, this properties page captures configuration information like the file delimiter and log verbosity that were in the snapshot.properties file. Other elements of snapshot.properties will be configured elsewhere – described in later sections.
You should also notice this is the interface that maintains the user/password for the integration; as well as controls the integration status. The status allows you to run your integration in testing mode, where you can see the results in the log without actually committing the data changes to Learn.
Snapshot Flat File Operation Modes
In this section we are going to expand on how the Framework relies less on the use of Data Source Keys for managing groups of data. The various modes for processing data relating those to known CLI modes and function are also covered.
Similar to the CLI Snapshot, the Framework allows you to execute the processing of data files in three modes:
Snapshot Flat File Data Ownership vs SNPSHT and Data Source Keys
While Data Source Keys are still important to the management of records in the system, they are not as critical to the operation of the Framework as they were with the CLI Snapshot.
The big change comes from the fact that the CLI Snapshot tool required the Data Source Key to do a SNAPSHT operation, because that's how it sets the boundaries of what was safe to disable from the system, since SNAPSHT operations implicitly delete all records from the data source that are not found in the feed.
With the analogous SIS Framework operation (Complete Refresh), the Data Source Key does NOT control the boundaries of what’s safe to disable, these boundaries are controlled by whether the record was created by a specific integration. We may state that a record is “owned” by the integration that created it.
In both cases, the reason for the safety boundaries is to prevent a SNAPSHT operation or a SIS Complete Refresh from deleting too many records, including potentially system-created user accounts and so forth. Note, it remains possible to run disables and purges in Learn by Data Source Key using the Data Sources Admin Panel tool.
Like with CLI Snapshot, the SIS Framework can change the DSK of a record. Ownership of the records is no longer a significant issue as one integration may alter data 'owned' by another - effectively changing the data ownership to the integration making the data change. This allows for more seamless data management across supported integration types. Note that records owned by IMS Enterprise integration types may be altered by other integration types but this is a one-way change and once altered the record may no longer be managed via the IMS Enterprise integration - this is due to limitations of the IMS Enterprise integration.
While record 'ownership' is still based on the integration, Complete Refresh by Data Source does allow control of data based on the data boundaries specified by a data source and thus operates only on the data associated with the data source assigned to the integration. Subsequently you may have a single integration that manages the data from multiple data sources, where the DSK is set in-data, and then have supporting integrations which operate only on sub-groups (based on data source keys configured on the integration) of that data. See Snapshot Flat File Examples for further explanation.
How to Use the New Methods
The implication for your CLI integration will depend on your use of multiple Data Source Keys and the SNAPSHT method. You will have to figure out the best configuration for your institution, but we will use a common deployment configuration as an example to help your planning.
Because it was one of the features of the Blackboard Consulting Snapshot Controller, it is common for customers to use the name of the data file to define the Data Source Key of the records. A typical set of integration files and corresponding DSKs might look like this:
Integration Files and Corresponding DSKs Data Type Contents Filename Users Active user accounts Users.txt Secondary Institution Roles 2ndary role Assignments 2ndary_roles.txt Courses Fall 2012 Courses FA2012_CRS.txt Student Enrollments Fall 2012 Enrollments FA2012_ENR.txt Instructor Assignments Fall 2012 Instructor Assignments FA2012_STAFF.txt Courses Winter 2013 Courses WI2013_CRS.txt Student Enrollments Winter 2013 Enrollments WI2013_ENR.txt Instructor Assignments Winter 2013 Instructor Assignments WI2013_STAFF.txt
Table 1: Example DSKs and Files
If you want to use the “Full Replace” method in the SIS Framework, you will need to define separate integration deployments that segregate groups of data of the same type. With the example above, we would recommend that you deploy three integrations - one for the data types that are not term based and one for all the files that belong to a particular term. The result might look like this:
Figure 3: Example Integration Deployments
This would allow you execute a “Full Replace” on the Fall 2012 enrollment records without inadvertently disabling any other enrollments on the system. And note, do to this, you don’t NEED to have separate integrations for every file.
Data/Field “Ownership” Control
The CLI Snapshot used properties in the snapshot.properties files to identify the attributes in Learn that would be controlled or “owned” by Learn. By listing an attribute, the CLI Snapshot would ignore that field when performing data updates. Hence, users could change these values in the Learn GUI and not have the integration process overwrite them.
In the Framework, this same control is maintained at the attribute level by a checkbox in the Field Mapping interface. In the capture below, the green arrow indicates the column “Change on Update?” When the box is checked, the values in the file will overwrite whatever is in Learn. With the box unchecked, the attribute will be populated during an INSERT, but ignored for all subsequent UPDATEs.
Figure 4: Ownership Checkbox
For example: Let’s assume that you pre-populate Learn with users’ passwords based on some known convention. Your data file includes this initial value for each user. However, you encourage users to change their password when they first log-in and give them the option to change their password whenever they want. If you un-check the box, subsequent runs of the integration will NOT reset the password.
Course Copy/Template Changes
The new framework supports the use of Course Copy and Course Templates just as does CLI. The key difference is operational and configuration.
Past: Course Copy configuration is done using the snapshot.properties file. Course copies (COPYINTO) are done using the CLI tools CRS_SNPSHT or CRS_MANUAL with the –t option and the TEMPLATE_COURSE_KEY field in the feed file.
COPYINTO operations use the Template_Course_Key to specify the source of the copy operation based on the ownership setting in the snapshot.properties file.
Present: Course Copy configuration is done using the SIS Framework admin GUI. Course Copies are completed as part of the processing of STORE or FULL REPLACE requests. The feed files must contain the TEMPLATE_COURSE_KEY (Source Course Copy Key in the UI).
In all cases using the COPYINTO is only meant for populating new Courses. Using this method with existing Courses will not copy the data. Settings for Course Copy are found in the Advanced Configuration Settings for the integration under Courses:Course Copy Settings.
For more information on Course COPYINTO and using Templates see the Examples knowledge path.
As discussed, the SIS Framework includes HTTP “end points” which are the programmatic interfaces for delivering the data files to the Learn system. To automate batch integration, you will need to establish your own method for securely posting your files to these endpoints that does not require human intervention. Customers are encouraged to work with technology that is familiar and easily supported at your institution.
Also, you will have to decide the exact location from which the files are going to be posted. Consider both the security and stability of the system(s) to ensure that your integration runs and that no unauthorized people have access to the data.
Figure 5: Shows the endpoints for a fictitious integration. Notice how each entity/mode has a unique URL. Also, it is important to recognize that it is the integration Username that distinctly identifies this integration from others. All integrations of the type Snapshot Flat File use the same URLs – the username determines which integration is executed.
Figure 5: HTTP End Points
Automation Scripting: A cURL Example
To give you an example of how this can be done, we are providing some instructions for how to use cURL, a command line tool for transferring data with URL syntax. You can download the version of cURL you want here: http://curl.haxx.se/download.html. Make sure to use a version that supports SSL.<p class="note>The following example is for a Windows system. cURL must be downloaded and installed. The appropriate binary may be acquired at the curl project website: <a class=" external"="" data-cke-saved-href="http://curl.haxx.se/latest.cgi?curl=win64-ssl-sspi" href="http://curl.haxx.se/latest.cgi?curl=win64-ssl-sspi">http://curl.haxx.se/latest.cgi?curl=win64-ssl-sspi
Linux systems may or may not require installation of cURL. Downloads are available from the curl project website: http://curl.haxx.se/download.html
Select the latest stable version (highlighted) for your version of Linux.
See the Snapshot Flat File Automation Knowledge Path for more information regarding automating a Snapshot integration type.
With the curl.exe file on the PC, we execute a post using the following command syntax:
curl -k -H "Content-Type:text/plain" -u 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/webapps/...t/course/store --data-bin @courses_fall2012.txt
The syntax is explained below:
At it's most simple, you can build a batch or shell script file that contains the commands you want to execute. Make sure you follow the logical order of creating data objects before you try to assign them to something. (For example, run your Users and Courses files before you run Enrollments).
This is the most basic example of a batch file that would execute the upload of three files. The assumption is that you would name this something like integration_post.bat and then create a scheduled job to run automatically at times when you know the files will be there. Clearly, you can be more sophisticated in your scripting and scheduling based on your skills and requirements. For shell script (Linux) and other automation strategies see the Examples knowledge path.
- “Store”: performs explicit add and update operations on the data records that appear in the file. This is analogous to the CLI Snapshot mode MANUAL.
- “Full Replace”: performs add and update operations on the data records in the file AND it implicitly will execute a remove operation on records that are not in the file but are members of the integration. This is like the SNPSHT mode with the CLI.
- “Delete”: executes explicit remove operations on the records included in the file. This is just like CLI Snapshot REMOVE mode.
- -k = allow connection without verifying the SSL certificate.
- -H = determines this a required HTTP header line for Learn
- "Content-Type:text/plain" tells Learn that we are sending a plain text file.
- If you were using XML, it would be "Content-Type:text/xml"
- -u = username and password for authentication.
- 2d2f8833-90b7-444e-8bea-514111d5be70:password is from our setup example (see Figure 1)
- --url = the page to which you will post the file.
- End-point URLs are listed in Learn on the “HTTP Information” page for all integrations. For flat files this is specific to the Data Type (users, course, enrollments, etc.) and mode (store, complete refresh or delete) we want the data
- --data-binary = HTTP post of binary data
- @courses_fall2012.ext = the path to the file that is being sent to Learn. In this example the command is being run while in the directory where the file is located.
REM.-- Simple script to run three integration files to Learn
REM. --USERS posted in STORE mode
curl -k -H "Content-Type:text/plain" -U 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/person/store --data-bin @users.txt
REM. --COURSES posted in STORE mode
curl -k -H "Content-Type:text/plain" -U 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/course/store --data-bin @courses_fall2012.txt
REM. --ENROLLMENTS posted in STORE mode
curl -k -H "Content-Type:text/plain" -U 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/membership/store --data-bin @enrollments_fall2012.txt
The cURL commands will return standard HTTP status codes from Learn as an indication of the success or failure of the file post - specifically whether the endpoint is valid. Code 200 is the only one that indicates a fully successful file post. You may reference http://en.wikipedia.org/wiki/List_of_HTTP_status_codes for a list of HTTP status codes.
The successful post of the file does not indicate the successful processing of the data. For that you will have to examine the log entries on the server – in the GUI or use the reference code contained in the POST response in a call to the dataSetStatus endpoint.
Contained in the POST result is a reference code (dataSetUid) which may be passed to the integration dataSetStatus endpoint to receive an xml format result indicating the current status of the processing of the data set. More information on using dataSetStatus in monitoring your integration is available via the Snapshot Flat File Automation knowledge path.