Skip to main content
Blackboard Help

Snapshot Flat File Automation

An integral aspect of a trouble free SIS integration is the ability to automate and monitor the flow of the data to Learn as presented by the integration. Learn 9.1 SP12 provides two new capabilities that facilitate this process:

1. Results from integration POST URLs contain a data set process uid. For example:

Success: Feed File Uploaded. Use the reference code afc3d6e84df84f51944a06cccee8f59a to track these records in the logs.

2. SP12 includes a new Data Set Status URL which when contacted with the data set process id returns an xml block containing specifics regarding the data set status - note this may be called well after completion of the original POSTing of the data file.

https:// ... /webapps/bb-data-integration-flatfile-BBLEARN/endpoint/dataSetStatus/afc3d6e84df84f51944a06cccee8f59a

which returns something like:

    <dataIntegrationId type="blackboard.platform.dataintegration.DataIntegration">_123_1</dataIntegrationId>

These additions facilitate scripted monitoring of automated snapshot flat file integrations.

This topic provides an overview of how to automate and monitor a Snapshot Flat File integration. The examples provided are written for a Linux/Oracle platform using CRON for automation and the Unix shell scripting language BASH, but the concepts may be applied to any language which is capable of POST/GET operations, parsing of strings and dates, performing database queries, and sending email. Thus PERL, JAVA, PHP, Ruby are suitable for development as are other shell or batch languages.

The goal is to provide a functional reference implementation for developing a monitored and automated SIS integration using Snapshot Flat File. This will contain two components: automation and monitoring. The reference implementation is available via the link at the bottom of this page.

Inclusion of logged error messages via the presented reference implementation is currently only compatible with self-hosted systems. This capability is outlined in the script and requires uncommenting code for use in self-hosted environments. Managed Hosting clients will continue to rely on the Learn SIS Framework logging interface to access error messages. A future improvement to this document will provide a means of 'in-script' access to these log messages.

Use Case: Automating SIS Flat File processing

The following use case serves as the basis for the reference implementation and as a guide for your own development:


Automate the processing of SIS generated Snapshot Flat File data and email reports to specified administrators. The solution should periodically determine the presence of new data files, the object and data source, and present the data to Learn. Post-processing files should be placed in a separate directory for archival purposes. The solution also should provide the ability to manually process data files. In all cases, status emails on failure or success should be sent to administrators. The email should contain available data regarding the process and any error messages.


SIS, Operating System scheduler, Learn


SIS generates and stores in a specified directory&n Snapshot Flat File formatted text files for the Learn objects of Users, Courses, Enrollments, and Staff Assignments.


  1. The SIS provides snapshot flat files to a script specified data directory
  2. The operating system scheduler (CRON) starts the sis_snpshtFF_auto script
  3. The sis_snpshtFF_auto script checks the data directory for the presence of any files
  4. The sis_snpshtFF_auto script determines the Learn object type of each file
  5. The sis_snpshtFF_auto script calls the sis_snpshtFF_manual script accordingly and in the object hierarchy - Users, Courses, Memberships
  6. The sis_snpshtFF_manual script POSTs the data to Learn, determines completion state
  7. If there are errors and configured to do so sis_snpshtFF_manual queries the integration logs for error messages
  8. The sis_snpshtFF_manual script constructs an email containing status information and emails it to configured email addresses
  9. Steps 5-8 are repeated for each data file.
  10. On completing processing of all data files the sis_snpshtFF_auto script sends a status email to configured email addresses


The data provided by the SIS generated flat files is sequentially by data object added to Learn and the original files are archived with the processing timestamp added to the original filename. Script configured administrators receive status emails per processed file and an overall status email covering the full processing task.

The Solution

Setting aside the generation of the data files by the SIS as that is out of scope for this document - there are three components to the automation problem - determining WHEN to run the processing of the provided data, WHAT data objects and data sources that provided data covers, and HOW or the processing and monitoring of that data. Using  the above use case we can build a set of configurable scripts which can determine if a data file exists (should this task do anything), what object type that data applies to, call a processing script with the appropriate parameters to meet the integration goals, process the data, and on completion of processing archive the data file. First, we will cover the WHAT, then HOW, and finally the WHEN. Putting all three in place will provide an automated process (that you can monitor) for moving SIS data into Learn via the Snapshot Flat File integration Type.

The BASH script is heavily commented so will not be provided here rather let's focus on the overall flow and processing details.

The HOW: Using and sis_snpshtFF_manual scripts

Based on the above use case the bash script below ( performs the following operations:

  • Checks if there are files in the specified directory
  • Determines the object type and operation based on the header information in the file
  • Orders the processing so that files are processed in the correct order. For example: Users then Courses then Memberships.
  • Calls a sub-script ( for processing, monitoring, and admin email notification of the processing status
  • Archives the data file on completion of processing
  • Processes the next data file if it exists.
  • Mails a report indicating the cumulative result of the call to the automation script.

This provides the following general flow shown in Figure 1:


Figure 1: General flow of automating Snapshot Flat File processing as demonstrated in the provided reference solution.

Further breakdown of the scripted portion of the process we have two scripts (represented on the left half of Figure 2.) and (represented on the right half of Figure 2.


Figure 2.  Detailed workflow

The comments in each script comprising the automation reference implementation describe in detail the operations sketched out in Figure 2. 


At a high level, the sis_snpsht_auto script loads files found in the configured data directory and determines by analysis of the file data header the object type referenced and adds the file to the appropriate list for later processing.  This analysis of the headers determines the type of object the file is referring to and thus it's order in the snapshot processing queue. This sorting based on the header allows for a single drop point for the SIS generated flat files.

After all of the files have been analyzed and are grouped into lists of their object type the lists are processed in logic order of Users, Courses, Memberships by handing off each file to the sis_snpshtFF_manual script for processing (which may be run singly from the command-line btw) along with the appropriate arguments for each object type. The sis_snpshtFF_manual script takes the incoming arguments and after a sanity check for operation uses the appropriate url to POST the data file to Learn. After posting the script enters a monitoring loop and on completion builds a report, emails it to the configured list of recipients and then returns to the sis_snpsht_auto script which sends the next file for processing - this is repeated until all files have been processed. After all files have  been processed sis_snpshtFF_auto emails a final report to the list of configured recipients.

The WHEN: Using CRON to schedule snapshots

The heart of the automation is the ability to run the script set w/o human intervention. Unix provides this capability with CRON - a built-in scheduling application. A system process, CRON, periodically checks the system crontab - a system file that contains a list of commands and settings for when they should run - and contained entries are then assessed for whether they should be run now or skipped until later as indicated in the crontab entry.

The frequency of cron jobs and processing times for operations should be considered in setting crontab entries, as REFRESH operations may take longer than STORE operations. Processing of REFRESH vs STORE data may be handled via separate crontab entries and separate data source directories in the script arguments if using the script provided below.

CRON Expressions for Crontab Entries

CRON utilizes a very specific format for scheduling data - it relies on a space separated list of five required data fields:

Field Description Allowed Value
MIN Minute field 0 to 59
HOUR Hour field 0 to 23
DOM Day of Month 1-31
MON Month field 1-12
DOW Day Of Week 0-6
CMD Command Any command to be executed.

In practice, this format may be applied as simply or as complex you choose.

Simple Examples:

An example of a simple crontab entry would be to run a task at the top of every hour:

0 * * * * /usr/local/blackboard/apps/snapshot/scripts/

  • 0 – 0th Minute
  • * – Every Hour
  • * – Every Day
  • – Every Month
  • * – Every day of the week

or once at midnight every day:

0 0 * * * /usr/local/blackboard/apps/snapshot/scripts/

  • 0 – 0th Minute
  • 0 – Every Hour
  • * – Every Day
  • – Every Month
  • * – Every day of the week
Complex Examples:

Run a task twice a day during the work week:

0 11,16 * * 1-5 /usr/local/blackboard/apps/snapshot/scripts/

  • 0 – 0th Minute (Top of the hour)
  • 11,16 – 11 AM and 4 PM
  • * – Every day
  • * – Every month
  • 1-5 – Monday through Friday

or every two hours Monday through Friday

0 */2 * * Mon-Fri /usr/local/blackboard/apps/snapshot/scripts/

  • 0 – 0th Minute (Top of the hour)
  • */2 – Every even hour 12, 2, 4, 6, 8, 10, 12, 14, etc. or every other hour.
  • * – Every day
  • * – Every month
  • Mon-Fri – Monday through Friday

A CRON Reference:

You may also view the man page for your system via the command line:

$ man 5 crontab

Adding a crontab

Using the above cron settings we can add a crontab entry for scheduling when the snapshot task will run.

To edit your root crontab file, as the root user type the following command at the UNIX/Linux shell prompt:

$ crontab -e

Note that -e drops you into a vi editor.

To run the Flat File automated processing script at midnight every day add the following to the list of tasks:

root@dev$ crontab -e

type i - this places you into insert edit mode

type the following line

0 0 * * * /usr/local/blackboard/apps/snapshot/scripts/

press escape - this exits edit mode

type :wq - this saves your edit and quits the editor


The following people contributed ideas, input, and suggestions for the BASH reference implementation:

Kelt Dockins contributed his BASH-based implementation for Snapshot Flat File content analysis.

Ross Brown and Jerald Todd caught some early issues exposed in the first version of the reference implementation.


This downloadable SIS_SnpshtFF_Bash_Scripts archive (zip) contains functional and commented code demonstrating the concepts presented in this topic.