The requirement to move content from an external system into Jive SBS is very common. You have discussion forum content and blog content in whatever system (Wordpress, Jive Integrated, Teligent, Dot Net Nuke, PHPBB, etc.) and you need to get it into Jive SBS. And then complicating the situation further is that Jive SBS may be in a hosted environment where firewalls will prevent any form of external database connections.

While working with Quest Software, I started the process of doing migrations about one year ago. Prior to speaking with Jive Professional Services, we looked mostly at three options.

1. Use webservices such as REST and SOAP to push in the content.

2. Transfer the content from the originating database into XML. Place the files in a location accessible to Jive SBS such as an external webserver. Then transfer the content in using a custom plugin to parse the XML and use POJOs to push in content into Jive SBS using the various managers made accessible in Jive.

3. Use a plugin and establish a JDBC connection from the old system and the new Jive SBS database. Use application logic in the plugin to prepare the content for entry into the Jive SBS database.

My first experiments were with option 1. There were questions of whether or not one administrator could push in content for any user. It did appear to work successfully – and that was without rewriting an authentication filter for the webservices. Nevertheless, the slowness of the webservices appeared to be a serious issue. Each call to the REST api took more than a second, and if we were to migrate a very large forum, that would mean the process may need to run for a very long time. It also had the problem of sending out emails to subscribers. Once having an opportunity to discuss migrations with Jive Professional Services, we all agreed that this was not the best approach.

Option 3 was immediately tossed out as an option as Jive SBS was in a hosted environment and firewalls prevented any sort of JDBC connections between the two systems. Furthermore, it is not recommended to do straight dumps into the database of Jive SBS as certain meta-data may not be properly pushed in and the content may not be checked for validity and could cause serious issues in the application.

Option 2 was the perfect solution and in my assessment, even the best solution even when the environment is not hosted. By exporting content into XML, content could be safely and easily smoothed prior to the push. By using the managers made available by jive accessible in doing plugin development, all data can be pushed in using the same methods as used by someone posting direct in the application.

I proceeded and wrote a migration tool to push in wordpress blog content. I completed the blog migration tool and successfully used the tool. Afterwards, I was 2/3 complete with adding functionality to push in discussion forums when a decision from management was made to engage Jive Professional Services to build a reusable utility for migrating discussion forums. Since we had about 30 forums to migrate, management felt the engagement was justified. The cost was $60,000. I frankly opposed the expense as being unnecessary.

I will now proceed to describe both tools and the differences between the two tools.

Tool 1 (Brian’s Solution):
A console java application is used to push all content into multiple files.

ids.html – a plain text html file with a line by line list of all thread ids from the external system was created as a master file.
So for this simple example, the ids.html file would simply look like:

24
25
29

Then (a thread id).xml file for each thread (or blog post id) would be created to hold the content of each thread or blog post including comments or follow-on treaded discussion messages for the thread.

Example file names would be:

24.xml
25.xml
29.xml

By using this approach there would never be any scalability issues as no XML file would ever be too large for the plugin to handle.

The files would all get placed onto a webserver where a plugin on JiveSBS could reach into them over the web and stream in the content.

The plugin would grab the accessible managers that are available to all plugin developers and push in the content one message at a time.

Images and attachments would be streamed in. This is done by parsing out source information in the content and reading in the binary stream over the internet and saving it to the system.

A dictionary is established to record all operations. The dictionary is used to create 301 redirects after the fact in an external redirects server. The application could have gone one step further with the dictionary and used to rollback the content in the event of a failure anywhere in the process.

There was one downside to the application which I never resolved was that the smtp mail server would need to be renamed to a non-existent location while the process ran. Not doing so would result in spamming the end users with migration related emails. On the down side, this also meant that users would not receive emails during the time of the migration. Otherwise, the process worked smoothly and quickly. Migrations could be done in quick bursts preferably during non peak hours and not take much time at all.

Tool 2 (Jive PS Solution):
Jive Professional Services put this application together using preexisting code long standing on the Jive team. Thier solution used a groovy script to generate xml files that were placed in a handful of subfolders.

/users/users.xml
/community/community.xml
/msg/msg.xml
/msgcontent/msgcontent.xml
/attachments/ – a list of xml files specially named including the messageid with the binary value of the attachment stored in the <data> section of the xml.
/images/ – a list of xml files specially named including the messageid with the binary value of the image stored in the <data> section of the xml.

The files would need to be placed physically on the server with Jive on all nodes of the operating systems, and application logic from the plugin would pull in the content and push it in. The dictionary was used also in this process for the later construction of 301 redirects.

There were some major “gotcha’s” with the application.

1. Once the plugin is installed, the entire Jive SBS system is placed in read only mode resulting in downtime for the application – something that I believe is very undesireable for an external facing enterprise application. A maintenance page is best put up during this time to avoid end user confusion. Without the maintenance page, users can see content but when they attempt to post content, they recieve a frustrating error message.

2. Each individual subspace has to run as a separate batch. Following the run, the application is required to go through the app restart process which takes ten minutes each time in the hosted environment.

3. Jive Hosting support is required during the migration times for placing files on the Jive server.

4. The application is very finiky and easily breaks over unexpected and difficult to troubleshoot issues usually regarding attachments.

With these constraints, all migrations must be done on Saturdays. And some migrations may take even 8 hours to complete depending on how many subspaces are touched in the process.

My end assessment regarding the Jive migration tool from Jive Professional Services is as follows:
If you are okay with spending $60,000 for a tool that requires significant amounts of down time – then go to Jive Professional Services for your migration tool. The tool will work for you if you do.

Optionally, you can spend significantly less money, and I could build the tool for you which is equally as effective and requires no more downtime than one or two cycles of application restart for plugin installation and plugin unistallation when complete.

So there you have it – the big picture of two workable systems used to migrate content into Jive SBS.