Media Platform Upgrade In Progress

Beginning November 21, 2014, we started a major upgrade of our media server platform. The upgrade is a significant enhancement involving both our streaming media encoder farm in our production offices and all of our media servers in our data centers. The upgrade will be done in phases during the remainder of 2014 (basically, the last six weeks of the year).

This upgrade will be challenging because the site is active and in continuous 24x7 service. We are challenged to perform such a significant upgrade WHILE the site is being used by visitors and students. In analogy, this is like changing the engine on an airliner while it is in flight! We have designed the upgrade in a way that prioritizes the work so as to be as minimally disruptive as possible.

We are doing this upgrade at this time because the late fall period, just before and during Advent, is one of our two least-busy times of the year with the fewest people online each day.

Phase 1, the switch-over to the new server platform itself, began on Friday, November 21 and should be completed by Monday, November 24. Phase 1 is the biggest single step and will be the only phase that is service-impacting and may cause disruptions of service on some video objects. During this Phase 1 upgrade window, some of the videos shown on the website may not play until updated and re-encoded.

By open of business Monday, November 24, all of the high-impact work should be done, and 80% of the currently-in-service public portion of the site should be functioning on the new platform.

We will be working progressively, first through all of active courses and then the public website to minimize the impact and disruptions to as many of our visitors as possible. We will then work progressively through the full archive of all community and event videos and then the list of currently-inactive courses (so they are ready for their next offering cycle.)

It is possible that you may NOT experience ANY disruption of service, depending on your video viewing habits because we are going to do the most-viewed active videos first. However, if you do experience a video that does not play, we're sorry and please know that we are working three shifts a day until Phase 1 has been completed in order to minimize any impact of this upgrade. Any disruption should only last hours in the worst case.

While we are proud of our legacy media platform (the one we are replacing) as one of the state-of-the-art systems in the industry, we have used this version of our technology for approximately the last two years. Like all technology, improvements occur in the state-of-the-art that require upgrades. Sometimes these updates require significant changes ("a folk-lift upgrade") instead of incremental changes. This upgrade is one of the major ones that will require replacing major subsystems in our data centers. This upgrade will allow us to implement new functions, gain even higher quality of our videos and reduce our operational costs for both video storage and video bandwidth. We think you will find the upgrade worth the effort in the resulting better-quality videos, particularly for users with slower connection speeds.

This upgrade includes using a new video encoder that produces higher-quality results PLUS smaller file sizes; all of the things that we look for in a video encoder. We are still using the H.264 CODEC, but one that has been implemented in an encoder from another vendor that is much more efficient and results in better video quality. To fully benefit from this technical upgrade, this means that we must re-encode ALL of our videos using this new encoder. The video encoder upgrade will be the bulk of the work in Phase 1.

In Phase 1, we will encode new very-high bit-rate digital master files of ALL of our videos using all of the advantages of the new encoding technology in our new mastering process. Yes, this mean we must re-master every video in every course, every community and all public area of the website. Next, we will use the new master files to encode (using our new production-files encoder) each master file into all of the Adaptive Streaming sets and the progressive download versions of our videos. Each of these steps is very compute-intensive (lots of computer processing time) and requires time to perform the work on each and every video in our website. Also, the resulting video-data files are very large and, when finished, require extended time to upload to the streaming platform. For every hour of finished video product, the mastering, production encoding and uploading of the finished product to the media servers will collectively take about eight hours of work to perform.

Once the new version of the videos have been re-encoded and uploaded, we will then have to edit each web page that contains a video so the embedded media player points to the new media server and the specific files that need to be played. Once this editing has been completed for a page, the video service for that page will be restored, and the player will play the new video file with all of its improved video quality.

In Phase  1, we are working three shifts (24-hours) per day and using multiple encoding computers so we can complete this work as quickly as possible. However, it is a rather massive amount of work and will require some weeks to complete the work on the entire site.

Phases 2 and 3 upgrades will be performed beginning as soon as we finish Phase 1 (likely beginning in the first quarter of 2015) and will build upon the foundation achieved in Phase 1. We do not expect Phase 2 and 3 to have any service impact (no disruptions) other than you will simply beginning seeing better videos with more features.

Running update reports of the status of our upgrade work will be provided below. All dates and times mentioned in my reports will be based on the Central time zone.

First Step of a Bigger Overall Plan

This upgrade moves us off our existing media servers. When this step has been finished, we will be operating entirely on our new media platform. The completion of this step will provide more capabilities than we have had before.

However, to fully exploit our new capabilities, we will have to perform far more extensive work to employ all of the new features and functions that will require extensive software development. We will begin this work in the next several weeks and expect that it will take several months.

Also, in the next steps, we will be changing our transport protocol to "HTTP Live Streaming" (HLS). At that time, we will also be migrating to a new cloud-based encoding service. To fully exploit HLS encoding in the cloud, we will take the opportunity to remaster all of our videos to DNxHD intermediate masters using 10-bit samples. Our overall objective is to improve further our already-high quality standard.

Our current plan is to complete the new development and migrate all of the media to the new platform during the summer lull in July 2015. Just like the Advent season, the middle of summer is our second period of lowest student activity. We often use the summer lull for major upgrades.

When completed, we will have many exciting new features, functions and capacities.

Updates

Update - Upgrading of all videos for all currently-active courses was completed and made accessible by 12:00am (midnight) Saturday, November 22. All active students had minimal disruptions with the work performed Friday night and Saturday when class work is typically not attempted by students (the period of least activity.) This part of the work went very smoothly and significantly ahead of schedule. The worked targeted for completion by Monday morning was actually finished by Saturday night. Work on other portions of the website continues but will take several weeks or so to complete. In order to be least disruptive to our active students, the videos of our active courses were our highest priority and the upgrade went very smoothly.

Update - On Sunday and Monday, most of the public videos used in the public portion of the website were upgraded and all upgrades were made accessible by 5:00pm Monday, November 24th. The remainder of the public videos should be completed by Tuesday and work on the community videos will begin in earnest. As a start on the community videos, the ten (10) most-recent episodes of Chuck Knows Church were upgraded and re-uploaded. Work continues. We did a count of the inventory of all remaining videos requiring upgrading and found that over 250 separate videos that are currently active in all portions of the website that still need to be re-encoded with our upgraded encoders and uploaded. This work will continue tomorrow.

Update - As of 6:00pm Wednesday, November 26th, all public videos on the public portion of the website have been upgraded. We have just started work on the videos contained in our Member and Special-Interest Communities. This work has only just started and there is a lot left to go. However, the entire catalog of back episodes of Chuck Knows Church have been re-encoded, including the first 40 episodes that we have not carried to date. The Chuck Knows Church pages will be edited to include the re-encoded videos over the holiday as time allows.

Update - We will be observing the Thanksgiving holiday for the most part but will be doing some work on Saturday and at other times during the holiday as circumstances allow. We are leaving for the holiday with the upgrade well in hand. There is considerable amount of work remaining to be done but most of the remaining videos requiring work are ones that are viewed less frequently and therefore have less impact while offline. Happy Thanksgiving to all and we'll be back on Monday.

Update - As of 7:30pm Tuesday, December 2nd, the videos for the entire Chuck Knows Church catalog has now been posted and made accessible. This includes episodes we have previously carried PLUS all other episodes and specials published by Chuck! We now carry the entire Chuck catalog, almost 100 unique videos (up from 40 videos when the upgrade began.) BTW, thanks to the team who gave up their Thanksgiving holiday to get so much of the re-encoding complete!

Update - As of 4:00pm Wednesday, December 3rd, the videos from all of our communities have been upgraded, re-encoded with our new CODEC, uploaded to the new media platform and their carrier pages updated. With the work completed on our active courses, our public site and our communities, the only work that remains are the videos carried in our currently-inactive courses and therefore are classes with no active students in their virtual classrooms. In other words, all videos currently accessible by students, public visitors and private users should be working normally after the upgrade. The only remaining work to be done is the videos that are part of courses that are currently inactive and therefore currently without students who need to view the videos. The remaining work on the currently-inactive courses will continue and we will complete all of the work on the videos for the currently-inactive courses well before the courses become active again; for the most part, in the next couple of weeks. In the totality of this upgrade, when finished, we will have processed over 700 video sets (each with 6-8 Adaptive-Streaming file versions in each set) and it is possible that in our hast, we might have bungled something just because of the sheer volume of this work. While we will be doing quality-assurance work in the coming week to proactively find and correct any errors, please feel free to report any video that does not play correctly. If you do, please include the URL of the page where the error occurs, We will make any required corrections immediately.

Update - As of 5:00pm, Friday, December 5th, all of the audios and videos for the following currently-inactive courses have been upgraded:

• Event 001
• RL-03
• BB-03

Work on the other courses continues.

Update - As of 6:00pm, Saturday, December 6th, all of the videos for the following currently-inactive courses have been upgraded:

• BB-02
• DH-01

Work on the other courses continues.

Update - As of 3:00pm, Monday, December 8th, all of the videos for the following currently-inactive courses have been upgraded:

• DH-01b
• RL-02
• RL-02c

Work on the other courses continues.

Update - As of 6:00pm, Tuesday, December 9th, all of the videos for the following currently-inactive courses have been upgraded:

• RL-01

As of 6:00pm, Wednesday, December 10th, the Phase 1 upgrade has been finished, checked and all new data archived. We do anticipate doing the upgrades for the TCP courses and should know shortly about scheduling. In the interim, the team is going to take a couple of days off to rest and recharge. They have worked three shifts per day, seven days a week since November 21, so they deserve a break until Monday, December 15. The only day in this period we work wasn't being done was Thanksgiving Day. I have checked the log files, and it looks like we completed this upgrade without any interruption to any registered Member or Student. Not a single student requested a video that wasn't ready when they click "Play." This was our goal. Plus, there were only very minor interruptions to our public visitors. I have only been able to find a few instances where a public video was requested that we haven't gotten around to re-encoding when a video was requested by a visitor. My thanks to a great technical team and their efforts!!!