February 3, 2013
8:15pm Desire2Learn reported today that "all systems have operated above expected performance and we have seen no indication of recurring issues." They also report significant use (up to 80% of peak usage) and yet the system is not loaded (operating at 20% of its capacity). I have had no problem reports from staff who were checking this weekend. Everything looks good, and seems to have returned to normal ever since Friday at 12:30pm.
February 1, 20134:30pm This is the email I've been wanting to send since about 3:26pm on Tuesday. The Waterloo LEARN system has been running since about 12:30pm. We have done local testing and people are on the system, and it appears to be performing normally. We feel the system is stable and that it can go back into normal use. Desire2Learn continues to monitor their hosting site and we will continue to check for any alerts. A thorough investigation into events will begin next week, in collaboration with D2L.
1:40pm We had a phone call with the chief operating officer from Desire2Learn a short time ago. You may have already noted that our system is back online. They believe the problem is resolved for us, have performed their functional testing, and are asking for us to test as well. Our first priority is in making sure we have a fully functioning system. We are going through a verification testing and hope to be able to report back by 3:00 pm on how well the system is performing. D2L will be monitoring the systems closely and will send us any updates from other universities who are in the same situation as well.
Over the next week, we will undergo reviews with D2L and work with them to improve the planning process for changes that involve our system, so that we are aware of timing, potential risks, and participate in the decisions around planned changes. We also will work locally on improving contingency plans so we are better prepared to deal with such extraordinary circumstances.
Thank you for your patience and we will report again when we have more information on the system's stability now that it is back online.
8:45am In an update this morning (not the official general one from D2L, which had no news, rather from one of our reps), D2L has moved the majority of clients to the upgraded data infrastructure put in place last fall, and those clients are performing normally now.
While they continue to move data, they also are attempting to make improvements to the older infrastructure to allow remaining clients, including us, to return to normal, but that will require verification and testing.
Presumably one of these two-pronged approaches will be in place first and we will be able to run.
We have had several promises for a time line to completion but as of 8:45 D2L does not have a resolution time for us.
We have assurances there is no loss data, with verifying at every stage which takes longer.
We will update you as soon as we have substantive information. There was to have been a "material status update" in the official general updates from D2L, at about 7:30am, but that has not arrived.
January 31, 201310:00pm Sorry for the long lapse, but there were no definitive answers or time
lines to report. We have been in touch with many of you as we focused on
getting messages to the Deans to send to their professors (sent to Deans
at the end of the day and won't be forwarded until morning in some cases).
Also, we were providing information for a message to students (see posted
on the LEARN Help site, thanks to Daspina:
https://uwaterloo.ca/learn-help/news/message-provost-regarding-learn )
As of a 9:22pm update, D2L has confirmed that they know the solution to the problem, and it involves moving our data (and that for about 25% of their customers) so it is not behind a device that is causing the problems. They are working towards that now. In parallel, they are calculating estimates for how long the solution will take, and will send that information when it is available. They have said they will also update the LEARN redirect site with information on progress, presumably once they have established the expected time to completion. Otherwise, the next update is at midnight.
You may have noted already the article in the Record. ihttp://www.therecord.com/news/business/article/878997--desire2learn-s-cloud-data-centre-issues-have-uw-wlu-offline
Thank you, everyone, for your support and patience in this, and for all that you are doing to try to support faculty and students during this very frustrating time.
2:18pm
There has been no significant news on a recovery time. This update is to let you know there is a memo going to the Deans, meant as a message to send to all instructors. In it will be a pointer to a support site, showing instructors how to generate a mailing list to contact their students by email. Instructions are also available for how to give access to large files, rather than sending them through email. https://uwaterloo.ca/learn-help/while-learn-down
News on LEARN is at the same site. https://uwaterloo.ca/learn-help/news
We're not sure when the message will be sent to instructors, but this is a heads up of the support they may be looking for. More news as we get it.
10:45am
The latest news indicates that other clients are also experiencing the slowness, and that specific clients (and we don't yet know if we are one of them) will have data migrated to a different storage device. Later I will send an update from a phone call Dave Wallace and I had with some D2L executives. We will have short calls with the Deans over lunch hour to discuss the situation.
8:55am
We reported to D2L the HTTP 500 errors and also that our system was very slow. They know there is a problem and were working on it. We agreed it was better to have the system unavailable than run with those problems. We do not have a recovery time as yet, sorry.
7:45am
Desire2Learn made our system available again and said we can "assume that all work has been completed and that we have finished our verification testing of your site". However, the support team is seeing errors (HTTP 500) and has had reports of problems logging in by students and faculty members. The system is very slow. We will be talking to D2L about this and we will keep you updated as soon as we know more.
4:25amD2L sent an update at midnight, indicating that the process they had anticipated finishing by then was 98% done, and that once it completed, they would start verification.
At 4:25am the update indicates a 2-3 hour process needed. It is a bit confusing as to what this process is, except that it involves migration of our data and files to the new configuration. They will provide a further status update upon completion, which could be 6:30 - 7:30am based on their estimate. January 30, 20135:45pm The changes anticipated to be complete by 6:00pm are not complete. The revised finish time is estimated for between 10:00pm and midnight, followed by a verification process.
The Waterloo LEARN system remains inaccessible during this time.
We expect the next update at about 10pm.
3:30pm No change.
12:30pm Desire2Learn reports that they have received new devices at the hosted site, to fix the problem. There were to be configuration changes expected to finish at noon which have not completed. They expect it to be completed by 6pm ET, which seems to imply that is when we will be back up. Our next update from D2L will be at 3:00pm.
Unless we hear anything more before then, that will be the time of our next update.
8:07am We have heard from Desire2Learn (8:07am) and have done some testing of our own. According to D2L, they have "partially implemented" their solution. They caution that some sites may still see impacts. They "anticipate steady improvement to normal performance" by noon today.
From our tests, Waterloo LEARN seems to be very slow for login and for accessing files. (At about 7am it was performing well, but is not now.)
We will keep you apprised as we hear more. Our sincere apologies for this significant interruption. We will be pursuing detailed explanations from D2L.
January 29, 2013 Waterloo LEARN is currently unavailable, and has been since about 3:25pm today. The problem is at the hosted facility, an external site where our Desire2Learn system runs. We receive hourly updates from D2L but do not yet know the cause of the problem, or the uptime.
Keywords: (none)
