*UPDATE 6* – Services are beginning to be restored. Here’s the word from RIM:

Thursday 13th October – 9:00 (GMT-5)
BlackBerry services are operating well globally. BlackBerry Support teams continue to monitor the situation around the clock to ensure ongoing service stability. Some customers in Canada and Latin America who are sending messages to other regions may see intermittent message delays. Support teams are actively addressing this.

ADVERTISEMENT

*UPDATE 5* – RIM’s CIO had this to say on their status update page:

Service update from RIM CIO

To All BlackBerry Customers:

I want to first apologize for the service interruptions and delays many of you have been experiencing this week. I also wanted to connect with you directly, give you an update on the service issues we are trying to solve, and answer some of the questions and concerns you’ve expressed.

You’ve depended on us for reliable, real-time communications, and right now we’re letting you down. We are taking this very seriously and have people around the world working around the clock to address this situation. We believe we understand why this happened and we are working to restore normal service levels in all markets as quickly as we can.

Here is the current status of service and issues for the various regions that were impacted:

For Europe, Middle East, India and Africa (EMEIA):

  • Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers.
  • BBM traffic is online and traffic is passing successfully
  • Browsing is temporarily unavailable as the Support teams monitor service stability and continue to assess when this service can be safely brought online
  • Support teams have added capacity to help with message delivery between regions and continents

For Canada and Latin America:

  • Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers
  • BBM and browsing services are online and traffic is passing successfully (except for three carrier networks in Latin America that are serviced by the EMEIA infrastructure – browsing is temporarily unavailable for those three carrier networks)
  • Support teams are investigating reports of BBM delays

For the U.S.:

  • Email systems are operating and we are continuing to clear any backlogged messages. Support teams are working to minimize the impact on our customers.
  • Support teams have added capacity to help with message delivery between regions and continents
  • BBM and browsing services are online and traffic is passing successfully
  • Support teams are investigating reports of BBM delays

We will provide regular updates on BlackBerry.com, RIM.com and via our social channels. We are doing everything in our power to restore regular service everywhere and to restore your trust in us.

ADVERTISEMENT

Yours sincerely,

Robin Bienfait
Chief Information Officer, RIM

*UPDATE 4* – If it looks fake, it probably is. Some genius started a BBM broadcast message hoax. Here’s what RIM says about it:

Wednesday 12th October – 16:12 (GMT-5)
RIM is aware of a hoax message that has been circulated recently amongst certain BlackBerry Messenger (BBM) users. This is a hoax chain message. The message did not originate from RIM and does not impact the security of BlackBerry smartphones. Hoax messages are unfortunately an industry-wide issue. Any social messaging application on any platform, mobile or PC can be used to attempt to convince users to forward such hoax messages to one or more of their contacts. RIM recommends that users simply ignore the message and do not forward it, since this would only serve to expand the reach of the hoax message.

*UPDATE 3* – RIM has finally created a services update page regarding the outages and areas affected. The latest is as follows:

Wednesday 12th October – 9:45 (GMT-5)
BlackBerry subscribers in the Americas may be experiencing intermittent service delays this morning. We are working to resolve the situation as quickly as possible and we apologize to our customers for any inconvenience. We will provide a further update as soon as more information is available.

*UPDATE 2* – Users in North America are beginning to report BBM delays among other BlackBerry data outage. Here is the latest update we’ve received from our source on what RIM is doing to resolve the issue in EMEA and South America:

BlackBerry support teams continue to work on resolving this issue.

  • Some customers in Europe, Middle East, Africa, India, Argentina, Chile, and Brazil may be experiencing delays in sending or receiving messages, or delays with instant messaging or browsing.
  • The BlackBerry Support teams have successfully transitioned over to a backup database.
  • This database has double the processing power and this will assist in processing queued messages and database queries more quickly.
  • BlackBerry services are being reintroduced in a slow and controlled manner. Service recovery is progressing well.

The messaging and browsing delays being experienced by BlackBerry users in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina were caused by a core switch failure within RIM’s infrastructure.  Although the system is designed to failover to a back-up switch, the failover did not function as previously tested. As a result, a large backlog of data was generated and we are now working to clear that backlog and restore normal service as quickly as possible.

*UPDATE* – RIM released an official statement on the matter via their Facebook page. Here is what it says: “The messaging and browsing delays being experienced by BlackBerry users in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina were caused by a core switch failure within RIM’s infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested. As a result, a large backlog of data was generated, and we are now working to clear that backlog and restore normal service as quickly as possible. We apologize for any inconvenience, and we will continue to keep you informed.”

We’ve received information from one of our sources regarding the massive BlackBerry Internet Service (BIS) outage affecting Europe, Middle East, and Africa. There was a resolution estimate of 4 hours. However, that has been updated to ‘no resolution time available’. Here are the full details on the EMEA outage:

Latest update from RIM sent to carriers…

Please see the latest update on the issue. There is still no resolution time available.

  • The BlackBerry Support Team has identified a network connectivity issue between data centers and Support Teams are working on resolution of this now.
  • BES email is recovering well.
  • BIS email may be experiencing some delays in sending/receiving messages.
  • BBM is impacted and many users are experiencing delays.
  • BlackBerry Browsing may be impaired.
  • The support teams are continuing to manage message queues and are performing controlled restarts of any affected components.

Initial notification sent to carriers…

SERVICE(S) AFFECTED
Service Name: BlackBerry Services
Start Date/Time: 11-Oct-2011 10:58:00 GMT
Duration: Ongoing
Region of Impact: EMEA
State of Service: Severely Degraded
% of Subscribers Affected (estimated): 50

Service Impact: Subscribers may experience delays in sending or receiving messages, roaming in another location, or using other services such as Internet browsing or instant messaging. Subscribers may also be unable to create new accounts, integrate third-party email accounts, or use BlackBerry subscriber web sites. Devices may not receive new service books. BlackBerry-enabled devices that request a new PIN may not receive a PIN. BlackBerry Enterprise Servers may be unable to connect to the BlackBerry Infrastructure. Wireless service providers and device resellers may experience delays in using BlackBerry administration web sites, creating subscriber accounts, or provisioning services for subscribers. [00017]

Estimated Time to Resolution: Approximately 4 hours

Action(s) Taken:  BlackBerry support teams have been engaged and are working toward resolution as quickly as possible.

Thank you for your patience during this time.

RIM has been apologizing via Twitter to those affected. Although, this can’t come at a more crucial time when RIM needs to make sure these kind of mass outages are thwarted, especially as BIS is constantly criticized as being an archaic system. If you’re suffering issues, check out our first discussion on it here.