Streaming video in the Middle East at a good quality, without buffering, is still much harder than it should be: there is no real central point in the region to serve everyone from, so in most cases the streams will run through Europe. The throughput has been increasing (the actual bandwidth to the countries), with capacity constantly being added. But latency is still a big issue.
I remember a few years back I was at the MENOG (Middle East Network Operators Group) event in Muscat, Oman; I was on a panel with people from Netflix and Google (who are basically trying to solve the same problem as we are in the region). Seeing a room full of representatives of MENA telcos in front of me, from various countries in the region, the analogy was obvious. “Look, I arrived on a plane from Dubai to Muscat here this morning. It took about 45 minutes, and I didn’t even have to go through Frankfurt.” That got a laugh from the audience because they all knew exactly what I was talking about: internet traffic in the Gulf will often take some interesting detours, and Frankfurt is an old favorite for ISPs to connect.
Of course the Internet is faster than planes are, and connections are roughly at the speed of light. But distance matters: there’s a big difference between the 20 milliseconds it should be taking from Muscat to Dubai; and the actual 300 ms I was measuring for the connection at the time. Milliseconds may not sound like much, but it means that every API call our apps make will take half a second. It ruins the experience: to the user it just looks like the app is really sluggish and slow, every button press and every screen load takes longer than it should. For video it doesn’t just add to the start-up time (the time it takes from pressing “play” to the video actually starting to play); it also adds hops, with the potential of packet loss and other issues – which means the dreaded spinning circle on the screen, “buffering”. So we’re not even running our video streams from Dubai, but from Amsterdam – because at least then, we’re pretty much in the middle of how most connections to and from the Middle East are running.
To mitigate those issues, we use CDNs (Content Delivery Networks). The problem is that they’re expensive (especially given the volume of traffic our SHAHID VOD service has); and not one single CDN covers all our target markets in MENA. So we use 6 different CDNs, and we’re building our own as well. We’re constantly measuring which CDN is better for which country, and where to put our own servers (similar to Netflix’s Open Connect). SHAHID ran about 250 million videos plays for 30 million users in Ramadan (May), and it’s long form video, so we’re responsible for about 5% of internet traffic in the region (depending on how you calculate). Yet most of it still comes from Europe, first.
When I started working at MBC, I had to dig in my memory how the internet actually works again – in Europe and the US, relatively speaking, this is an almost “solved” problem, and I hadn’t had to deal with routing details since my days as a sysadmin in the nineties. But in Dubai, I have a map of the submarine cables forming the Internet backbone on my wall. That’s how much the network topology matters, and how much time I have to spend with my team figuring out how to improve our delivery.
So it was good to hear at this year’s MENOG conference in Beirut that the topic had shifted away from the Frankfurt internet exchange. It isn’t the main interconnection point for the Middle East anymore.
It’s now Marseille. :/
That may sound disappointed, but fortunately, there are multiple initiatives to actually bring exchanges to the region (such as the UAE-IX in Dubai, which, like the Frankfurt and Marseille exchanges, is run by DE-CIX). But in most cases, these have to operate under the license of a local operator; which means their competitors are hesitant to connect there. So progress is still slow, and Marseille is still a potential single point of failure for much of the internet connections in the Maghreb, Levant, and Gulf. For us, as a content provider, this is a chicken and the egg kind of problem: which comes first? If the connectivity isn’t available at the local exchanges, there is no point in us having a presence there – but if the content isn’t there, there’s no incentive for the operators to connect to the exchanges.
An additional problem is the dearth of cloud services in the region. Our current best bet is running on AWS from Frankfurt, and Azure from Amsterdam, because as I mentioned – if your connections go via Europe, then why not put your service in Europe and be in the middle of the path? The good news there is that we now have Azure in the UAE, and soon we’ll have AWS in Bahrain. (I’d like to think that’s because I’ve been badgering Amazon and Microsoft for years for this, but in reality it’s probably mostly to serve enterprise loads.) But when I met Amazon’s Werner Vogels a few months back, I had just one question for him: how is your Bahrain Region going to be connected? Because the potential for fast connections is there, but it’s definitely not a given.
We’re eager to move our services to the region, and greatly improve the experience. So we’ll be upgrading our own internet exchange presence from proof of concept to pilot; and hopefully start using Bahrain and UAE datacenters as “local cloud” as well. Because, to use a terrible analogy; sometimes, in order to get out of a chicken and the egg problem, you have to make an omelet first.
As for that trip to Oman – I’m happy to say Emirates now flies the A380 direct from Dubai to Muscat. And the latency is 15ms. Things are improving!