Streaming Video: Would you do a stop-over in Frankfurt, on a flight from Dubai to Oman?

Streaming video in the Middle East at a good quality, without buffering, is still much harder than it should be: there is no real central point in the region to serve everyone from, so in most cases the streams will run through Europe. The throughput has been increasing (the actual bandwidth to the countries), with capacity constantly being added. But latency is still a big issue.

https://www.submarinecablemap.com/

I remember a few years back I was at the MENOG (Middle East Network Operators Group) event in Muscat, Oman; I was on a panel with people from Netflix and Google (who are basically trying to solve the same problem as we are in the region). Seeing a room full of representatives of MENA telcos in front of me, from various countries in the region, the analogy was obvious. “Look, I arrived on a plane from Dubai to Muscat here this morning. It took about 45 minutes, and I didn’t even have to go through Frankfurt.” That got a laugh from the audience because they all knew exactly what I was talking about: internet traffic in the Gulf will often take some interesting detours, and Frankfurt is an old favorite for ISPs to connect.

Of course the Internet is faster than planes are, and connections are roughly at the speed of light. But distance matters: there’s a big difference between the 20 milliseconds it should be taking from Muscat to Dubai; and the actual 300 ms I was measuring for the connection at the time. Milliseconds may not sound like much, but it means that every API call our apps make will take half a second. It ruins the experience: to the user it just looks like the app is really sluggish and slow, every button press and every screen load takes longer than it should. For video it doesn’t just add to the start-up time (the time it takes from pressing “play” to the video actually starting to play); it also adds hops, with the potential of packet loss and other issues – which means the dreaded spinning circle on the screen, “buffering”. So we’re not even running our video streams from Dubai, but from Amsterdam – because at least then, we’re pretty much in the middle of how most connections to and from the Middle East are running.

To mitigate those issues, we use CDNs (Content Delivery Networks). The problem is that they’re expensive (especially given the volume of traffic our SHAHID VOD service has); and not one single CDN covers all our target markets in MENA. So we use 6 different CDNs, and we’re building our own as well. We’re constantly measuring which CDN is better for which country, and where to put our own servers (similar to Netflix’s Open Connect). SHAHID ran about 250 million videos plays for 30 million users in Ramadan (May), and it’s long form video, so we’re responsible for about 5% of internet traffic in the region (depending on how you calculate). Yet most of it still comes from Europe, first.

When I started working at MBC, I had to dig in my memory how the internet actually works again – in Europe and the US, relatively speaking, this is an almost “solved” problem, and I hadn’t had to deal with routing details since my days as a sysadmin in the nineties. But in Dubai, I have a map of the submarine cables forming the Internet backbone on my wall. That’s how much the network topology matters, and how much time I have to spend with my team figuring out how to improve our delivery.

So it was good to hear at this year’s MENOG conference in Beirut that the topic had shifted away from the Frankfurt internet exchange. It isn’t the main interconnection point for the Middle East anymore.

It’s now Marseille. :/

That may sound disappointed, but fortunately, there are multiple initiatives to actually bring exchanges to the region (such as the UAE-IX in Dubai, which, like the Frankfurt and Marseille exchanges, is run by DE-CIX). But in most cases, these have to operate under the license of a local operator; which means their competitors are hesitant to connect there. So progress is still slow, and Marseille is still a potential single point of failure for much of the internet connections in the Maghreb, Levant, and Gulf. For us, as a content provider, this is a chicken and the egg kind of problem: which comes first? If the connectivity isn’t available at the local exchanges, there is no point in us having a presence there – but if the content isn’t there, there’s no incentive for the operators to connect to the exchanges.

An additional problem is the dearth of cloud services in the region. Our current best bet is running on AWS from Frankfurt, and Azure from Amsterdam, because as I mentioned – if your connections go via Europe, then why not put your service in Europe and be in the middle of the path? The good news there is that we now have Azure in the UAE, and soon we’ll have AWS in Bahrain. (I’d like to think that’s because I’ve been badgering Amazon and Microsoft for years for this, but in reality it’s probably mostly to serve enterprise loads.) But when I met Amazon’s Werner Vogels a few months back, I had just one question for him: how is your Bahrain Region going to be connected? Because the potential for fast connections is there, but it’s definitely not a given.

We’re eager to move our services to the region, and greatly improve the experience. So we’ll be upgrading our own internet exchange presence from proof of concept to pilot; and hopefully start using Bahrain and UAE datacenters as “local cloud” as well. Because, to use a terrible analogy; sometimes, in order to get out of a chicken and the egg problem, you have to make an omelet first.

As for that trip to Oman – I’m happy to say Emirates now flies the A380 direct from Dubai to Muscat. And the latency is 15ms. Things are improving!

“I’m sorry, but I never signed up to be your customer”: when your supplier gets acquired

Big Fish Eating Small FishFact of life in technology: smaller companies get acquired by large ones all the time. If, like me, you tend to favor best-of-breed solutions; where you’re an important customer to a medium sized company; and get to influence the roadmap – this is a problem. Suddenly, you’ll get an email: “We’re proud to announce”, some details of the acquisition, and “it will be business as usual”. The founders are happy with their big pay day (as well they should), and their friends are congratulating them on Twitter and LinkedIn.
Continue reading ““I’m sorry, but I never signed up to be your customer”: when your supplier gets acquired”

Social Media in One or Two Words

socmedIt’s an interesting question: what is social media, anyway, to a modern professional? As a hectic work week was dwindling to an end on Thursday afternoon (Dubai work weeks are Sunday to Thursday — starting on Sunday is still weird, but starting the weekend on Thursday never gets old), I was chatting with my colleague Fouad Massoud. He’s in charge of MBC’s social media efforts (which he built from zero to an audience of millions), so he usually has some interesting points of view. As he was discussing how professional networking is transitioning from Twitter to LinkedIn, he pointed out one issue with LinkedIn: where do you find active, interesting groups? Continue reading “Social Media in One or Two Words”

How to be an Expert in Absolutely Everything

Did you ever come across someone who seemed to know everything? An expert in not just one thing, but many things across different disciplines? Who not only stunned you with a wealth of trivia, but also all the applicable anecdotes, and the relevant facts to almost anything that would come up in conversation? I’m sure you have. And if that made you think you’d want to be more like that, the good news is: it’s easy! You only need two skills and this one weird trick. Continue reading “How to be an Expert in Absolutely Everything”

The Problem with Hadoop

Fake ElephantFirst, let’s be clear: I really like Hadoop, and not just because it’s named after a yellow toy elephant. But over the past few years, “Hadoop” has also become an almost mystical term, happily sprinkled throughout marketing brochures. So, to be fair, it’s not Hadoop that is the problem — the problem is about Hadoop and how it is perceived. And I’m not going to go into great technical details, but there are a few problems with that perception. Continue reading “The Problem with Hadoop”

Presentation: From Pages to People

Magnolia ConferenceAt the end of June, I spent a few days in Basel, Switzerland at the Magnolia CMS Conference. In my presentation, in forty minutes, I tried to cover how a modern site needs to change from funnel to journey, what tools you can use for this, the infrastructure that ties all this together. And of course, penguins. A lot of penguins. Magnolia has created great videos of all the presentations, so if you have some time to waste, here’s mine. Continue reading “Presentation: From Pages to People”

“Visitors”: between Big Data and CRM

LHC : CMS waiting for tracker insertionIf you’re a media company, or a publisher, and you look at “Big Data”, chances are you’re not talking about something like the 25 Petabytes produced annually by the Large Hadron Collider. The visitors of your sites or the users of your apps may be a large group of unknowns to you; but you’re not trying to find a Higgs in a haystack. You’re likely thinking about a lot of data; but is that really Big Data? Continue reading ““Visitors”: between Big Data and CRM”

Managing Content in the Middle East

Tunisia-3749 - Here comes the sun.....It’s been two years now since I graduated from being “regular visitor” to “permanent resident” in the Middle East. In the digital space, it’s an area of huge potential, and that’s of course what attracted me in the first place. It’s tempting to say the region is a few years behind the state-of-the-art in the U.S. and E.U., but that’s an oversimplification. In many ways, it’s just….different. Continue reading “Managing Content in the Middle East”