Rogers had a catastrophic failure in July 2022. It affected emergency services (as in: people couldn't call 911, but also some 911 services themselves failed), hospitals (which couldn't access prescriptions), banks and payment systems (as payment terminals stopped working), and regular users as well. The outage lasted almost a full day, and Rogers took days to give any technical explanation on the outage, and even when they did, details were sparse. So far the only detailed account is from outside actors like Cloudflare which seem to point at an internal BGP failure.

Its impact on the economy has yet to be measured, but it probably cost millions of dollars in wasted time and possibly lead to life-threatening situations. Apart from holding Rogers (criminally?) responsible for this, what should be done in the future to avoid such problems?

It's not the first time something like this has happened: it happened to Bell Canada as well. The Rogers outage is also strangely similar to the Facebook outage last year, but, to its credit, Facebook did post a fairly detailed explanation only a day later.

The internet is designed to be decentralised, and having large companies like Rogers hold so much power is a crucial mistake that should be reverted. The question is how. Some critics were quick to point out that we need more ISP diversity and competition, but I think that's missing the point. Others have suggested that the internet should be a public good or even straight out nationalized.

I believe the solution to the problem of large, private, centralised telcos and ISPs is to replace them with smaller, public, decentralised service providers. The only way to ensure that works is to make sure that public money ends up creating infrastructure controlled by the public, which means treating ISPs as a public utility. This has been implemented elsewhere: it works, it's cheaper, and provides better service.

Note, this is an idea I have already proposed in 2012. See also the shorter, french version.

  1. A modest proposal
    1. Municipal broadband
    2. Provincial public internet
    3. International public internet
    4. Wireless networks
  2. Counter points
    1. I don't want the state to control my internet
    2. Isn't the government incompetent and corrupt?
    3. A single monopoly will be less reliable
    4. Wouldn't this create an isolated network?
    5. What about Google and Facebook?
  3. What does not work
    1. Legal pressure and regulation
    2. Subsidies
  4. Conclusion
  5. Updates
    1. Hacker news
    2. Acquisitions
    3. Washington Post article
    4. AMBER failure
    5. New CAQ fund and french article

A modest proposal

Global wireless services (like phone services) and home internet inevitably grow into monopolies. They are public utilities, just like water, power, railways, and roads. The question of how they should be managed is therefore inherently political, yet people don't seem to question the idea that only the market (i.e. "competition") can solve this problem. I disagree.

10 years ago (in french), I suggested we, in Québec, should nationalize large telcos and internet service providers. I no longer believe is a realistic approach: most of those companies have crap copper-based networks (at least for the last mile), yet are worth billions of dollars. It would be prohibitive, and a waste, to buy them out.

Back then, I called this idea "Réseau-Québec", a reference to the already nationalized power company, Hydro-Québec. (This idea, incidentally, made it into the plan of a political party.)

Now, I think we should instead build our own, public internet. Start setting up municipal internet services, fiber to the home in all cities, progressively. Then interconnect cities with fiber, and build peering agreements with other providers. This also includes a bid on wireless spectrum to start competing with phone providers as well.

And while that sounds really ambitious, I think it's possible to take this one step at a time.

Municipal broadband

In many parts of the world, municipal broadband is an elegant solution to the problem, with solutions ranging from Stockholm's city-owned fiber network (dark fiber, layer 1) to Utah's UTOPIA network (fiber to the premises, layer 2) and municipal wireless networks like Guifi.net which connects about 40,000 nodes in Catalonia.

A good first step would be for cities to start providing broadband services to its residents, directly. Cities normally own sewage and water systems that interconnect most residences and therefore have direct physical access everywhere. In Montréal, in particular, there is an ongoing project to replace a lot of old lead-based plumbing which would give an opportunity to lay down a wired fiber network across the city.

This is a wild guess, but I suspect this would be much less expensive than one would think. Some people agree with me and quote this as low as 1000$ per household. There is about 800,000 households in the city of Montréal, so we're talking about a 800 million dollars investment here, to connect every household in Montréal with fiber and incidentally a quarter of the province's population. And this is not an up-front cost: this can be built progressively, with expenses amortized over many years.

(We should not, however, connect Montréal first: it's used as an example here because it's a large number of households to connect.)

Such a network should be built with a redundant topology. I leave it as an open question whether we should adopt Stockholm's more minimalist approach or provide direct IP connectivity. I would tend to favor the latter, because then you can immediately start to offer the service to households and generate revenues to compensate for the capital expenditures.

Given the ridiculous profit margins telcos currently have — 8 billion $CAD net income for BCE (2019), 2 billion $CAD for Rogers (2020) — I also believe this would actually turn into a profitable revenue stream for the city, the same way Hydro-Québec is more and more considered as a revenue stream for the state. (I personally believe that's actually wrong and we should treat those resources as human rights and not money cows, but I digress. The point is: this is not a cost point, it's a revenue.)

The other major challenge here is that the city will need competent engineers to drive this project forward. But this is not different from the way other public utilities run: we have electrical engineers at Hydro, sewer and water engineers at the city, this is just another profession. If anything, the computing science sector might be more at fault than the city here in its failure to provide competent and accountable engineers to society...

Right now, most of the network in Canada is copper: we are hitting the limits of that technology with DSL, and while cable has some life left to it (DOCSIS 4.0 does 4Gbps), that is nowhere near the capacity of fiber. Take the town of Chattanooga, Tennessee: in 2010, the city-owned ISP EPB finished deploying a fiber network to the entire town and provided gigabit internet to everyone. Now, 12 years later, they are using this same network to provide the mind-boggling speed of 25 gigabit to the home. To give you an idea, Chattanooga is roughly the size and density of Sherbrooke.

Provincial public internet

As part of building a municipal network, the question of getting access to "the internet" will immediately come up. Naturally, this will first be solved by using already existing commercial providers to hook up residents to the rest of the global network.

But eventually, networks should inter-connect: Montréal should connect with Laval, and then Trois-Rivières, then Québec City. This will require long haul fiber runs, but those links are not actually that expensive, and many of those already exist as a public resource at RISQ and CANARIE, which cross-connects universities and colleges across the province and the country. Those networks might not have the capacity to cover the needs of the entire province right now, but that is a router upgrade away, thanks to the amazing capacity of fiber.

There are two crucial mistakes to avoid at this point. First, the network needs to remain decentralised. Long haul links should be IP links with BGP sessions, and each city (or MRC) should have its own independent network, to avoid Rogers-class catastrophic failures.

Second, skill needs to remain in-house: RISQ has already made that mistake, to a certain extent, by selling its neutral datacenter. Tellingly, MetroOptic, probably the largest commercial dark fiber provider in the province, now operates the QIX, the second largest "public" internet exchange in Canada.

Still, we have a lot of infrastructure we can leverage here. If RISQ or CANARIE cannot be up to the task, Hydro-Québec has power lines running into every house in the province, with high voltage power lines running hundreds of kilometers far north. The logistics of long distance maintenance are already solved by that institution.

In fact, Hydro already has fiber all over the province, but it is a private network, separate from the internet for security reasons (and that should probably remain so). But this only shows they already have the expertise to lay down fiber: they would just need to lay down a parallel network to the existing one.

In that architecture, Hydro would be a "dark fiber" provider.

International public internet

None of the above solves the problem for the entire population of Québec, which is notoriously dispersed, with an area three times the size of France, but with only an eight of its population (8 million vs 67). More specifically, Canada was originally a french colony, a land violently stolen from native people who have lived here for thousands of years. Some of those people now live in reservations, sometimes far from urban centers (but definitely not always). So the idea of leveraging the Hydro-Québec infrastructure doesn't always work to solve this, because while Hydro will happily flood a traditional hunting territory for an electric dam, they don't bother running power lines to the village they forcibly moved, powering it instead with noisy and polluting diesel generators. So before giving me fiber to the home, we should give power (and potable water, for that matter), to those communities first.

Update: some of those people are just doing it themselves.

So we need to discuss international connectivity. (How else could we consider those communities than peer nations anyways?c) Québec has virtually zero international links. Even in Montréal, which likes to style itself a major player in gaming, AI, and technology, most peering goes through either Toronto or New York.

That's a problem that we must fix, regardless of the other problems stated here. Looking at the submarine cable map, we see very few international links actually landing in Canada. There is the Greenland connect which connects Newfoundland to Iceland through Greenland. There's the EXA which lands in Ireland, the UK and the US, and Google has the Topaz link on the west coast. That's about it, and none of those land anywhere near any major urban center in Québec.

We should have a cable running from France up to Saint-Félicien. There should be a cable from Vancouver to China. Heck, there should be a fiber cable running all the way from the end of the great lakes through Québec, then up around the northern passage and back down to British Columbia. Those cables are expensive, and the idea might sound ludicrous, but Russia is actually planning such a project for 2026. The US has cables running all the way up (and around!) Alaska, neatly bypassing all of Canada in the process. We just look ridiculous on that map.

(Addendum: I somehow forgot to talk about Teleglobe here was founded as publicly owned company in 1950, growing international phone and (later) data links all over the world. It was privatized by the conservatives in 1984, along with rails and other "crown corporations". So that's one major risk to any effort to make public utilities work properly: some government might be elected and promptly sell it out to its friends for peanuts.)

Wireless networks

I know most people will have rolled their eyes so far back their heads have exploded. But I'm not done yet. I want wireless too. And by wireless, I don't mean a bunch of geeks setting up OpenWRT routers on rooftops. I tried that, and while it was fun and educational, it didn't scale.

A public networking utility wouldn't be complete without providing cellular phone service. This involves bidding for frequencies at the federal level, and deploying a rather large amount of infrastructure, but it could be a later phase, when the engineers and politicians have proven their worth.

At least part of the Rogers fiasco would have been averted if such a decentralized network backend existed. One might even want to argue that a separate institution should be setup to provide phone services, independently from the regular wired networking, if only for reliability.

Because remember here: the problem we're trying to solve is not just technical, it's about political boundaries, centralisation, and automation. If everything is ran by this one organisation again, we will have failed.

However, I must admit that phone services is where my ideas fall a little short. I can't help but think it's also an accessible goal — maybe starting with a virtual operator — but it seems slightly less so than the others, especially considering how closed the phone ecosystem is.

Counter points

In debating these ideas while writing this article, the following objections came up.

I don't want the state to control my internet

One legitimate concern I have about the idea of the state running the internet is the potential it would have to censor or control the content running over the wires.

But I don't think there is necessarily a direct relationship between resource ownership and control of content. Sure, China has strong censorship in place, partly implemented through state-controlled businesses. But Russia also has strong censorship in place, based on regulatory tools: they force private service providers to install back-doors in their networks to control content and surveil their users.

Besides, the USA have been doing warrantless wiretapping since at least 2003 (and yes, that's 10 years before the Snowden revelations) so a commercial internet is no assurance that we have a free internet. Quite the contrary in fact: if anything, the commercial internet goes hand in hand with the neo-colonial internet, just like businesses did in the "good old colonial days".

Large media companies are the primary censors of content here. In Canada, the media cartel requested the first site-blocking order in 2018. The plaintiffs (including Québecor, Rogers, and Bell Canada) are both content providers and internet service providers, an obvious conflict of interest.

Nevertheless, there are some strong arguments against having a centralised, state-owned monopoly on internet service providers. FDN makes a good point on this. But this is not what I am suggesting: at the provincial level, the network would be purely physical, and regional entities (which could include private companies) would peer over that physical network, ensuring decentralization. Delegating the management of that infrastructure to an independent non-profit or cooperative (but owned by the state) would also ensure some level of independence.

Update: this article from the OECD forum talks about such redundancy is also important in war time.

Isn't the government incompetent and corrupt?

Also known as "private enterprise is better skilled at handling this, the state can't do anything right"

I don't think this is a "fait accomplit". If anything, I have found publicly ran utilities to be spectacularly reliable here. I rarely have trouble with sewage, water, or power, and keep in mind I live in a city where we receive about 2 meters of snow a year, which tend to create lots of trouble with power lines. Unless there's a major weather event, power just runs here.

I think the same can happen with an internet service provider. But it would certainly need to have higher standards to what we're used to, because frankly Internet is kind of janky.

A single monopoly will be less reliable

I actually agree with that, but that is not what I am proposing anyways. Current commercial or non-profit entities will be free to offer their services on top of the public network.

And besides, the current "ha! diversity is great" approach is exactly what we have now, and it's not working. The pretense that we can have competition over a single network is what led the US into the ridiculous situation where they also pretend to have competition over the power utility market. This led to massive forest fires in California and major power outages in Texas. It doesn't work.

Wouldn't this create an isolated network?

One theory is that this new network would be so hostile to incumbent telcos and ISPs that they would simply refuse to network with the public utility. And while it is true that the telcos currently do also act as a kind of "tier one" provider in some places, I strongly feel this is also a problem that needs to be solved, regardless of ownership of networking infrastructure.

Right now, telcos often hold both ends of the stick: they are the gateway to users, the "last mile", but they also provide peering to the larger internet in some locations. In at least one datacenter in downtown Montréal, I've seen traffic go through Bell Canada that was not directly targeted at Bell customers. So in effect, they are in a position of charging twice for the same traffic, and that's not only ridiculous, it should just be plain illegal.

And besides, this is not a big problem: there are other providers out there. As bad as the market is in Québec, there is still some diversity in Tier one providers that could allow for some exits to the wider network (e.g. yes, Cogent is here too).

What about Google and Facebook?

Nationalization of other service providers like Google and Facebook is out of scope of this discussion.

That said, I am not sure the state should get into the business of organising the web or providing content services however, but I will point out it already does do some of that through its own websites. It should probably keep itself to this, and also consider providing normal services for people who don't or can't access the internet.

(And I would also be ready to argue that Google and Facebook already act as extensions of the state: certainly if Facebook didn't exist, the CIA or the NSA would like to create it at this point. And Google has lucrative business with the US department of defense.)

What does not work

So we've seen one thing that could work. Maybe it's too expensive. Maybe the political will isn't there. Maybe it will fail. We don't know yet.

But we know what does not work, and it's what we've been doing ever since the internet has gone commercial.

In 1984 (of all years), the US Department of Justice finally broke up AT&T in half a dozen corporations, after a 10 year legal battle. Yet a decades later, we're back to only three large providers doing essentially what AT&T was doing back then, and those are regional monopolies: AT&T, Verizon, and Lumen (not counting T-Mobile that is from a different breed). So the legal approach really didn't work that well, especially considering the political landscape changed in the US, and the FTC seems perfectly happy to let those major mergers continue.

In Canada, we never even pretended we would solve this problem at all: Bell Canada (the literal "father" of AT&T) is in the same situation now. We have either a regional monopoly (e.g. Videotron for cable in Québec) or an oligopoly (Bell, Rogers, and Telus controlling more than 90% of the market). Telus does have one competitor in the west of Canada, Shaw, but Rogers has been trying to buy it out. The competition bureau seems to have blocked the merger for now, but it didn't stop other recent mergers like Bell's acquisition one of its main competitors in Québec, eBox.

Regulation doesn't seem capable of ensuring those profitable corporations provide us with decent pricing, which makes Canada one of the most expensive countries (research) for mobile data on the planet. The recent failure of the CRTC to properly protect smaller providers has even lead to price hikes. Meanwhile the oligopoly is actually agreeing on their own price hikes therefore becoming a real cartel, complete with price fixing and reductions in output.

There are actually regulations in Canada supposed to keep the worst of the Rogers outage from happening at all. According to CBC:

Under Canadian Radio-television and Telecommunications Commission (CRTC) rules in place since 2017, telecom networks are supposed to ensure that cellphones are able to contact 911 even if they do not have service.

I could personally confirm that my phone couldn't reach 911 services, because all calls would fail: the problem was that towers were still up, so your phone wouldn't fall back to alternative service providers (which could have resolved the issue). I can only speculate as to why Rogers didn't take cell phone towers out of the network to let phones work properly for 911 service, but it seems like a dangerous game to play.

Hilariously, the CRTC itself didn't have a reliable phone service due to the service outage:

Please note that our phone lines are affected by the Rogers network outage. Our website is still available: https://crtc.gc.ca/eng/contact/

https://mobile.twitter.com/CRTCeng/status/1545421218534359041

I wonder if they will file a complaint against Rogers themselves about this. I probably should.

It seems the federal government is thinking more of the same medicine will fix the problem and has told companies should "help" each other in an emergency. I doubt this will fix anything, and could actually make things worse if the competitors actually interoperate more, as it could cause multi-provider, cascading failures.

Subsidies

The absurd price we pay for data does not actually mean everyone gets high speed internet at home. Large swathes of the Québec countryside don't get broadband at all, and it can be difficult or expensive, even in large urban centers like Montréal, to get high speed internet.

That is despite having a series of subsidies that all avoided investing in our own infrastructure. We had the "fonds de l'autoroute de l'information", "information highway fund" (site dead since 2003, archive.org link) and "branchez les familles", "connecting families" (site dead since 2003, archive.org link) which subsidized the development of a copper network. In 2014, more of the same: the federal government poured hundreds of millions of dollars into a program called connecting Canadians to connect 280 000 households to "high speed internet". And now, the federal and provincial governments are proudly announcing that "everyone is now connected to high speed internet", after pouring more than 1.1 billion dollars to connect, guess what, another 380 000 homes, right in time for the provincial election.

Of course, technically, the deadline won't actually be met until 2023. Québec is a big area to cover, and you can guess what happens next: the telcos threw up their hand and said some areas just can't be connected. (Or they connect their CEO but not the poor folks across the lake.) The story then takes the predictable twist of giving more money out to billionaires, subsidizing now Musk's Starlink system to connect those remote areas.

To give a concrete example: a friend who lives about 1000km away from Montréal, 4km from a small, 2500 habitant village, has recently got symmetric 100 mbps fiber at home from Telus, thanks to those subsidies. But I can't get that service in Montréal at all, presumably because Telus and Bell colluded to split that market. Bell doesn't provide me with such a service either: they tell me they have "fiber to my neighborhood", and only offer me a 25/10 mbps ADSL service. (There is Vidéotron offering 400mbps, but that's copper cable, again a dead technology, and asymmetric.)

Conclusion

Remember Chattanooga? Back in 2010, they funded the development of a fiber network, and now they have deployed a network roughly a thousand times faster than what we have just funded with a billion dollars. In 2010, I was paying Bell Canada 60$/mth for 20mbps and a 125GB cap, and now, I'm still (indirectly) paying Bell for roughly the same speed (25mbps). Back then, Bell was throttling their competitors networks until 2009, when they were forced by the CRTC to stop throttling. Both Bell and Vidéotron still explicitly forbid you from running your own servers at home, Vidéotron charges prohibitive prices which make it near impossible for resellers to sell uncapped services. Those companies are not spurring innovation: they are blocking it.

We have spent all this money for the private sector to build us a private internet, over decades, without any assurance of quality, equity or reliability. And while in some locations, ISPs did deploy fiber to the home, they certainly didn't upgrade their entire network to follow suit, and even less allowed resellers to compete on that network.

In 10 years, when 100mbps will be laughable, I bet those service providers will again punt the ball in the public courtyard and tell us they don't have the money to upgrade everyone's equipment.

We got screwed. It's time to try something new.

Updates

Hacker news

There was a discussion about this article on Hacker News which was surprisingly productive. Trigger warning: Hacker News is kind of right-wing, in case you didn't know.

Acquisitions

Since this article was written, at least two more major acquisitions happened, just in Québec:

In the latter case, vMedia was explicitly saying it couldn't grow because of "lack of access to capital". So basically, we have given those companies a billion dollars, and they are not using that very money to buy out their competition. At least we could have given that money to small players to even out the playing field. But this is not how that works at all. Also, in a bizarre twist, an "analyst" believes the acquisition is likely to help Rogers acquire Shaw.

Washington Post article

Also, since this article was written, the Washington Post published a review of a book bringing similar ideas: Internet for the People The Fight for Our Digital Future, by Ben Tarnoff, at Verso books. It's short, but even more ambitious than what I am suggesting in this article, arguing that all big tech companies should be broken up and better regulated:

He pulls from Ethan Zuckerman’s idea of a web that is “plural in purpose” — that just as pool halls, libraries and churches each have different norms, purposes and designs, so too should different places on the internet. To achieve this, Tarnoff wants governments to pass laws that would make the big platforms unprofitable and, in their place, fund small-scale, local experiments in social media design. Instead of having platforms ruled by engagement-maximizing algorithms, Tarnoff imagines public platforms run by local librarians that include content from public media.

(Links mine: the Washington Post obviously prefers to not link to the real web, and instead doesn't link to Zuckerman's site all and suggests Amazon for the book, in a cynical example.)

AMBER failure

And in another example of how the private sector has failed us, there was recently a fluke in the AMBER alert system where the entire province was warned about a loose shooter in Saint-Elzéar except the people in the town, because they have spotty cell phone coverage. In other words, millions of people received a strongly toned, "life-threatening", alert for a city sometimes hours away, except the people most vulnerable to the alert. Not missing a beat, the CAQ party is promising more of the same medicine again and giving more money to telcos to fix the problem, suggesting to spend three billion dollars in private infrastructure.

New CAQ fund and french article

A french redux of this article was posted here, and sent to Le Devoir, which never published it.

Instead, they published what seems to be trial balloon smack on the frontpage. Basically, the CAQ now believes they need to invest in public fiber networks and have reserved 3 billion dollars to do ... something. There's some vague talks about dark fiber with Hydro and noises made about how we nationalized Hydro back in the glory days. The article and the CAQ are very careful at attributing those ideas to anyone else, either here or to Québec Solidaire, which has had this in their program since 2018.

I suspect this is going to turn out to be yet another wealth transfer from the public to the private, but who knows, things might turn out ... okay?

In the meantime, I still don't have access to fiber in the center of the largest french-speaking city of the western hemisphere and the second largest city in Canada. And that, even if Bell's old competitor (and now subsidiary) Ebox is now offering fiber to the home. Obviously, they haven't increased coverage significantly.

Interesting to compare this to Australia
Australia has a somewhat similar system to your proposed one (though managed at the country-level rather than the state). We do have a split between retailers and the wholesaler (NBN - https://en.wikipedia.org/wiki/National_Broadband_Network) though, and I wonder if this split is something to avoid (though given the history of the internet connections here, maybe the split was inevitable...).
Comment by aragilar+anarcat
Comments on this page are closed.
Created . Edited .