Matrix notes

  1. blog

I have some concerns about Matrix (the protocol, not the movie that came out recently, although I do have concerns about that as well). I've been watching the project for a long time, and it seems more a promising alternative to many protocols like IRC, XMPP, and Signal.

This review may sound a bit negative, because it focuses on those concerns. I am the operator of an IRC network and people keep asking me to bridge it with Matrix. I have myself considered just giving up on IRC and converting to Matrix. This space is a living document exploring my research of that problem space. The TL;DR: is that no, I'm not setting up a bridge just yet, and I'm still on IRC.

This article was written over the course of the last three months, but I have been watching the Matrix project for years (my logs seem to say 2016 at least). The article is rather long. It will likely take you half an hour to read, so copy this over to your ebook reader, your tablet, or dead trees, and lean back and relax as I show you around the Matrix. Or, alternatively, just jump to a section that interest you, most likely the conclusion.

  1. Introduction to Matrix
  2. Security and privacy
    1. Data retention defaults
    2. GDPR in the federation
    3. Matrix.org privacy policy
    4. Metadata handling
    5. Amplification attacks on URL previews
  3. Moderation
    1. The mjolnir bot
    2. The command-line tool
    3. Rate limiting
    4. Fundamental federation problems
    5. Room admins
  4. Availability
    1. How this works in IRC
    2. User identities
    3. Rooms
    4. Spaces
    5. Home servers
    6. Delegations
  5. Performance
    1. Horizontal scalability
    2. Other implementations
    3. Latency
    4. Transport
  6. Usability
    1. Onboarding and workflow
    2. 5 minute clients evaluation
    3. Bots
    4. Working on Matrix
  7. Conclusion
    1. Looking back at history
    2. Where the federated services are in history
  8. Updates
    1. Other discussions
    2. External research on Matrix privacy, exhibit A
    3. External research on Matrix, exhibit B
  9. comments
    1. 2.1 data retention, 5.4 transport,
    2. 2.3 matrix.org policies, 2.5 url previews,
    3. 3.1 mjolnir bot, 3.2 cmd tool,
    4. 3.4 Fundamental federation problems
    5. 3.5 hijacking room admins
    6. 4. Availability
    7. 5. Performance
    8. 5.3 Latency,
    9. 6.1 onboarding
    10. 6.2 clients, 6.3 bots
    11. 6.4 specifiaction
    12. 7.1 History
    13. 7.2 Everyone's behind IRC

Introduction to Matrix

Matrix is an "open standard for interoperable, decentralised, real-time communication over IP. It can be used to power Instant Messaging, VoIP/WebRTC signalling, Internet of Things communication - or anywhere you need a standard HTTP API for publishing and subscribing to data whilst tracking the conversation history".

It's also (when compared with XMPP) "an eventually consistent global JSON database with an HTTP API and pubsub semantics - whilst XMPP can be thought of as a message passing protocol."

According to their FAQ, the project started in 2014, has about 20,000 servers, and millions of users. Matrix works over HTTPS but over a special port: 8448.

Security and privacy

I have some concerns about the security promises of Matrix. It's advertised as a "secure" with "E2E [end-to-end] encryption", but how does it actually work?

Data retention defaults

One of my main concerns with Matrix is data retention, which is a key part of security in a threat model where (for example) an hostile state actor wants to surveil your communications and can seize your devices.

On IRC, servers don't actually keep messages all that long: they pass them along to other servers and clients as fast as they can, only keep them in memory, and move on to the next message. There are no concerns about data retention on messages (and their metadata) other than the network layer. (I'm ignoring the issues with user registration, which is a separate, if valid, concern.) Obviously, an hostile server could log everything passing through it, but IRC federations are normally tightly controlled. So, if you trust your IRC operators, you should be fairly safe. Obviously, clients can (and often do, even if OTR is configured!) log all messages, but this is generally not the default. Irssi, for example, does not log by default. IRC bouncers are more likely to log to disk, of course, to be able to do what they do.

Compare this to Matrix: when you send a message to a Matrix homeserver, that server first stores it in its internal SQL database. Then it will transmit that message to all clients connected to that server and room, and to all other servers that have clients connected to that room. Those remote servers, in turn, will keep a copy of that message and all its metadata in their own database, by default forever. On encrypted rooms those messages are encrypted, but not their metadata.

There is a mechanism to expire entries in Synapse, but it is not enabled by default. So one should generally assume that a message sent on Matrix is never expired.

Update: one key thing with Matrix is that it's not a linear storage of discussions like IRC or email. It's actually an directed acyclic graph that servers try to continuously merged. And as such it's actually really hard to delete past history. Even with expiration enabled, that's advisory for other servers and some room parameters (like join/part/bans) must be kept forever. See this lengthy discussion for details.

GDPR in the federation

But even if that setting was enabled by default, how do you control it? This is a fundamental problem of the federation: if any user is allowed to join a room (which is the default), those user's servers will log all content and metadata from that room. That includes private, one-on-one conversations, since those are essentially rooms as well.

In the context of the GDPR, this is really tricky: who is the responsible party (known as the "data controller") here? It's basically any yahoo who fires up a home server and joins a room.

In a federated network, one has to wonder whether GDPR enforcement is even possible at all. But in Matrix in particular, if you want to enforce your right to be forgotten in a given room, you would have to:

  1. enumerate all the users that ever joined the room while you were there
  2. discover all their home servers
  3. start a GDPR procedure against all those servers

I recognize this is a hard problem to solve while still keeping an open ecosystem. But I believe that Matrix should have much stricter defaults towards data retention than right now. Message expiry should be enforced by default, for example. (Note that there are also redaction policies that could be used to implement part of the GDPR automatically, see the privacy policy discussion below on that.)

Also keep in mind that, in the brave new peer-to-peer world that Matrix is heading towards, the boundary between server and client is likely to be fuzzier, which would make applying the GDPR even more difficult.

Update: this comment links to this post (in german) which apparently studied the question and concluded that Matrix is not GDPR-compliant.

In fact, maybe Synapse should be designed so that there's no configurable flag to turn off data retention. A bit like how most system loggers in UNIX (e.g. syslog) come with a log retention system that typically rotate logs after a few weeks or month. Historically, this was designed to keep hard drives from filling up, but it also has the added benefit of limiting the amount of personal information kept on disk in this modern day. (Arguably, syslog doesn't rotate logs on its own, but, say, Debian GNU/Linux, as an installed system, does have log retention policies well defined for installed packages, and those can be discussed. And "no expiry" is definitely a bug.

Matrix.org privacy policy

When I first looked at Matrix, five years ago, Element.io was called Riot.im and had a rather dubious privacy policy:

We currently use cookies to support our use of Google Analytics on the Website and Service. Google Analytics collects information about how you use the Website and Service.

[...]

This helps us to provide you with a good experience when you browse our Website and use our Service and also allows us to improve our Website and our Service.

When I asked Matrix people about why they were using Google Analytics, they explained this was for development purposes and they were aiming for velocity at the time, not privacy (paraphrasing here).

They also included a "free to snitch" clause:

If we are or believe that we are under a duty to disclose or share your personal data, we will do so in order to comply with any legal obligation, the instructions or requests of a governmental authority or regulator, including those outside of the UK.

Those are really broad terms, above and beyond what is typically expected legally.

Like the current retention policies, such user tracking and ... "liberal" collaboration practices with the state set a bad precedent for other home servers.

Thankfully, since the above policy was published (2017), the GDPR was "implemented" (2018) and it seems like both the Element.io privacy policy and the Matrix.org privacy policy have been somewhat improved since.

Notable points of the new privacy policies:

I'm not super happy with all the trackers they have on the Element platform, but then again you don't have to use that service. Your favorite homeserver (assuming you are not on Matrix.org) probably has their own Element deployment, hopefully without all that garbage.

Overall, this is all a huge improvement over the previous privacy policy, so hats off to the Matrix people for figuring out a reasonable policy in such a tricky context. I particularly like this bit:

We will forget your copy of your data upon your request. We will also forward your request to be forgotten onto federated homeservers. However - these homeservers are outside our span of control, so we cannot guarantee they will forget your data.

It's great they implemented those mechanisms and, after all, if there's an hostile party in there, nothing can prevent them from using screenshots to just exfiltrate your data away from the client side anyways, even with services typically seen as more secure, like Signal.

As an aside, I also appreciate that Matrix.org has a fairly decent code of conduct, based on the TODO CoC which checks all the boxes in the geekfeminism wiki.

Metadata handling

Overall, privacy protections in Matrix mostly concern message contents, not metadata. In other words, who's talking with who, when and from where is not well protected. Compared to a tool like Signal, which goes through great lengths to anonymize that data with features like private contact discovery, disappearing messages, sealed senders, and private groups, Matrix is definitely behind. (Note: there is an issue open about message lifetimes in Element since 2020, but it's not at even at the MSC stage yet.)

This is a known issue (opened in 2019) in Synapse, but this is not just an implementation issue, it's a flaw in the protocol itself. Home servers keep join/leave of all rooms, which gives clear text information about who is talking to. Synapse logs may also contain privately identifiable information that home server admins might not be aware of in the first place. Those log rotation policies are separate from the server-level retention policy, which may be confusing for a novice sysadmin.

Combine this with the federation: even if you trust your home server to do the right thing, the second you join a public room with third-party home servers, those ideas kind of get thrown out because those servers can do whatever they want with that information. Again, a problem that is hard to solve in any federation.

To be fair, IRC doesn't have a great story here either: any client knows not only who's talking to who in a room, but also typically their client IP address. Servers can (and often do) obfuscate this, but often that obfuscation is trivial to reverse. Some servers do provide "cloaks" (sometimes automatically), but that's kind of a "slap-on" solution that actually moves the problem elsewhere: now the server knows a little more about the user.

Overall, I would worry much more about a Matrix home server seizure than a IRC or Signal server seizure. Signal does get subpoenas, and they can only give out a tiny bit of information about their users: their phone number, and their registration, and last connection date. Matrix carries a lot more information in its database, and a lot of that information is carried permanently, think "blockchain-level" permanence.

Amplification attacks on URL previews

I (still!) run an Icecast server and sometimes share links to it on IRC which, obviously, also ends up on (more than one!) Matrix home servers because some people connect to IRC using Matrix. This, in turn, means that Matrix will connect to that URL to generate a link preview.

I feel this outlines a security issue, especially because those sockets would be kept open seemingly forever. I tried to warn the Matrix security team but somehow, I don't think this issue was taken very seriously. Here's the disclosure timeline:

There are a couple of problems here:

  1. the bug was publicly disclosed in September 2020, and not considered a security issue until I notified them, and even then, I had to insist

  2. no clear disclosure policy timeline was proposed or seems established in the project (there is a security disclosure policy but it doesn't include any predefined timeline)

  3. I wasn't informed of the disclosure

  4. the actual solution is a size limit (10MB, already implemented), a time limit (30 seconds, implemented in PR 11784), and a content type allow list (HTML, "media" or JSON, implemented in PR 11936), and I'm not sure it's adequate

  5. (pure vanity:) I did not make it to their Hall of fame

I'm not sure those solutions are adequate because they all seem to assume a single home server will pull that one URL for a little while then stop. But in a federated network, many (possibly thousands) home servers may be connected in a single room at once. If an attacker drops a link into such a room, all those servers would connect to that link all at once. This is an amplification attack: a small amount of traffic will generate a lot more traffic to a single target. It doesn't matter there are size or time limits: the amplification is what matters here.

It should also be noted that clients that generate link previews have more amplification because they are more numerous than servers. And of course, the default Matrix client (Element) does generate link previews as well.

That said, this is possibly not a problem specific to Matrix: any federated service that generates link previews may suffer from this.

I'm honestly not sure what the solution is here. Maybe moderation? Maybe link previews are just evil? All I know is there was this weird bug in my Icecast server and I tried to ring the bell about it, and it feels it was swept under the rug. Somehow I feel this is bound to blow up again in the future, even with the current mitigation.

Update: thinking about this again recently made me realize the problem is not whether or not clients or server generate the link previews. That's missing the core of the question: the real question is whether the sender or the receiver generates link previews. In Signal, for example, the sender generates the link preview, which removes a whole class of attacks, including the above denial-of-service attack, but also de-anonymization attacks (which I haven't mentioned above, but that is still an issue in Matrix of course).

In other words, right now, still, you can find out the IP address of any Matrix user (or their home server, or both, depending on their configuration) just by sharing a link with them. Neat no? (Of course, a homeserver's IP address is normally discoverable, but there might be a space in the future where it isn't or should be hidden behind a proxy, for example. That still feels like a security issue, and that still isn't something Matrix seem to care about.)

Mastodon has the same problem.

Moderation

In Matrix like elsewhere, Moderation is a hard problem. There is a detailed moderation guide and much of this problem space is actively worked on in Matrix right now. A fundamental problem with moderating a federated space is that a user banned from a room can rejoin the room from another server. This is why spam is such a problem in Email, and why IRC networks have stopped federating ages ago (see the IRC history for that fascinating story).

Update: it's much worse than I imagined. Abuse is really easy on Matrix, with simple attacks like flood-joining permanently crippling rooms. According to Why Not Matrix this is a problem that's been known for years, a fundamental design flaw that's barely acknowledged, but usually swept under the rug by Matrix people.

The mjolnir bot

The mjolnir moderation bot is designed to help with some of those things. It can kick and ban users, redact all of a user's message (as opposed to one by one), all of this across multiple rooms. It can also subscribe to a federated block list published by matrix.org to block known abusers (users or servers). Bans are pretty flexible and can operate at the user, room, or server level.

Matrix people suggest making the bot admin of your channels, because you can't take back admin from a user once given.

The command-line tool

There's also a new command line tool designed to do things like:

This tool and Mjolnir are based on the admin API built into Synapse.

Rate limiting

Synapse has pretty good built-in rate-limiting which blocks repeated login, registration, joining, or messaging attempts. It may also end up throttling servers on the federation based on those settings.

Fundamental federation problems

Because users joining a room may come from another server, room moderators are at the mercy of the registration and moderation policies of those servers. Matrix is like IRC's +R mode ("only registered users can join") by default, except that anyone can register their own homeserver, which makes this limited.

Server admins can block IP addresses and home servers, but those tools are not easily available to room admins. There is an API (m.room.server_acl in /devtools) but it is not reliable (thanks Austin Huang for the clarification).

Matrix has the concept of guest accounts, but it is not used very much, and virtually no client or homeserver supports it. This contrasts with the way IRC works: by default, anyone can join an IRC network even without authentication. Some channels require registration, but in general you are free to join and look around (until you get blocked, of course).

I have seen anecdotal evidence (CW: Twitter, nitter link) that "moderating bridges is hell", and I can imagine why. Moderation is already hard enough on one federation, when you bridge a room with another network, you inherit all the problems from that network but without the entire abuse control tools from the original network's API...

Room admins

Matrix, in particular, has the problem that room administrators (which have the power to redact messages, ban users, and promote other users) are bound to their Matrix ID which is, in turn, bound to their home servers. This implies that a home server administrators could (1) impersonate a given user and (2) use that to hijack the room. So in practice, the home server is the trust anchor for rooms, not the user themselves.

That said, if server B administrator hijack user joe on server B, they will hijack that room on that specific server. This will not (necessarily) affect users on the other servers, as servers could refuse parts of the updates or ban the compromised account (or server).

It does seem like a major flaw that room credentials are bound to Matrix identifiers, as opposed to the E2E encryption credentials. In an encrypted room even with fully verified members, a compromised or hostile home server can still take over the room by impersonating an admin. That admin (or even a newly minted user) can then send events or listen on the conversations.

This is even more frustrating when you consider that Matrix events are actually signed and therefore have some authentication attached to them, acting like some sort of Merkle tree (as it contains a link to previous events). That signature, however, is made from the homeserver PKI keys, not the client's E2E keys, which makes E2E feel like it has been "bolted on" later.

Update: note that this flaw has actually been called a "serious vulnerability" by Ars Technica, based on a paper from researchers that confirmed the flaw I hinted at above. The response from Matrix.org has been rather underwhelming, with many issues still unaddressed.

Availability

While Matrix has a strong advantage over Signal in that it's decentralized (so anyone can run their own homeserver,), I couldn't find an easy way to run a "multi-primary" setup, or even a "redundant" setup (even if with a single primary backend), short of going full-on "replicate PostgreSQL and Redis data", which is not typically for the faint of heart.

How this works in IRC

On IRC, it's quite easy to setup redundant nodes. All you need is:

  1. a new machine (with it's own public address with an open port)

  2. a shared secret (or certificate) between that machine and an existing one on the network

  3. a connect {} block on both servers

That's it: the node will join the network and people can connect to it as usual and share the same user/namespace as the rest of the network. The servers take care of synchronizing state: you do not need to worry about replicating a database server.

(Now, experienced IRC people will know there's a catch here: IRC doesn't have authentication built in, and relies on "services" which are basically bots that authenticate users (I'm simplifying, don't nitpick). If that service goes down, the network still works, but then people can't authenticate, and they can start doing nasty things like steal people's identity if they get knocked offline. But still: basic functionality still works: you can talk in rooms and with users that are on the reachable network.)

User identities

Matrix is more complicated. Each "home server" has its own identity namespace: a specific user (say @anarcat:matrix.org) is bound to that specific home server. If that server goes down, that user is completely disconnected. They could register a new account elsewhere and reconnect, but then they basically lose all their configuration: contacts, joined channels are all lost.

(Also notice how the Matrix IDs don't look like a typical user address like an email in XMPP. They at least did their homework and got the allocation for the scheme.)

Rooms

Users talk to each other in "rooms", even in one-to-one communications. (Rooms are also used for other things like "spaces", they're basically used for everything, think "everything is a file" kind of tool.) For rooms, home servers act more like IRC nodes in that they keep a local state of the chat room and synchronize it with other servers. Users can keep talking inside a room if the server that originally hosts the room goes down. Rooms can have a local, server-specific "alias" so that, say, #room:matrix.org is also visible as #room:example.com on the example.com home server. Both addresses refer to the same room underlying room.

(Finding this in the Element settings is not obvious though, because that "alias" are actually called a "local address" there. So to create such an alias (in Element), you need to go in the room settings' "General" section, "Show more" in "Local address", then add the alias name (e.g. foo), and then that room will be available on your example.com homeserver as #foo:example.com.)

So a room doesn't belong to a server, it belongs to the federation, and anyone can join the room from any server (if the room is public, or if invited otherwise). You can create a room on server A and when a user from server B joins, the room will be replicated on server B as well. If server A fails, server B will keep relaying traffic to connected users and servers.

A room is therefore not fundamentally addressed with the above alias. Instead, it has a internal Matrix ID, which basically a random string. It has a server name attached to it, but that was made just to avoid collisions. That can get a little confusing. For example, the #fractal:gnome.org room is an alias on the gnome.org server, but the room ID is !hwiGbsdSTZIwSRfybq:matrix.org. That's because the room was created on matrix.org, but the preferred branding is gnome.org now.

As an aside, rooms, by default, live forever, even after the last user quits. There's an admin API to delete rooms and a tombstone event to redirect to another one, but neither have a GUI yet. The latter is part of MSC1501 ("Room version upgrades") which allows a room admin to close a room, with a message and a pointer to another room.

Spaces

Discovering rooms can be tricky: there is a per-server room directory, but Matrix.org people are trying to deprecate it in favor of "Spaces". Room directories were ripe for abuse: anyone can create a room, so anyone can show up in there. It's possible to restrict who can add aliases, but anyways directories were seen as too limited.

In contrast, a "Space" is basically a room that's an index of other rooms (including other spaces), so existing moderation and administration mechanism that work in rooms can (somewhat) work in spaces as well. This enables a room directory that works across federation, regardless on which server they were originally created.

New users can be added to a space or room automatically in Synapse. (Existing users can be told about the space with a server notice.) This gives admins a way to pre-populate a list of rooms on a server, which is useful to build clusters of related home servers, providing some sort of redundancy, at the room -- not user -- level.

Home servers

So while you can workaround a home server going down at the room level, there's no such thing at the home server level, for user identities. So if you want those identities to be stable in the long term, you need to think about high availability. One limitation is that the domain name (e.g. matrix.example.com) must never change in the future, as renaming home servers is not supported.

The documentation used to say you could "run a hot spare" but that has been removed. Last I heard, it was not possible to run a high-availability setup where multiple, separate locations could replace each other automatically. You can have high performance setups where the load gets distributed among workers, but those are based on a shared database (Redis and PostgreSQL) backend.

So my guess is it would be possible to create a "warm" spare server of a matrix home server with regular PostgreSQL replication, but that is not documented in the Synapse manual. This sort of setup would also not be useful to deal with networking issues or denial of service attacks, as you will not be able to spread the load over multiple network locations easily. Redis and PostgreSQL heroes are welcome to provide their multi-primary solution in the comments. In the meantime, I'll just point out this is a solution that's handled somewhat more gracefully in IRC, by having the possibility of delegating the authentication layer.

Update: this was previously undocumented, but not only can you scale the frontend workers to multiple hosts, you can also shard the backend so that tables are distributed across multiple database hots. This has been documented only on 2022-07-11, weeks after this article was written, so you will forgive me for that omission, hopefully. Obviously, this doesn't resolve the "high availability" scenario since you still have a central server for that data, but it might help resolving performance problems for very large instances.

Delegations

If you do not want to run a Matrix server yourself, it's possible to delegate the entire thing to another server. There's a server discovery API which uses the .well-known pattern (or SRV records, but that's "not recommended" and a bit confusing) to delegate that service to another server. Be warned that the server still needs to be explicitly configured for your domain. You can't just put:

{ "m.server": "matrix.org:443" }

... on https://example.com/.well-known/matrix/server and start using @you:example.com as a Matrix ID. That's because Matrix doesn't support "virtual hosting" and you'd still be connecting to rooms and people with your matrix.org identity, not example.com as you would normally expect. This is also why you cannot rename your home server.

The server discovery API is what allows servers to find each other. Clients, on the other hand, use the client-server discovery API: this is what allows a given client to find your home server when you type your Matrix ID on login.

Performance

The high availability discussion brushed over the performance of Matrix itself, but let's now dig into that.

Horizontal scalability

There were serious scalability issues of the main Matrix server, Synapse, in the past. So the Matrix team has been working hard to improve its design. Since Synapse 1.22 the home server can horizontally scale to multiple workers (see this blog post for details) which can make it easier to scale large servers.

Other implementations

There are other promising home servers implementations from a performance standpoint (dendrite, Golang, entered beta in late 2020; conduit, Rust, beta; others), but none of those are feature-complete so there's a trade-off to be made there. Synapse is also adding a lot of feature fast, so it's an open question whether the others will ever catch up. (I have heard that Dendrite might actually surpass Synapse in features within a few years, which would put Synapse in a more "LTS" situation.)

Latency

Matrix can feel slow sometimes. For example, joining the "Matrix HQ" room in Element (from matrix.debian.social) takes a few minutes and then fails. That is because the home server has to sync the entire room state when you join the room. There was promising work on this announced in the lengthy 2021 retrospective, and some of that work landed (partial sync) in the 1.53 release already. Other improvements coming include sliding sync, lazy loading over federation, and fast room joins. So that's actually something that could be fixed in the fairly short term.

But in general, communication in Matrix doesn't feel as "snappy" as on IRC or even Signal. It's hard to quantify this without instrumenting a full latency test bed (for example the tools I used in the terminal emulators latency tests), but even just typing in a web browser feels slower than typing in a xterm or Emacs for me.

Even in conversations, I "feel" people don't immediately respond as fast. In fact, this could be an interesting double-blind experiment to make: have people guess whether they are talking to a person on Matrix, XMPP, or IRC, for example. My theory would be that people could notice that Matrix users are slower, if only because of the TCP round-trip time each message has to take.

Transport

Some courageous person actually made some tests of various messaging platforms on a congested network. His evaluation was basically:

So that was interesting. I suspect IRC would have also fared better, but that's just a feeling.

Other improvements to the transport layer include support for websocket and the CoAP proxy work from 2019 (targeting 100bps links), but both seem stalled at the time of writing. The Matrix people have also announced the pinecone p2p overlay network which aims at solving large, internet-scale routing problems. See also this talk at FOSDEM 2022.

Update: there is a formal proposal (MSC3079) about a low-bandwidth, UDP-based transport, but it's been stuck at that stage for a few years now, because of security issues inherent to the UDP protocol, at first glance. There's also a golang implementation.

Usability

Onboarding and workflow

The workflow for joining a room, when you use Element web, is not great:

  1. click on a link in a web browser
  2. land on (say) https://matrix.to/#/#matrix-dev:matrix.org
  3. offers "Element", yeah that's sounds great, let's click "Continue"
  4. land on https://app.element.io/#/room%2F%23matrix-dev%3Amatrix.org and then you need to register, aaargh

As you might have guessed by now, there is a specification to solve this, but web browsers need to adopt it as well, so that's far from actually being solved. At least browsers generally know about the matrix: scheme, it's just not exactly clear what they should do with it, especially when the handler is just another web page (e.g. Element web).

In general, when compared with tools like Signal or WhatsApp, Matrix doesn't fare so well in terms of user discovery. I probably have some of my normal contacts that have a Matrix account as well, but there's really no way to know. It's kind of creepy when Signal tells you "this person is on Signal!" but it's also pretty cool that it works, and they actually implemented it pretty well.

Registration is also less obvious: in Signal, the app confirms your phone number automatically. It's friction-less and quick. In Matrix, you need to learn about home servers, pick one, register (with a password! aargh!), and then setup encryption keys (not default), etc. It's a lot more friction.

And look, I understand: giving away your phone number is a huge trade-off. I don't like it either. But it solves a real problem and makes encryption accessible to a ton more people. Matrix does have "identity servers" that can serve that purpose, but I don't feel confident sharing my phone number there. It doesn't help that the identity servers don't have private contact discovery: giving them your phone number is a more serious security compromise than with Signal.

There's a catch-22 here too: because no one feels like giving away their phone numbers, no one does, and everyone assumes that stuff doesn't work anyways. Like it or not, Signal forcing people to divulge their phone number actually gives them critical mass that means actually a lot of my relatives are on Signal and I don't have to install crap like WhatsApp to talk with them.

5 minute clients evaluation

Throughout all my tests I evaluated a handful of Matrix clients, mostly from Flathub because almost none of them are packaged in Debian.

Right now I'm using Element, the flagship client from Matrix.org, in a web browser window, with the PopUp Window extension. This makes it look almost like a native app, and opens links in my main browser window (instead of a new tab in that separate window), which is nice. But I'm tired of buying memory to feed my web browser, so this indirection has to stop. Furthermore, I'm often getting completely logged off from Element, which means re-logging in, recovering my security keys, and reconfiguring my settings. That is extremely annoying.

Coming from Irssi, Element is really "GUI-y" (pronounced "gooey"). Lots of clickety happening. To mark conversations as read, in particular, I need to click-click-click on all the tabs that have some activity. There's no "jump to latest message" or "mark all as read" functionality as far as I could tell. In Irssi the former is built-in (alt-a) and I made a custom /READ command for the latter:

/ALIAS READ script exec \$_->activity(0) for Irssi::windows

And yes, that's a Perl script in my IRC client. I am not aware of any Matrix client that does stuff like that, except maybe Weechat, if we can call it a Matrix client, or Irssi itself, now that it has a Matrix plugin (!).

As for other clients, I have looked through the Matrix Client Matrix (confusing right?) to try to figure out which one to try, and, even after selecting Linux as a filter, the chart is just too wide to figure out anything. So I tried those, kind of randomly:

Unfortunately, I lost my notes on those, I don't actually remember which one did what. I still have a session open with Mirage, so I guess that means it's the one I preferred, but I remember they were also all very GUI-y.

Maybe I need to look at weechat-matrix or gomuks. At least Weechat is scriptable so I could continue playing the power-user. Right now my strategy with messaging (and that includes microblogging like Twitter or Mastodon) is that everything goes through my IRC client, so Weechat could actually fit well in there. Going with gomuks, on the other hand, would mean running it in parallel with Irssi or ... ditching IRC, which is a leap I'm not quite ready to take just yet.

Oh, and basically none of those clients (except Nheko and Element) support VoIP, which is still kind of a second-class citizen in Matrix. It does not support large multimedia rooms, for example: Jitsi was used for FOSDEM instead of the native videoconferencing system.

Update: I'm now (2024) using FluffyChat on desktop (through Flatpak) and Android (through F-Droid).

Bots

This falls a little aside the "usability" section, but I didn't know where to put this... There's a few Matrix bots out there, and you are likely going to be able to replace your existing bots with Matrix bots. It's true that IRC has a long and impressive history with lots of various bots doing various things, but given how young Matrix is, there's still a good variety:

One thing I haven't found an equivalent for is Debian's MeetBot. There's an archive bot but it doesn't have topics or a meeting chair, or HTML logs.

Update: it's not a bot but progval/matrix2051 is quite interesting for me, as a long-time IRC user: it's a homeserver gateway that presents itself as an IRC server. So you can treat Matrix as one big weird IRC server. Main limitation is DMs are basically broken, but lack of TLS also keeps it from being useful as a drop-in replacement for migrating an existing IRC network.

Working on Matrix

As a developer, I find Matrix kind of intimidating. The specification is huge. The official specification itself looks somewhat digestable: it's only 6 APIs so that looks, at first, kind of reasonable. But whenever you start asking complicated questions about Matrix, you quickly fall into the Matrix Spec Change specification (which, yes, is a separate specification). And there are literally hundreds of MSCs flying around. It's hard to tell what's been adopted and what hasn't, and even harder to figure out if your specific client has implemented it.

(One trendy answer to this problem is to "rewrite it in rust": Matrix are working on implementing a lot of those specifications in a matrix-rust-sdk that's designed to take the implementation details away from users.)

Just taking the latest weekly Matrix report, you find that three new MSCs proposed, just last week! There's even a graph that shows the number of MSCs is progressing steadily, at 600+ proposals total, with the majority (300+) "new". I would guess the "merged" ones are at about 150.

That's a lot of text which includes stuff like 3D worlds which, frankly, I don't think you should be working on when you have such important security and usability problems. (The internet as a whole, arguably, doesn't fare much better. RFC600 is a really obscure discussion about "INTERFACING AN ILLINOIS PLASMA TERMINAL TO THE ARPANET". Maybe that's how many MSCs will end up as well, left forgotten in the pits of history.)

And that's the thing: maybe the Matrix people have a different objective than I have. They want to connect everything to everything, and make Matrix a generic transport for all sorts of applications, including virtual reality, collaborative editors, and so on.

I just want secure, simple messaging. Possibly with good file transfers, and video calls. That it works with existing stuff is good, and it should be federated to remove the "Signal point of failure". So I'm a bit worried with the direction all those MSCs are taking, especially when you consider that clients other than Element are still struggling to keep up with basic features like end-to-end encryption or room discovery, never mind voice or spaces...

Conclusion

Overall, Matrix is somehow in the space XMPP was a few years ago. It has a ton of features, pretty good clients, and a large community. It seems to have gained some of the momentum that XMPP has lost. It could be seen as having the most potential to replace Signal if something bad would happen to it (like, I don't know, getting banned or going nuts with cryptocurrency)...

But it's really not there yet, and I don't see Matrix trying to get there either, which is a bit worrisome.

Looking back at history

I'm also worried that we are repeating the errors of the past. The history of federated services is really fascinating:. IRC, FTP, HTTP, and SMTP were all created in the early days of the internet, and are all still around (except, arguably, FTP, which was removed from major browsers recently). All of them had to face serious challenges in growing their federation.

IRC had numerous conflicts and forks, both at the technical level but also at the political level. The history of IRC is really something that anyone working on a federated system should study in detail, because they are bound to make the same mistakes if they are not familiar with it. The "short" version is:

(The numbers were taken from the Wikipedia page and Netsplit.de. Note that I also include other networks launch in parenthesis for context.)

Pretty dramatic, don't you think? Eventually, somehow, IRC became irrelevant for most people: few people are even aware of it now. With less than a million users active, it's smaller than Mastodon, XMPP, or Matrix at this point.1 If I were to venture a guess, I'd say that infighting, lack of a standardization body, and a somewhat annoying protocol meant the network could not grow. It's also possible that the decentralised yet centralised structure of IRC networks limited their reliability and growth.

But large social media companies have also taken over the space: observe how IRC numbers peak around the time the wave of large social media companies emerge, especially Facebook (2.9B users!!) and Twitter (400M users).

Where the federated services are in history

Right now, Matrix, and Mastodon (and email!) are at the "pre-EFnet" stage: anyone can join the federation. Mastodon has started working on a global block list of fascist servers which is interesting, but it's still an open federation. Right now, Matrix is totally open, but matrix.org publishes a (federated) block list of hostile servers (#matrix-org-coc-bl:matrix.org, yes, of course it's a room).

Interestingly, Email is also in that stage, where there are block lists of spammers, and it's a race between those blockers and spammers. Large email providers, obviously, are getting closer to the EFnet stage: you could consider they only accept email from themselves or between themselves. It's getting increasingly hard to deliver mail to Outlook and Gmail for example, partly because of bias against small providers, but also because they are including more and more machine-learning tools to sort through email and those systems are, fundamentally, unknowable. It's not quite the same as splitting the federation the way EFnet did, but the effect is similar.

HTTP has somehow managed to live in a parallel universe, as it's technically still completely federated: anyone can start a web server if they have a public IP address and anyone can connect to it. The catch, of course, is how you find the darn thing. Which is how Google became one of the most powerful corporations on earth, and how they became the gatekeepers of human knowledge online.

I have only briefly mentioned XMPP here, and my XMPP fans will undoubtedly comment on that, but I think it's somewhere in the middle of all of this. It was co-opted by Facebook and Google, and both corporations have abandoned it to its fate. I remember fondly the days where I could do instant messaging with my contacts who had a Gmail account. Those days are gone, and I don't talk to anyone over Jabber anymore, unfortunately. And this is a threat that Matrix still has to face.

It's also the threat Email is currently facing. On the one hand corporations like Facebook want to completely destroy it and have mostly succeeded: many people just have an email account to register on things and talk to their friends over Instagram or (lately) TikTok (which, I know, is not Facebook, but they started that fire).

On the other hand, you have corporations like Microsoft and Google who are still using and providing email services — because, frankly, you still do need email for stuff, just like fax is still around — but they are more and more isolated in their own silo. At this point, it's only a matter of time they reach critical mass and just decide that the risk of allowing external mail coming in is not worth the cost. They'll simply flip the switch and work on an allow-list principle. Then we'll have closed the loop and email will be dead, just like IRC is "dead" now.

I wonder which path Matrix will take. Could it liberate us from these vicious cycles?

Updates

Other discussions

This generated some discussions on lobste.rs and Hacker News.

External research on Matrix privacy, exhibit A

I have also found a research on Matrix's privacy which touches on some of the issues outlined here. Unfortunately, it seems that research is rather heavily biased; it was written by the libremonde.org people who work on the Grid protocol, a fork of Matrix that seems to be pretty much dead anyways.

I have found that research through this post on hackea.org that is disturbingly badly sourced, with multiple sections containing precious statements like:

We have not seriously investigated those disturbing pieces of information, so let’s consider them as FUD.

We have not investigated further, so let’s consider there is no freedom issue at all

We have not read it. We do not think it is worth wasting more time.

So basically, they read a report critical of matrix, rumours about Amdocs, panicked and ran away screaming without reading anything else about the problem. I hesitated in even linking to the article here precisely because it was exactly the kind of FUD I do not think it's worth mentioning, but I also thought it was important to criticise the lack of verification in the article, and the source of the aforementioned research, possibly to defuse further comments linking to it as well.

Not that people read articles thoroughly before commenting anyways, of course.

External research on Matrix, exhibit B

This one is more damning and makes me reconsider using Matrix at all. I encourage anyone who is planning to do anything serious with Matrix to take a long, deep look at [why not matrix][].

It certainly gave me food for thought. As a basic example: I have sent myself an attachment over FluffyChat from my phone, in a (failed) attempt at dumping a file from my phone to "the internet" at a print shop. (It didn't work: I couldn't find the link to share.) But the file did get posted on the homeserver, so there was a link. Problem though: when I deleted the message from FluffyChat, the attachment was still there, and will presumably be present forever.

Ouch. Watch out what you upload to Matrix.

  1. According to Wikipedia, there are currently about 500 distinct IRC networks operating, on about 1,000 servers, serving over 250,000 users. In contrast, Mastodon seems to be around 5 million users, Matrix.org claimed at FOSDEM 2021 to have about 28 million globally visible accounts, and Signal lays claim to over 40 million souls. XMPP claims to have "millions" of users on the xmpp.org homepage but the FAQ says they don't actually know. On the proprietary silo side of the fence, this page says

    • Facebook: 2.9 billion users
    • WhatsApp: 2B
    • Instagram: 1.4B
    • TikTok: 1B
    • Snapchat: 500M
    • Pinterest: 480M
    • Twitter: 397M

    Notable omission from that list: Youtube, with its mind-boggling 2.6 billion users...

    Those are not the kind of numbers you just "need to convince a brother or sister" to grow the network...

    ↩
RSS
Some comments and whatnot from another Montréaler

Hello there. Some comments from an active Matrix user (aussi un Montréalais, but I don't work for matrix.org or New Vector), from top to bottom. (For other readers: they're AFAIK so please correct me if necessary.)

Element 2.2.1: mentions many more third parties

Writing all the third parties together is quite misleading and you should definitely separate them. Specifically, according to my understanding of the text:

  • Twillo, if you register with a phone number on an EMS-operated homeserver
  • Stripe and Quaderno if you're a paying customer
  • LinkedIn, Twitter, Google, Outplay, Salesforce, and Pipedrive if you click on an ad of Element from a third-party platform (presumably they're not used if you land on Element directly, so maybe they shouldn't be included)
  • Hubspot, Matomo (selfhosted) and Posthog for website analytics

So I don't think they're applicable to the client.

Whether privacy policies are actually followed is a different thing, but assumptions need to be clearly indicated, in my opinion.

As an aside, I also appreciate that Matrix.org has a fairly decent code of conduct

The enforcement of it (bans in #matrix-org-coc-bl:matrix.org whose reason is code of conduct violations) is commonly believed to be dubious at times. Specifically, they ban people (including those not exhibiting bad faith) too easily (often without warning) and there are no real appeal process (AFAIK abuse@matrix.org doesn't manually respond to most emails). Since #matrix-org-coc-bl:matrix.org is also used outside of official Matrix rooms (it applies to public communities directly related to EMS-operated homeservers as well, eg. FOSDEM, Arch Linux, etc.), there definitely should be some constraints w/r/t the use of bans.

Third-party homeservers seem to be much better moderated.

The mjolnir bot

Unfortunately, in practice, Mjolnir requires a homeserver to be run (given that the bot expects to be free of ratelimit, which requires an exception in the homeserver, which is unlikely to be granted to someone who's not related to the homeserver's administration). This makes it inaccessible to people who cannot run a homeserver (and cannot trust one who can to handle moderation). See also here.

Server admins can block IP addresses and home servers, but those tools are not currently available to room admins.

Room admins can block homeservers through ACL, it's not intuitive (/devtools => Explore Room State => m.room.server_acl) but it is indeed available. But ACL itself is not reliable.

Also, Mjolnir user bans support wildcard (@*:example.com).

Matrix has the concept of guest accounts, but it is not used very much, and virtually no client supports it.

Element does, but virtually no homeserver supports it (since it is easily abusable).

as servers could refuse parts of the updates

pretty sure you can't do that without affecting all future updates

I have heard anecdotal evidence that "moderating bridges is hell"

Since bridge puppets are still users, you could ban them or redact their messages. If the platform can be accessed through multiple bridges running the same bridge software, you could use Mjolnir to ban @bridge_username:* with the server name as wildcard. Of course, that doesn't prevent the situation where multiple public bridges exist that run different software and do not require explicit approval (eg. Bifrost for XMPP), but that's rare.

and then that room will be available on your example.com homeserver as #foo:example.com

But the room is already available as, say, #foo:matrix.org, and if that is the selected main alias then that's what's gonna be shown even when you add the room to example.com's room directory. So local aliases are only for someone on your homeserver to link this room, and non-main public aliases are only for someone on any homeserver to link this room, which means neither have any actual uses other than 1. vanity, and 2. to act as a redundancy in case all other public aliases fail (due to homeserver outage).

tombstone event

It does have a GUI in Element:

  • /upgraderoom
  • For rooms before version 9, room settings => Security & Privacy => "Space members" has an "upgrade required" which is clickable if you have the permission to upgrade it (IIRC)

but Matrix.org people are trying to deprecate it in favor of "Spaces"

Citation required. Also Spaces are rooms and so they can also be included in room directories.

New users can be added to a space or room automatically in Synapse

In public homeservers, this may leak account age.

It's possible to restrict who can add aliases

Only public aliases (local aliases are unrestricted afaik). Also they're not required for listing on room directories.

I have heard that Dendrite might actually surpass Synapse in features within a few years

Given that this is a thing, it is likely to be the goal.

In Matrix, you need to learn about home servers, pick one,

How Element markets effectively forces everyone onto matrix.org (or EMS) and that's a problem

register (with a password! aargh!)

Don't you also have a password on Signal?

but I don't feel confident sharing my phone number there

You could share email addresses, but yes, I get your point.

Use of identity servers became opt-in in late? 2020 amid concerns that it makes Riot.im (then) a spyware by forcing a call-home. In fact a selling point of Matrix is the non-requirement of email or phone number (if applicable; but it is obvious that this will lead to abuse).

It does not support large multimedia rooms

Rooms with more than 2 users don't have native VoIP yet.

Working on Matrix

That's what happens when the same people effectively own both the protocol and the first client... But then you said of IRC, that

If I were to venture a guess, I'd say that infighting, lack of a standardization body, and a somewhat annoying protocol meant the network could not grow.

So it could also be an advantage that Matrix has a standardization body from the get-go (whether the body itself is good is a different question).

I just want secure, simple messaging. Possibly with good file transfers, and video calls.

I don't think that has been Matrix's goal from the get-go, though you could say they're now working towards that.

For me, Matrix is still only for two purposes:

  1. For individuals, to run a community (replacing Telegram and Discord), and
  2. For organizational communication (replacing Slack and MS Teams).

Sure, interoperability (which Matrix has probably lobbied for), but the cost of bridging is still quite high.

Mastodon has started working on a global block list of fascist servers

Some (certainly not mainstream) consider "not blocking abusive servers" as grounds for blocking so it's already happening on the fediverse.

but matrix.org publishes a (federated) block list of hostile servers

Element effectively encourages the use of blocklists, so abuse of which will bound to happen. Look, of course I support having people choosing their own moderation policy, but that only works in theory: in practice greater power does not lead to greater responsibility.

Comment by Austin Huang —
some responses

Allo!

So first off, I should note that I actually put some poor Matrix.org through the pain of reading a draft of this article before publishing it, so I consider it somewhat accurate, or at least as much as reasonably possible considering the sheer size of the thing.

Now, as to your specific comments...

Writing all the third parties together is quite misleading and you should definitely separate them.

My reviewer also had that objection, but I will point out that the distinction is not made in the privacy policy. So I think it would actually be unfair to split it out here: it's not like you can actually pick and choose which part of the privacy policy you accept when you start using Matrix services.

In fact I'd even say it's a problem that all of those are indiscrimenatly present in the policy.

As an aside, I also appreciate that Matrix.org has a fairly decent code of conduct

[response summary: they ban people too easily]

I err on the side of banning more people than less, to be honest, so I'm perfectly fine with this. Maybe it would be better to have two rooms, one for explicit code violations and one, more "liberal" room where anything goes?

You also made comments regarding Mastodon and moderation policies which I don't quite grasp the point of.

[...]

Thanks for the clarifications on Mjolnir, room blocking, and guest accounts, I updated the article accordingly.

tombstone event

It does have a GUI in Element:

  • /upgraderoom

That's literally not a GUI: it's a text command.

but Matrix.org people are trying to deprecate it in favor of "Spaces"

Citation required. Also Spaces are rooms and so they can also be included in room directories.

The reviewer didn't wish to be cited, so I can't actually provide a quote here. Happy to stand corrected, but it does feel like a large feature overlap, no?

New users can be added to a space or room automatically in Synapse

In public homeservers, this may leak account age.

Considering the entire history of everything is available forever on home servers, that seems like a minor compromise to get higher room availability in a distributed cluster (which is one of my use cases, that, granted, I did not make very clear).

Only public aliases (local aliases are unrestricted afaik). Also they're not required for listing on room directories.

Now I'm even more confused than I was before. Local addresses, aliases, public aliases, wtf?

Given that this is a thing, it is likely to be the goal.

added a link.

register (with a password! aargh!)

Don't you also have a password on Signal?

Not really. They really want you to set a PIN so that you can do account recovery when you lose your phone, but I am not sure it's mandatory. Even if it was, it's not something you get prompted for all the time, only when you lose a device. In Element, I frequently had to login again, including in the Android app. I never had to use my Signal PIN so far (although for a while they used to prompt for it so that you wouldn't forget it, but they thankfully stopped that annoying practice).

Thanks for the review!

Comment by anarcat —
Lot of correct insights but from a very specific point of view

Thank you for this detailed writeup. I believe the most important - and missing - statement is that it is az ircop point of view. Many of the menioned "problems" are related to the huge difference between irc and matrix, which may seem to be small from an irc-users' point of view. I try not to get into details, just some samples.

comments

2.1 data retention, 5.4 transport,

It is a fundamental function of Matrix to keep history, which is a fundamental shortcoming of IRC. There are bazillion separate solutions on irc to keep history, in various forms, places, and processability. Matrix histoy keeping results a few attributes, like scrolling back, synchronising, out of order display of ordered messages etc. It is just not proper to compare to a system with a fire-and-forget architecture. Also you mentioned that severs log even one-to-one rooms: that's right but it has no means to abuse: all the messages are end-to-end crypted by default.

2.3 matrix.org policies, 2.5 url previews,

Morg is just one server, and not a particularly loved one. There are 2000+ servers on the network and a very few shares settings and policies with them. Admins keep trying to create a system to help new users to find good servers. It's still in infancy.

3.1 mjolnir bot, 3.2 cmd tool,

These are just tools. It's like complaining about eggdrop in an irc server review. The potocol is pretty simple (REST), writing a bot is not really a hard task, only nobody have given one enough love yet.

3.4 Fundamental federation problems

I don't think you're right here. First, room moderation is not connected to homeserver policies at all (they are federated). If you mean room ops don't see IP addresses - indeed, similar to masked irc users. Second, my server have guest accounts and it works with more or less success, but indeed if it was abused I'd switch it off. However it is true that bridges are a hell to moderate but the reason is because they use different restriction solutions which aren't simple to convert.

3.5 hijacking room admins

It is more secure than IRC, where a server admin or a service admin can hijack anything. The signing may change, it's up to debate as of now, along with account decoupling from servers.

4. Availability

You are looking for a wrong solution. In Synapse (which is, remember, just one of the possible many servers) Redundancy works on call level (running parallel workers) and db and redis can be easily replicated (my test replication was up in 10 minutes, it is much simpler today than 5 years ago). But you're comparing with a database-less irc.

5. Performance

This is all about Synapse. Synapse is a not-so-good server, to say it politely. It can now scale horizontally just okay (I use that) but still a resource hog vertically. I've been used The Construct (an alternative server, not production ready) for many years and it was extremely unlike Synapse, both in performance and resource use, also it has been designed to scale in both ways. It's not ready for use (and maybe never will) but shows that your concerns are similar like the early irc servers were, many decades ago.

5.3 Latency,

My internal latency is about 500ms, the gross round-trip is usually about 1500ms. First sync/join is braindead, mostly due to bad protocol design and it is being fixed AFAIR. Normal sync is not that bad now, especially since lazy sync was implemented. (a few secs)

6.1 onboarding

You're unfair. My server's onboarding is: 1. get on the login webpage, enter your data 2. click on the link in your verifiaction email 3. you're in. Also since my server have guest allowed you can even just pop in a room and start talking. Zero onboarding (but pretty inconvenient for everyone).

6.2 clients, 6.3 bots

You can use a bash script with a few curl calls as a client/bot, you cannot get simpler than that. Still, it's a young system, needs better clients.

6.4 specifiaction

Exactly as you say, it's a mess. But you're not right in saying "matrix people", as MSCs are from everyone. There are useful ones and there are which is not.

7.1 History

["Finnish", not "finish" ;-)] But you see, 1988 was 34 years ago and matrix is a few years old. Also you see that irc has been split and broken 4-5 times in epic proportions and multiple times afterwards. Matrix can be split, and if it's run by Element, Inc. the way it is now it's bound to be split. There are a lot of people toying with the idea of redesigning the protocol, keeping the concept. We'll see.

7.2 Everyone's behind IRC

Nah, funny but.. not quite. IRC went the closed-circle professional run server way, Matrix choose the homeserver way. I do not believe that the cabal-operated networks are the best way, and you've seen what happened to freenode. Nobody can really takeover matrix, xmpp or activitypub, since they are distributed systems, and not a tightly controlled, closed circle. Their weakness (from your POV: openness) is also their strength (resilience).

Comment by grin —
response

Well, there's a lot to unpack here, but let's start with this:

I believe the most important - and missing - statement is that it is az ircop point of view.

I don't understand how you can simultaneously come to the conclusion that I am an IRCop and claim that I did not state this in the article. Maybe you'd like to reread the second paragraph, which says:

I am the operator of an IRC network

Now onto specifics.

It is just not proper to compare to a system with a fire-and-forget architecture.

I think it's perfectly fair to compare the two, since it's two technologies I am considering. I will also note that IRCv3 has provisions for storing messages server-side as well, just not the way Matrix does it.

Also you mentioned that severs log even one-to-one rooms: that's right but it has no means to abuse: all the messages are end-to-end crypted by default

Sure, but E2EE is not all: it encrypts only the content, not the metadata, and there's lots of metadata that flies around not encrypted. That is the whole point here.

2.3 matrix.org policies, 2.5 url previews, 3.1 mjolnir bot, 3.2 cmd tool,

... which are basically:

Morg is just one server. [...] These are just tools.

Yes, but those tools matter. Matrix.org is where people will first go and will greatly influence policy. It should set the right example. mjolnir is something that's actively proposed by Matrix people as a solution to moderation. Yes, it's just one tool, but there is no other. I think such tools should be built-in to clients more tightly, without people having to run their own moderation bot (and therefore home server). Anyone considering moderation problems in Matrix should first read this admin's post and see how better to solve this problem.

Just waving your hand saying "there's an API" does not help users in any useful way.

3.4 Fundamental federation problems

I don't think you're right here. [...]

I don't understand the point you're trying to make here, at all. I'm trying to argue that moderating a federation is hard, including on IRC. You're responding that ... you have a guest mode? and you agree bridges are hell? Which part am I wrong about here specifically?

3.5 hijacking room admins

It is more secure than IRC, where a server admin or a service admin can hijack anything.

That section of mine claims that Matrix admins can hijack users (and therefore rooms as well), how is that more secure than IRC? Maybe you can't hijack rooms directly, but that doesn't seem like a large enough improvement to me.

  1. Availability

You are looking for a wrong solution. In Synapse, [...]

  1. Performance

This is all about Synapse. Synapse is a not-so-good server, to say it politely.[...] I've been used The Construct (an alternative server, not production ready) for many years[...]

Okay let's pretend we're reviewing the web here. The main website is Gulag.com and it's the search engine. Its performance is mediocre and it disappears off the internet once in a while because it's not built for availability. Yet there's this other "alternative server" that is "not production ready" yet out there, somewhere. Will the web succeed?

Maybe. Maybe everyone will start using that mysterious search engine instead of the Gulag. But that's not what is happening with Matrix right now. Every time this discussion comes up, people mention Conduit, dendrite, and what not: none of those servers have feature parity with Matrix yet, and they do not, as far as I know, provide a highly available setup. So maybe they fix some performance problems, but you lose features.

Synapse is currently the flagship server: if it fails at something, Matrix fails. Claiming those problems are solved by some "not production ready" server does not help user or defeat my point about those limitations.

5.3 Latency

[500-1500ms latency]

That sounds horrible.

6.1 onboarding

You're unfair. My server's onboarding is: 1. get on the login webpage, enter your data 2. click on the link in your verifiaction email 3. you're in.

No I'm not: the above is just your server (and not Matrix.org). And it's more onboarding than Signal, which I'm comparing against here.

Also since my server have guest allowed you can even just pop in a room and start talking. Zero onboarding (but pretty inconvenient for everyone).

Let me know when the trolls find out abuot that guest system of yours and how you handle it, then we'll talk about onboarding again.

6.3 bots

Did you read that section? It's one of those where I actually have basically only positive comments about Matrix...

[...]

7.2 Everyone's behind IRC

You changed the title of this section, it is called "Where the federated services are in history".

IRC went the closed-circle professional run server way, Matrix choose the homeserver way.

It's really strange to make such a parallel because it seems to me that Matrix.org is way more engaged towards the business side of things (working with major european institutions for example) than IRC ever was. I think that IRC servers are way closer to "home servers" (ie. servers you run from your home) than Matrix is.

I do not believe that the cabal-operated networks are the best way, and you've seen what happened to freenode.

What happened to freenode is actually quite interesting, and I'm kind of sorry I didn't talk more about it here. But what happened is somewhat of a good thing, in the end: they got rid of a critical flaw in their administrative structure and survived. It's now called libera.chat, and freenode is dead, long live libera.

Nobody can really takeover matrix, xmpp or activitypub, since they are distributed systems, and not a tightly controlled, closed circle. Their weakness (from your POV: openness) is also their strength (resilience).

I don't think it's black and white. Openness, in networks, brings weakness. It's a fundamental property of distributed systems, and I don't see Matrix trying to address those problems directly enough to believe they will fix the issues that are plaguing other protocols including email, IRC, XMPP, and Signal.

Sure, no one can take over Matrix, XMPP, or ActivityPub. No one can take over IRC or email either, but someone is certainly trying to anyways, and it seems they (Google and Microsoft) are succeeding with email. One could argue that XMPP has been taken over and scuttled by Google and Facebook already.

Not taking those threats under consideration would be a serious mistake that Matrix should definitely avoid if it wants to survive in the long run.

Comment by anarcat —
further comments

Before anyone else piles on the comments here, I urge you to actually read the article before responding: don't just plunder the table of contents and assume what the content is. I approved the previous comment as an example, but I take a dim view of people that do not actually read the text they comment on, and will not approve further comments of that nature.

Thank you in advance.

Comment by anarcat —
I will be really short... still not enough

1) I have read the article, also read again the sections I was responding. What you see is that my phrasing was specific but was not detailed enough - apologies; this commenting isn't really good for discussions. (You can reach me on matrix as @grin:grin.hu, I guess that isn't a shocking surprise for you.) [Just responding to your first sentence: I have seen you're an ircop, I was missing that you expressively stated that what you write is not the general, neutral review but a very specific ircop-centered (and skewed) one; "why irc is the best, compared to matrix"]

2) You compare a 35 years old (and very mature) technology to a betatesting and fairly young system, and you do it by comparing features specifically require some maturity to be even comparable. You compare a reference non-performant betatest server to multiple production-ready irc servers. You think that's okay (so it is, really, since this is your article after all), and I think it's a mistake (and it is, probably, for a reader not realising that your view is strongly biased). I also have grown up on irc [mainly IrcNet], I am intimately familiar with it (or, rather, what it was 15+ years ago), and I see its pros and cons. I am also way too familiar with matrix internals, both technically and politically, and it really have effects on my views.

3) There are more than 200 open, stable matrix servers (and also some 2000+ used by fewer than a dozen users). Every time you say "the matrix network equals to matrix.org, its server or its organisation" you offend a lot of people (and not many of them have high opinions of the aforementioned bunch). You reject the view that one specific and pretty problematic server isn't The Network, and while this is your freedom (like picking an irc server with a mad admin and use it as an example for the whole network) it's probably very unfair to everyone involved, also makes any debate pointless. We cannot help whatever Element, Inc. does, and we do it differently. Whatever problem you prove by pointing at morg isn't necessarily valid elsewhere on the network.

4) re 500ms is horrible: I laughed out loud. I still remember 42 SECONDS lag, that's, uh, 42000 ms. And continuous connection losses. Reconnects. Actually, plenty of disconnect/reconnect messages seeing that the network is unstable. And of course the netsplits. I am not sure you said that in all seriousness. (For the record: the minimal full [user-server1-server2-bot-server2-server1-user] round-trip is about 70ms [measured on Construct servers last year], anything longer is caused by dozens of factors I will not detail here, apart from Synapse.)

5) re trolls: I am using both python antispam modules and database backed scripts, and I have my fair share of trolls. But you seem not to be pretty familiar how it works, and I am not sure you really want to know, so let's just say guests cannot join random rooms.

6) You do not seem to understand what "homeserver" means. It means that the server has one, or a few users, run by individuals at home. Show me a [large] irc network where the users run their own servers. (In fact, show me some irc networks with 2+ million users and full message history, and let us compare them.)

7) I agree that "don't see Matrix trying to address those problems" of openness vs. idiots. I (and many others) have been told that from the beginning, and they ignored it. I am pretty sure it's not worse off than irc, and I can't decide whether it's better in this field.

8) "No one can take over IRC either" - Freenode, friend. It has been taken over. If an organisation owns or controls all the servers, or controls the services they can do whatever they please to that network, and all the ircops can do is to create a new network, like Libera did. It is not Freenode - freenode was taken from them. They created a new network. In contrast, morg/element can't do anything to even bring down the matrix network: they can switch off the morg server, or blank the git repo, but they cannot even have effect the communication between servers and can't do squat about forks of the spec.

I could debate your "already taken over" statement about email or XMPP but I see no point, apart from saying you're wrong (in my opinion, which is, you know, a thing everyone have). There are threats, as there've been for the whole internet, but not less or more than other continuous factors like dictatoric states with powers, local governments or large companies in general.

Maybe the matrix network needs a fork, like irc had many times, to streamline protocol development, and honestly throw out a lot of unnecessary crap and bad hacks. Time will tell. Until then, it is not irc, and I don't think they can simply compared. You think "message history" is a feature and distributed servers are just an interesting addon but these actually define matrix, which is an "eventually consistent, distributed, direct acyclic graph of tagged and organised json messages". Very different from irc, which is a store-and-forward text service with multiple interconnected servers.

All that said - your review had many important and good insights and listed very real and important problems; my triggers were the unjust weights and false assumptions about the real reasons for those problems, and whether these are unsolvable (by being inherent to the system) or not.

Comment by grin —
talking in "you"

Wow, there's a lot in there for a "really short" comment. I'll just address this, which seems to be the gist of that last comment:

I could debate your "already taken over" statement about [...] but I see no point, apart from saying you're wrong (in my opinion, which is, you know, a thing everyone have)

See, that's the thing with opinions. Everyone has them and can throw them around like a kid throws rocks in a park. But eventually, their parents show them that throwing rocks isn't great for the other kids and they stop.

In this case, just telling me "I'm wrong" is not a productive opinion to express on this blog. I really hesitated in publishing your comment at all, because it really doesn't seem to bring anything new to the conversation (other than, you know, "you're wrong").

I would have preferred recommendations on how to correct or rephrase what you feel is so unfair about my article. You can disagree with it all you like, but I'm looking at better ways to improve the article, not an endless debates about peccadilles.

In the meantime, I won't be approving further "you're wrong" comments unless they bring constructive corrections to the article.

I don't believe I have been "unjust" in my analysis or have made "false assumptions". If I did, I am happy to correct those, but I will point out that a 8000 words text is bound to be interpreted in many different ways, so assuming intent of the writer is a mistake you should probably avoid.

Comment by anarcat —
Agree With the Issues Concerning Reliance on Matrix.org's Code Of Conduct Violations

Earlier somebody mentioned how it's impossible to appeal a ban? I just got that issue last night; I made the mistake of posting an article which criticized the company that funded Matrix for the first three years of it's life, and then furthered the opinion that the author might have had a problem with the fact that the company is from Israel (my comment was about the author not myself), but Mjolnir still thought that I deserved to be banned for "trolling, false information, spreading fud". Either way, I'm going to have to rejoin via my alternate account on the Opensuse.org homeserver, and hope to God that it doesn't use the ban list held by Matrix.org, because chat.schncs.de does, causing me to lose nearly all of my presences in every room that agregates bans from those lists.

The other flaw with this is how smaller admins of smaller services use the ban list from the larger ones; was told by an admin that Mjolnir is not smart enough to be able to lift bans on different servers if the ban is still in affect for the server holding the ban list. I'm not whining or complaining here, for I'll obviously take my punishment like a woman, but having a nearly universal ban on nearly every server (I was just banned from even some bridges, for goodness sakes), does make things difficult without starting again. And if I start with another account, how then do I know that I won't get caught out and banned again arbitrarily by my alias alone? Not to mention,the fact that my alias on Matrix has the same prefix as nearly everything else involving me on the internet? What could these sweeping ban hammers do for reputations in the future?

Comment by Kat —
mjolnir bans

It's true that mjolnir is a heck of a hammer: because Matrix's moderation systems are quite limited, they built that bot which does everything, everywhere. And because not everyone can run their own instance (or, even harder, maintain their own block list), everyone essentially runs the same block list and it makes it so situations like yours are really, really hard to get out of.

Now as to the actual topic you are refering to, I am aware of the conspiracy theories surrounding the funding of the Matrix project. Currently working at Tor, I must say those kind of discussions are rather pointless and fruitless. Yes, it does look like Matrix was originally funded by some Israel corporation that might have to do with the Israel secret services and / or military. But anyone using the Internet should probably be aware of its roots in ARPANET and the military. The concept of onion routing used in Tor was invented at the US Navy (basically), and Tor does get a significant amount of funding from the US governemnt (still).

That doesn't mean that Tor is backdoored. I think people over estimate the impact of funding in technical decisions. Or, more accurately, they misplace their concerns: Tor won't write a backdoor for the NSA because it's funded by the US state department. That doesn't even make sense. What does happen though is that funding directs Tor's work more towards anti-censorship (which is not really a problem in the US) instead of (say) resistance to surveillance from a global adversary (like the NSA, which is definitely a problem in the US).

(It should also be noted that such a threat model has always been out of scope for Tor, and there is probably no anonymity system out there that effectively solves that problem anyways, but you know, facts, who cares about those...)

So yeah, I haven't mentioned the Amdocs/Israel connexion in this review because, quite frankly, I am really tired of reading about it. Every time it comes up, it's the same thing: in the hackea review, they litterally say "Disturbing history (or maybe just FUD)... I mean why even mention it at this point?

I am much more concerned about the privacy violations that are fundamental to the current design. Those should be our primary critique of Matrix, not some weird references to some past funder...

And yes, Matrix moderation kind of sucks. I bet you tripped over some URL auto-ban that triggered when you posted a link about that annoying topic. I bet that the Matrix people have had this hundreds of times, and debated it dozens of times, without getting anywhere. I would totally understand them just getting tired of this and automatically banning people based on that URL. It's just too bad it's that global... They certainly would need a more fine-grained moderation system.

Comment by anarcat —
Re: Mjolnir

I did touch upon this in my previous comment.

Slight correction: turns out it's possible to run Mjolnir without running a homeserver, and most features would still work.

And because not everyone can run their own instance

People almost never use someone else's Mjolnir since they know that they'd have no control on how it behaves. That is, assuming that they know Mjolnir at the first place...

(or, even harder, maintain their own block list)

AFAIK Mjolnir requires you to write all manual (not automated) bans into at least one ban list, so in theory all Mjolnir users would be maintaining at least one ban list (regardless of its visibility).

everyone essentially runs the same block list

Rooms that are associated with a public organization (eg. Arch, Mozilla, etc.) that subscribe to Element Matrix Services generally use the matrix-org-coc-bl list (It is unclear whether they're obliged to do so, but in one case they do seem to agree with it. They also have their Mjolnir accounts on their domains), but otherwise its application is minimal (and voluntary). There also exists alternative ban lists as intended, such as Community Moderation Effort which is used for a few large privacy-related rooms.

I bet you tripped over some URL auto-ban

I don't think @abuse:matrix.org use the WordList protection (Even so, it would not publish the ban to a ban list). IIUC there are certain keywords that ping online admins and bans are applied manually.

Comment by austinhuang.me —
Slight correction

Bans are not related to servers: they are on rooms, and it does not matter which server you're on. Mjolnir puts bans in rooms, so if you're banned by the Matrix.Org people you don't go into their rooms. You can probably join most of the rooms (apart from element, inc. ones) even on matrix.org server, if you chose it for whatever unexplainable reason.

Servers can only disable your account of firewall you, everything else is up to the federation and the server(s) can do little about it.

(If you're banned on a non-element inc. room you may try to contact the room admins not to use over-restrictive morg bans. Chances of success are low, though.)

Comment by grin —
another perspective on moderation

I truly enjoyed reading this post from Mike Hoye, who's responsible for migrating Mozilla to IRC. At first they made it "non-federated" because he was worried about moderation, but he found Matrix's moderation systems to be better than other federated systems. In his own words:

Another way to say that is: among people or communities who trust each other in these decisions, an act of self-defense becomes, seamlessly and invisibly, an act of collective defense. No more everyone needing to fight their own fights alone forever, no more getting isolated and picked off one at a time, weakest first; shields-up means shields-up for everyone. Effective, practical defensive solidarity; it’s the most important new idea I’ve seen in social software in years. Every federated system out should build out their own version, and it’s very clear to me, at least, that is going to be the table stakes of a federated future very soon.

[...]

Maybe, just maybe, we can salvage this whole internet thing. Maybe all is not yet lost, and the future is not yet written.

Really interesting.

Comment by anarcat —
Comments on this page are closed.
Created . Edited .