Marcos est mon serveur principal, ainsi nommée en l'honneur du sous-commandant Marcos.

Marcos, the quintessential anti-leader, insists that his black mask is a mirror, so that "Marcos is gay in San Francisco, black in South Africa, an Asian in Europe, a Chicano in San Ysidro, an anarchist in Spain, a Palestinian in Israel, a Mayan Indian in the streets of San Cristobal, a Jew in Germany, a Gypsy in Poland, a Mohawk in Quebec, a pacifist in Bosnia, a single woman on the Metro at 10 p.m., a peasant without land, a gang member in the slums, an unemployed worker, an unhappy student and, of course, a Zapatista in the mountains". In other words, he is simply us: we are the leader we've been looking for.

-- Naomi Klein, Socialist Register

Voir aussi la documentation de maintenance, en particulier mail et backup.

  1. Specification
  2. Hardware maintenance
    1. 2020 Replacement
    2. BIOS config
    3. Remaining transplant TODO
    4. DONE
  3. Possible phase out
    1. NUC
    2. Vero
    3. GnuBee
    4. Helios
    5. Supermicro
      1. Micro-ITX build
      2. ATX build
    6. FreeNAS mini
    7. Other SoC boards
    8. Other SAN options

Specification

Hardware maintenance

See v1 for the initial setup notes. Those are kept only for historical reference, as the machine was rebuilt with new hardware in 2020.

See backup for backup and drive replacement procedures.

2020 Replacement

In 2020, hardware for marcos was swapped out into a new box.

The machine has 4x3.5" hotswap drives so there's plenty of room for expansion and it should be easier to replace drives when they fail. Here's the inventory of drives.

Currently in marcos:

There should also be spare drives in the office which could be used to create RAID-1 arrays of those. I believe there should be:

Unfortunately it looks like my old HDD inventory is too old to provide a good replacement strategy. We could get:

  1. 2x1TB m.2 SSDs to free up slots in the disk slots
  2. 3x SATA cables
  3. 1x4TB drive to create a RAID-1 array with the spare 4TB drive and the existing 8TB? (or just another 8TB)
  4. 1x8TB drive to build a server in the office
  5. the existing SSD in marcos would be reused in the office
  6. a mini-1/8" audio jack, either 3' or 6'
  7. a webcam?
  8. a label writer?
  9. a spare wifi router?

Prices are rather high right now (march 2020) -- the Ironwolf is the same cost it was in January 2018 -- so maybe wait until civilization collapses.

Possible newegg order:

Update: seems like Intel is more popular there, maybe try one in the mix. We support NVMe anyways, so might as well get one of those:

The samsung is 70$+ more so maybe not worth it?

BIOS config

New machine BIOS configuration:

Remaining transplant TODO

  1. The external backup drive (sdc2) could be swapped into one of the hotswap bay.

  2. need to setup RAID-1 on the SSDs

  3. SSD drive floating in bay because of missing tray adapter. (to be replaced by 2xNVMe drives)

  4. Missig serial port, required to switch to headless server (and remove the nvidia video card), although...

  5. ... no BIOS displayed in with adhoc serial connector (adafruit 954) but normal getty doesn't display properly either (garbled output).

  6. HDD LEDs seem to light up (but not the SSD!) in the BIOS, but are not lit when Linux is booted. tested ledmon but it seems to have issues with AMD devices and, surprise-suprise, i have a AMD board! but i'm not sure that is related because the leds are controlled by the Supermicro enclosure...

DONE

  1. missing a SATA cable for the port #3, because provided cables have an "elbow" that prevents them to be connected (and the bord connectors are on the side instead of on top (!!).

  2. RAID-1 on the HDDs

Possible phase out

marcos came online in early 2011 so it's heading towards its 8 year of service at the time of writing, which is stretching the usual 5-year depreciation cycle we usually have for computer hardware. So I started looking for replacements.

NUC

An easy first target is to buy another NUC like the curie workstation since it's cheap and works well. The downside is that it requires laptop (2.5") hard disks which are more expensive. A replacement drive for Marcos could cost around 200$CAD (2.5" 4TB Seagate at newegg.ca). The 500GB M.2 drives are still around 190$CAD. So a replacement would be at least 800$CAD, probably around 1000$ with memory.

Possible issues:

See also the Brix and Qotom mini-PCs.

Update: Purism announced their own mini-PC as well.

Vero

Another target would a home-cinema adapter like the Vero which I have heard good things about:

Possible issues:

Update: I've ordered a Vero 4k, we'll see how it goes. It will at least allow me to move the server outside of the living room for now, which will simplify things. It will also postpone storage issues if I just buy an external hard drive... It seems I have eaten over 1TB of storage only in 2018, according to Grafana / Prometheus / Munin. Ironically, Prometheus is about 1% of that (12GB)...

GnuBee

A possible solution is to shift storage to a SAN like the GnuBee. This would relieve stress on each device to provide large amount of storage and considering gigabit is wired everywhere now, we can use (abuse?) the local network and store files on a SAN. Unfortunately, the protocols still suck: we might be stuck on SSH or some similarly nasty interface for security.

Specs:

One problem with the GnuBee is that the manufacturers are not maintaining the drivers and source code for the hardware, which ships only with an outdated version of libreCMC. This means that installing Debian (for example), is tricky. Brown is still working on mainline support and reported in December 2018 that his 4.20 tree was working as "near-mainline".

The Gnubee's hardware also seems to be lacking by modern standards now: the CPU is slow and might have trouble maxing out all drives in the cluster, as 6 drives going full spin will generate a lot of I/O.

Helios

In pre-order: https://kobol.io/helios4/ Interesting alternative to the gnubee (more powerful, among other things).

Supermicro

Supermicro sells cases as well as motherboards, and some of those might be interesting for a home server / NAS solution. For example, the 721Tq (ATIC 235$) has four hot-swappable 3.5" bays for SATA hard drives and two internal 2.5" bays with a Mini-ITX motherboard backend (235$CAD at ATIC).

It would require building a whole new server since marcos is microATX but the Mini-ITX form factor is good because it fits well with SoC boards (see below). A friend specifically looked at this atom server which has a nice price tag (618$CAD).

The downside of MiniITX is that, because it's smaller, it's (A) more expensive and (B) more limited. For example, it's sometimes difficult to find > 8GB boards in that size, which is the case of that Atom server as well. The Supermicro Atom servers, in general, are slow, and comparable to the current Marcos CPU. See also this comparison between the different board standards.

Supermicro also has what they call mobile racks that can fit multiple drives in a 5.25" bays and might be a better fit. They also have larger cases, the CSE-733T-500B (ATIC 280$) or CSE-733TQ-500B (ATIC 300$), which might do the trick. It seems the 20$ extra is because the latter supports SGPIO which might presumably have better support for HDD LEDs, a welcome improvement.

Micro-ITX build

BOM:

Power budget:

See also ?power.org for a power usage estimates for UPSes.

ATX build

The above setup only has 6xSATA ports however. Here's a build with the larger case:

Last question left: APU or not? A build with the AMD Ryzen 5 2400G "wastes" two cores for a builtin GPU, compared to the six-core equivalent which is also cheaper (183$) but consumes more power (95W). The same-power equivalent is the "x-less" one which is more expensive than the 2400G (287$). That is kind of an aberration though, that CPU can be found cheaper elsewhere (e.g. canadacomputers, 170$).

As a comparison, the Supermicro 5028A-TN4 build based on the Intel Atom C2758 ships for 790$ at ATIC, without memory. It will, however, have significantly lower power usage (40W less).

Update: I ordered basically the above, with a system build as well. No graphics card, we'll see how that flies. :) Hopefully I can get away with a junk video card I'll find lying around somewhere, and use the serial port otherwise. The server is already pretty much headless anyways...

Update 2: ATIC didn't have the 2600 CPU in stock and instead proposed to switch to the more expensive 2600x without extra cost. It has a high TDP power consumption (+30W) but we estimate the total power consumption is still well below the PSU's 500W capacity. They also did not have the 2400G in stock, for future reference.

FreeNAS mini

The folks behind FreeNAS are offering NAS hardware pre-installed with the (FreeBSD-based) distribution:

https://www.ixsystems.com/freenas-mini/

Those are Atom-based processors, similar in power to what marcos currently uses, at a much lower power usage and better memory support.

It's unclear if I could just migrate marcos to this platform as is, and the prices might be slightly higher than what I would get when building it from scratch...

Other SoC boards

There are many SoC boards that could be used to create a device from scratch. Unfortunately, many of those don't meet my memory requirements (8+GB):

The Macchiatobin is interesting because it has a DDR4 socket so it supports up to 16GB of ram, but has features I don't need for a home server, like three SFP 10Gig-E ports... Could still be interesting for a SAN if I ever upgrade the network to 10G. 269$-369$. That said, someone on IRC had a very bad experience with it: the capacitors were broken and they refused to take it back.

The Beelink is also interesting: Intel N3450 4GB DDR, 64GB SSD.

The Fitlet 2 runs Debian by default and looks like a nice small machine.

See also the board-db for a full list.

Other SAN options

On top of the above SoC systems, a proper case or some system will be needed to handle more drives. Buying gigantic drives is nice, but it's really a single point of failure and I should probably do some RAID or ZFS at some point.

One option is to delegate this to an external enclosure, like those Orico multi-bay enclosures. They can connect over USB 3 and/or eSATA which should give good performance. It's hard to see how they work or if they work in Linux at all, so compatibility and reliability could be a problem.

SATA and USB throughput could also be a bottleneck. For example, the Seagate Ironwolf 10TB can do up to 210MB/s, which means ~1600Mbit/s, totally overloading USB 2 (480Mbps, so by a factor of 4). USB 3.0 should be able to handle one drive, maybe two or even three (5Gbit/s) but at that point, CPU might become a problem as well. There are also compatibility problems between newer drives and older SATA controllers: marcos' old 3Gbps SATA 2 controller only recognized 4TB out of a Seagate IronWolf 8TB NAS drive.

Of course, computer cases should be able to offer hot-swappable disk slots but strangely it's somewhat rare in user-level hardware. Most SATA controllers and disks support hot-swapping, but it needs to be double-checked.

Created . Edited .