marcos

  1. hardware
  2. server

Marcos est mon serveur principal, ainsi nommée en l'honneur du sous-commandant Marcos.

Marcos, the quintessential anti-leader, insists that his black mask is a mirror, so that "Marcos is gay in San Francisco, black in South Africa, an Asian in Europe, a Chicano in San Ysidro, an anarchist in Spain, a Palestinian in Israel, a Mayan Indian in the streets of San Cristobal, a Jew in Germany, a Gypsy in Poland, a Mohawk in Quebec, a pacifist in Bosnia, a single woman on the Metro at 10 p.m., a peasant without land, a gang member in the slums, an unemployed worker, an unhappy student and, of course, a Zapatista in the mountains". In other words, he is simply us: we are the leader we've been looking for.

-- Naomi Klein, Socialist Register

Voir aussi la documentation de maintenance, en particulier mail et backup.

  1. Specification
  2. Hardware maintenance
    1. 2024 Fan replacement
    2. 2020 Replacement
    3. BIOS config
    4. Remaining transplant TODO
    5. DONE
  3. Possible phase out
    1. NUC
    2. Vero
    3. GnuBee
    4. Helios (dead)
    5. Supermicro
      1. Micro-ITX build
      2. ATX build
    6. TrueNAS mini
    7. Pine 64
    8. Ten64
    9. Brix
    10. Qotom
    11. Protectli
    12. Other SoC boards
    13. Other SAN options
    14. Other builds

Specification

The server is also backed by a UPS, a APC 1500VA BX1500m.

Hardware maintenance

See v1 for the initial setup notes. Those are kept only for historical reference, as the machine was rebuilt with new hardware in 2020.

See backup for backup and drive replacement procedures.

2024 Fan replacement

From the 2020 replacement, the server has always been away, either in a basement or a closet, and its noisy stock fan didn't really matter. But now it's moved inside the office and damn that thing is noisy. It's noisy when idle and sounds like a hair drier when it spins up. So, it's got to go.

I've heard good things from friends about Noctua, so let's go with that. My previous "gold standard" was the stock Intel fan that came with the v1, which, according to this old review ranges from 14 to 26dBa in noise.

Noctua seems to be Amazon sellouts, but they have products at memoryexpress.com and NewEgg, so I guess I'll try Memory Express now. Out of their Noctua selection, their cheapest fan compatible with my CPU is the NH-D9L if I read all of this right, at 80$ + tax + 13$ purolator shipping. The specification on that is:

There's a big asterisk with Noctua which is that they don't follow the TDP (Thermal Design Power) rating and instead have their own NSPR (Noctua Standard Performance Rating). So who knows what that "88" stands for. In their compatibility table, it's rated as "low turbo/overclocking headroom", which is fine by me: I haven't done "overclocking" in decades at this point.

The sink is also compatible with the board, apparently, so presumably I don't need to worry about pins, voltage, or actually knowing WTF I am doing, in general, which is reassuring but probably wrong.

The other concern is fitting the fan in the case. There's a case compatibility chart but my case isn't there. The closest is the Supermicro-SuperChassis-743T-500B-1842 but that one is 25.5" x 17.8" x 7" while mine is 7" (4U) x 20.9" x 16.8". But really, both are 7" where it matters, which is 168mm, which is plenty of room for a 110mm fan.

Interestingly though, it marks the 158mm U12S as "insufficient clearance" while the C14S, at 115-142mm supposedly fits. Its fan (NF-A14-PWM) is less interesting for me: it's much noisier at 24.6db(A).

I have also considered:

Update: I ordered the NH-D9L and installed it and immediately realized the problem was actually not the CPU fan, but the board fan, which is spec'd at 45dB. That fan is a weird construction that has a 92x25mm fan inside an enclosure that directs the airflow, see FAN-0076L4 for the detailed specs. I asked Supermicro if they had a better replacement fan, and so far they just keep pointing me at that fan. So I asked Noctua if their fans would fit.

2020 Replacement

In 2020, hardware for marcos was swapped out into a new box.

The machine has 4x3.5" hotswap drives so there's plenty of room for expansion and it should be easier to replace drives when they fail. Here's the inventory of drives.

Currently in marcos:

There should also be spare drives in the office which could be used to create RAID-1 arrays of those. I believe there should be:

Unfortunately it looks like my old HDD inventory is too old to provide a good replacement strategy. We could get:

  1. 2x1TB m.2 SSDs to free up slots in the disk slots
  2. 3x SATA cables
  3. 1x4TB drive to create a RAID-1 array with the spare 4TB drive and the existing 8TB? (or just another 8TB)
  4. 1x8TB drive to build a server in the office
  5. the existing SSD in marcos would be reused in the office
  6. a mini-1/8" audio jack, either 3' or 6'
  7. a webcam?
  8. a label writer?
  9. a spare wifi router?

Prices are rather high right now (march 2020) -- the Ironwolf is the same cost it was in January 2018 -- so maybe wait until civilization collapses.

Possible newegg order:

Update: seems like Intel is more popular there, maybe try one in the mix. We support NVMe anyways, so might as well get one of those:

The samsung is 70$+ more so maybe not worth it?

BIOS config

New machine BIOS configuration:

Remaining transplant TODO

  1. The external backup drive (sdc2) could be swapped into one of the hotswap bay.

  2. need to setup RAID-1 on the SSDs

  3. SSD drive floating in bay because of missing tray adapter. (to be replaced by 2xNVMe drives)

  4. Missig serial port, required to switch to headless server (and remove the nvidia video card), although...

  5. ... no BIOS displayed in with adhoc serial connector (adafruit 954) but normal getty doesn't display properly either (garbled output).

  6. HDD LEDs seem to light up (but not the SSD!) in the BIOS, but are not lit when Linux is booted. tested ledmon but it seems to have issues with AMD devices and, surprise-suprise, i have a AMD board! but i'm not sure that is related because the leds are controlled by the Supermicro enclosure...

DONE

  1. missing a SATA cable for the port #3, because provided cables have an "elbow" that prevents them to be connected (and the bord connectors are on the side instead of on top (!!).

  2. RAID-1 on the HDDs

Possible phase out

marcos came online in early 2011 so it's heading towards its 8 year of service at the time of writing, which is stretching the usual 5-year depreciation cycle we usually have for computer hardware. So I started looking for replacements.

NUC

See the curie page for ideas on that space.

Vero

Another target would a home-cinema adapter like the Vero which I have heard good things about:

Possible issues:

Update: I've ordered a Vero 4k, we'll see how it goes. It will at least allow me to move the server outside of the living room for now, which will simplify things. It will also postpone storage issues if I just buy an external hard drive... It seems I have eaten over 1TB of storage only in 2018, according to Grafana / Prometheus / Munin. Ironically, Prometheus is about 1% of that (12GB)...

GnuBee

A possible solution is to shift storage to a SAN like the GnuBee. This would relieve stress on each device to provide large amount of storage and considering gigabit is wired everywhere now, we can use (abuse?) the local network and store files on a SAN. Unfortunately, the protocols still suck: we might be stuck on SSH or some similarly nasty interface for security.

Specs:

One problem with the GnuBee is that the manufacturers are not maintaining the drivers and source code for the hardware, which ships only with an outdated version of libreCMC. This means that installing Debian (for example), is tricky. Brown is still working on mainline support and reported in December 2018 that his 4.20 tree was working as "near-mainline".

The Gnubee's hardware also seems to be lacking by modern standards now: the CPU is slow and might have trouble maxing out all drives in the cluster, as 6 drives going full spin will generate a lot of I/O.

Helios (dead)

https://kobol.io/helios4/

Interesting alternative to the gnubee (more powerful, among other things).

Reviews: hardware, software.

Update: they shut down the shop.

Supermicro

Supermicro sells cases as well as motherboards, and some of those might be interesting for a home server / NAS solution. For example, the 721Tq (ATIC 235$) has four hot-swappable 3.5" bays for SATA hard drives and two internal 2.5" bays with a Mini-ITX motherboard backend (235$CAD at ATIC).

It would require building a whole new server since marcos is microATX but the Mini-ITX form factor is good because it fits well with SoC boards (see below). A friend specifically looked at this atom server which has a nice price tag (618$CAD).

The downside of MiniITX is that, because it's smaller, it's (A) more expensive and (B) more limited. For example, it's sometimes difficult to find > 8GB boards in that size, which is the case of that Atom server as well. The Supermicro Atom servers, in general, are slow, and comparable to the current Marcos CPU. See also this comparison between the different board standards.

Supermicro also has what they call mobile racks that can fit multiple drives in a 5.25" bays and might be a better fit. They also have larger cases, the CSE-733T-500B (ATIC 280$) or CSE-733TQ-500B (ATIC 300$), which might do the trick. It seems the 20$ extra is because the latter supports SGPIO which might presumably have better support for HDD LEDs, a welcome improvement.

Micro-ITX build

BOM:

Power budget:

See also power.org for a power usage estimates for UPSes.

ATX build

The above setup only has 6xSATA ports however. Here's a build with the larger case:

Last question left: APU or not? A build with the AMD Ryzen 5 2400G "wastes" two cores for a builtin GPU, compared to the six-core equivalent which is also cheaper (183$) but consumes more power (95W). The same-power equivalent is the "x-less" one which is more expensive than the 2400G (287$). That is kind of an aberration though, that CPU can be found cheaper elsewhere (e.g. canadacomputers, 170$).

As a comparison, the Supermicro 5028A-TN4 build based on the Intel Atom C2758 ships for 790$ at ATIC, without memory. It will, however, have significantly lower power usage (40W less).

Update: I ordered basically the above, with a system build as well. No graphics card, we'll see how that flies. :) Hopefully I can get away with a junk video card I'll find lying around somewhere, and use the serial port otherwise. The server is already pretty much headless anyways...

Update 2: ATIC didn't have the 2600 CPU in stock and instead proposed to switch to the more expensive 2600x without extra cost. It has a high TDP power consumption (+30W) but we estimate the total power consumption is still well below the PSU's 500W capacity. They also did not have the 2400G in stock, for future reference.

TrueNAS mini

The folks behind FreeNAS are offering NAS hardware pre-installed with the (FreeBSD-based) distribution:

https://www.ixsystems.com/freenas-mini/

Those are Atom-based processors, similar in power to what marcos currently uses, at a much lower power usage and better memory support.

It's unclear if I could just migrate marcos to this platform as is, and the prices might be slightly higher than what I would get when building it from scratch...

Update: FreeNAS is now called TrueNAS and they have pretty sexy cases now, checkout their TrueNAS mini boxes. Their 4-bay hotswap thing starts at 1000$.

Pine 64

They have SBCs of course -- that's how they started -- but also a neat little NAS case. Make sure you also get all the parts needed.

Ten64

https://www.crowdsupply.com/traverse-technologies/ten64/updates/building-a-nas-with-ten64-and-rockstor-and-new-turnkey-nas-bundle

Brix

Gigabyte (motherboard makers) have the Brix platform that's similar to the Intel NUC.

Qotom

Qotom also makes mini-PCs.

Qotom is interesting: they make rugged devices that are passively cooled, in a wide (and I mean very wide) variety. So wide that it's actually hard to find anything on their 13-page long product section. It seems like they basically do "mini PC" style main boards that range anywhere from 2 to 8 gigabit port, with USB-2, USB-3, audio, DP, HDMI, VGA, serial ports (or not).

All the cases are kind of a distraction on top of that. Was used by at least one Debian Planet user for a project with Proxmox.

Protectli

getting out of scope, but:

https://protectli.com/

Other SoC boards

There are many SoC boards that could be used to create a device from scratch. Unfortunately, many of those don't meet my memory requirements (8+GB):

The Macchiatobin is interesting because it has a DDR4 socket so it supports up to 16GB of ram, but has features I don't need for a home server, like three SFP 10Gig-E ports... Could still be interesting for a SAN if I ever upgrade the network to 10G. 269$-369$. That said, someone on IRC had a very bad experience with it: the capacitors were broken and they refused to take it back.

The Beelink is also interesting: Intel N3450 4GB DDR, 64GB SSD.

The Fitlet 2 runs Debian by default and looks like a nice small machine.

See also the board-db for a full list.

Other SAN options

On top of the above SoC systems, a proper case or some system will be needed to handle more drives. Buying gigantic drives is nice, but it's really a single point of failure and I should probably do some RAID or ZFS at some point.

One option is to delegate this to an external enclosure, like those Orico multi-bay enclosures. They can connect over USB 3 and/or eSATA which should give good performance. It's hard to see how they work or if they work in Linux at all, so compatibility and reliability could be a problem.

SATA and USB throughput could also be a bottleneck. For example, the Seagate Ironwolf 10TB can do up to 210MB/s, which means ~1600Mbit/s, totally overloading USB 2 (480Mbps, so by a factor of 4). USB 3.0 should be able to handle one drive, maybe two or even three (5Gbit/s) but at that point, CPU might become a problem as well. There are also compatibility problems between newer drives and older SATA controllers: marcos' old 3Gbps SATA 2 controller only recognized 4TB out of a Seagate IronWolf 8TB NAS drive.

Of course, computer cases should be able to offer hot-swappable disk slots but strangely it's somewhat rare in user-level hardware. Most SATA controllers and disks support hot-swapping, but it needs to be double-checked.

Other builds

See also https://mtlynch.io/budget-nas/.

At work, we built two rescue hosts with the PC engines boards, right before they went EOL. I documented a bunch of alternatives, see this list.

Created . Edited .