Testing new hardware with Stressant
I got a new computer and wondered... How can I test it? One of those innocent questions that brings hours and hours of work and questionning...
A new desktop: Intel NUC devices
After reading up on Jeff Atwood's blog and especially his article on the scooter computer, I have discovered a whole range of small computers that could answer my need for a faster machine in my office at a low price tag and without taking up too much of my precious desk space. After what now seems like a too short review I ended up buying a new Intel NUC device from NCIX.com, along with 16GB of RAM and an amazing 500GB M.2 hard drive for around 750$. I am very happy with the machine. It's very quiet and takes up zero space on my desk as I was able to screw it to the back of my screen. You can see my review of the hardware compatibility and installation report in the Debian wiki.
I wish I had taken more time to review the possible alternatives - for example I found out about the amazing Airtop PC recently and, although that specific brand is a bit too expensive, the space of small computers is far and wide and deserves a more thorough review than just finding the NUC by accident while shopping for laptops on System76.com...
Reviving the Stressant project
But this, and Atwood's Is Your Computer Stable? article, got me thinking about how to test new computers. It's one thing to build a machine and fire it up, but how do you know everything is actually really working? It is common practice to do a basic stress test or burn-in when you get a new machine in the industry - how do you proceed with such tests?
Back in the days when I was working at Koumbit, I wrote a tool exactly for that purpose called Stressant. Since I am the main author of the project and I didn't see much activity on it since I left, I felt it would be a good idea to bring it under my personal wing again, and I have therefore moved it to my Gitlab where I hope to bring it back to life. Parts of the project's rationale are explained in an "Intent To Package" the "breakin" tool (Debian bug #707178), which, after closer examination, ended up turning into a complete rewrite.
The homepage has a bit more information about how the tool works and its objectives, but generally, the idea is to have a live CD or USB stick that you can just plugin into a machine to run a battery of automated tests (memtest86, bonnie++, stress-ng and disk wiping, for example) or allow for interactive rescue missions on broken machines. At Koumbit, we had Debirf-based live images that we could boot off the network fairly easily that we would use for various purposes, although nothing was automated yet. The tool is based on Debian, but since it starts from boot, it should be runnable on any computer.
I was able to bring the project back to life, to a certain extent, by switching to vmdebootstrap instead of debirf for builds, but that removed netboot support. Also, I hope that Gitlab could provide with an autobuilder for the images, but unfortunately there's a bug in Docker that makes it impossible to mount loop images in Docker images (which makes it impossible to build Docker in Docker, apparently).
Should I start yet another project?
So there's still a lot of work to do in this project to get it off the ground. I am still a bit hesitant in getting into this, however, for a few reasons:
It's yet another volunteer job - which I am trying to reduce for health and obvious economic reasons. That's a purely personal reason and there isn't much you can do about it.
I am not sure the project is useful. It's one thing to build a tool that can do basic tests on a machine - I can probably just build an live image for myself that will do everything I need - it's another completely different thing to build something that will scale to multiple machines and be useful for more various use cases and users.
(A variation of #1 is how everything and everyone is moving to the cloud. It's become a common argument that you shouldn't run your own metal these days, and we seem to be fighting an uphill economic battle when we run our own datacenters, rack or even physical servers these days. I still think it's essential to have some connexion to metal to be autonomous in our communications, but I'm worried that focusing on such a project is another of my precious dead entreprises... )
Part #2 is obviously where you people come in. Here's a few questions I'd like to have feedback on:
(How) do you perform stress-testing of your machines before putting them in production (or when you find issues you suspect to be hardware-related)?
Would a tool like breakin or stressant be useful in your environment?
Which tools do you use now for such purposes?
Would you contribute to such a project? How?
Do you think there is room for such a project in the existing ecology of projects) or should I contribute to an existing project?
Any feedback here would be, of course, greatly appreciated.