One of our servers is a rotting carcass being kept alive by our collective prayers. It runs Windows 7 and custom software whose developer is dead and the source is missing, nothing has been updated for over a decade, and it has its own independent UPS because once it goes down, it has an extremely slim chance of recovering, and we’re afraid to test it. It controls the card entry system into the building, including the server room. Boss doesn’t want to replace it because we’d have to replace all of the terminals and controllers too, and it hasn’t catastrophically failed yet.
You’re right. It’s not a pet. It’s like one of the Saw movies: if it dies, we’re all fucked.
It’s surprisingly easy to get from the main hall to the server room. There are two doors between the entry hall and the server room, one can be bypassed by yanking it real hard, and that gives access to the breaker box for the electromagnets among others. The building is not particularly well-designed.
I’m sure there’s a good reason (or at least a believable reason) but I’m curious now, why can’t copies be made of the binary/data and start trying to get it running on a VM or another box?
The reason is that we want it to fail. My original comment was more emotive than descriptive. The system is horribly designed and a fucking menace on the best day, so short of direct sabotage, we’re doing what we can to force the bossmang to replace it.
The question I often ask clients who think this way is "How much would it cost if it did fail? Let’s say this happened today. What would be the cost to replace it NOW and not only that but make sure people who are working can still do so with the interruption?
Now how much would it cost to schedule the interruption and manage the fall out in a way that is controllable?
For some, the catastrophic failure points to “hey I fixed the thing!” And the incentives for that kind of person are different from the person whose job is to mitigate risk.
It sounds like your boss is the former. In which case it’s going to be fun when it fails.
I learned this relatively quickly running my own server with the intention of my family also using it. Data on a separate drive, backed up regularly and automatically. System on it’s own drive, dd’d when it’s in it’s final state and backed up before I screw around any deeper than trying out a new container. I can bring my server back up in however long it takes to transfer data.
New guy at work ran this to try to fix permissions on his home folder, accidentally ran it on root (both would have been bad)
Several highly paid and experienced Linux admins finally just gave up and deleted the server and built a new one from the backups.
Which, honestly, is the better way to go. Treat your compute resources like cattle, not pets.
One of our servers is a rotting carcass being kept alive by our collective prayers. It runs Windows 7 and custom software whose developer is dead and the source is missing, nothing has been updated for over a decade, and it has its own independent UPS because once it goes down, it has an extremely slim chance of recovering, and we’re afraid to test it. It controls the card entry system into the building, including the server room. Boss doesn’t want to replace it because we’d have to replace all of the terminals and controllers too, and it hasn’t catastrophically failed yet.
You’re right. It’s not a pet. It’s like one of the Saw movies: if it dies, we’re all fucked.
I gönne be honest, it sounds kinda stupid to be reliant on a server to open the door to the same server.
It’s surprisingly easy to get from the main hall to the server room. There are two doors between the entry hall and the server room, one can be bypassed by yanking it real hard, and that gives access to the breaker box for the electromagnets among others. The building is not particularly well-designed.
I’m sure there’s a good reason (or at least a believable reason) but I’m curious now, why can’t copies be made of the binary/data and start trying to get it running on a VM or another box?
The reason is that we want it to fail. My original comment was more emotive than descriptive. The system is horribly designed and a fucking menace on the best day, so short of direct sabotage, we’re doing what we can to force the bossmang to replace it.
The question I often ask clients who think this way is "How much would it cost if it did fail? Let’s say this happened today. What would be the cost to replace it NOW and not only that but make sure people who are working can still do so with the interruption?
Now how much would it cost to schedule the interruption and manage the fall out in a way that is controllable?
For some, the catastrophic failure points to “hey I fixed the thing!” And the incentives for that kind of person are different from the person whose job is to mitigate risk.
It sounds like your boss is the former. In which case it’s going to be fun when it fails.
So… the dead server controls who is even able to enter the building? Wow. That is one big juggernaut of a problem heading for you.
Typically a brick can control who can enter the building. Security man the doors for a few days until the new system is in.
I learned this relatively quickly running my own server with the intention of my family also using it. Data on a separate drive, backed up regularly and automatically. System on it’s own drive, dd’d when it’s in it’s final state and backed up before I screw around any deeper than trying out a new container. I can bring my server back up in however long it takes to transfer data.
Why does he have rights to use sudo in the first place?