A Painful Day of Technology
I just did my 100 situps. I wanted to do them at 5:30a when I woke up. But a technology emergency immediately grabbed me by the shorties and didn't let go until moments ago, 11:20p.
Today was an absolutely miserable day. Whenever a website I run is down I internalize all the people around the world who are without its service.
And that somehow brings into focus what's important in my own life. It's not computers.
First and foremost, I let Kelly and Matt down today. They got married and were kind enough to invite me. I hope you guys had a great wedding and I look forward to all the pics! You two make a great couple!
Second, I let my family down today. Lots of yelling behind closed doors. No family time. Brisk trips downstairs for whey protein.
Third, I didn't recover well. Today was a recovery day. I was supposed to eat well and relax. I can honestly say that I'm more physically exhausted than after long rides with Kindzia. My back is tight. My abs are tired. That isn't good recovery.
I was on all day. Over fifteen hours of solid debugging, fixing and cursing.
I won't bore you with the technical details. I'm too tired to type them up anyway. Something got broken. I spent the day fixing it.
It was very intense. When these things go down I devote every fiber of my energy (yes, my energy has fibers... I'm too tired to go back and type fiber of my being) to them. The first eight or nine hours evaporated quickly. Then the fatigue set in.
But with no fix in sight I had to push through. When I have these days I just won't accept anything but a solution. I want to give in. I say I'm giving in. I whine and bitch. But I keep at it until it's done. It's the commitment I make to people who use the technology I create.
The good news is that I learned a ton today. I upgraded to Hibernate 3.3, JGroups 2.6.5 and JBossCache 2.1.1. I also added a JTA Transaction Manager to the app. (Sorry for the technicalese.)
I've tried to move to Hibernate 3.3 (from 3.2) a couple times. I've spent the odd hours here or there over the course of weeks trying to get it to work. But I couldn't.
The technical issue from this morning forced my hand. I realized that I had to have 3.3 to move the ball down the field. So I went for it. And stuck at it through the technical glitches and learning curves.
What might normally have taken me a week or more or part time noodling to get done I did in 15 hours. That's not bad.
So some goodness came from today.
Another good thing is that I did move the scaling ball down the field. The new software is a lot easier on the CPU and memory (or at least they should be and so far are... I don't want to jinx myself again.) And I've got the notion of three databases running, meaning that the database load can now be split between three servers. Last night I was at a paltry two. Yesterday morning I had been crying in Chinese because I had spent three weeks trying to get some scalability with no results.
In The Structure of Scientific Revolutions Thomas Kuhn shows that progress in science happens in spurts. Things plateau for a long while and then somebody or something breaks through. Today was a breakthrough for the dNeero software. (Oh crap... there I go jinxing it again... I also said last night was a breakthrough and look where that got me.)
I'm a big believer in balance. You've got to feel the bitter to taste the sweet (or however that saying goes.) Today I got a lot of bitter.
And now at the end of the miserable day I do feel a sense of accomplishment.
And I just looked over at one of the servers and the database is pinned and the system is crashing. But, and this is important, the way it's crashing is great news! It's the database server alone that's pinned. That means that once we get the new servers into the data center we'll be able to distribute that load over three servers which will help. Whether it'll completely fix the issue is of course open for debate.
So it looks like I'll battle the beast again tomorrow. I just hope it's a little less stressful than it was today.