New Computer

I recently upgraded my main system from a Phenom II to an Ivy Bridge (Xeon E3-1270), and this may be stating the obvious, but dang is it faster...
I have a little benchmark program I wrote to test a simple C-library of data structures, that normally took 10 seconds on my old machine, that now completes in just under 4. If I launch eight of them at once (two per real core), the hyper-threading lets them complete in just under 6 seconds a piece. Wow!
With my spare cycles, I normally run GIMPS work units, which is a distributed prime number search that consists of heavy floating point operations. The new machine processes work units about 6 times faster than the old one. I'm assuming the new AVX instructions are mostly responsible.
Which brings me to an awkward problem... I normally run 4-threads of GIMPS at a niceness of 19, which usually translates into very little CPU usage, and I just let it run without typically noticing any performance hit. But with hyper-threaded cores, the Linux scheduler notices that I've got spare CPU's, and happily schedules them with GIMPS tasks. So what ends up happening, is I get my interactive workload on a core, and a GIMPS thread on it's hyper-peer, and they split CPU roughly 50% (modulo the hyper-threading boost). That kind of hurts, but I also hate to turn off hyper-threading since it was a 30% throughput boost when I do have real multi-threaded workloads.
At the moment I'm just living with it, and turning off GIMPS if I run into a situation where I care, but that kind of sucks. I tried playing with cgroups, but from all I can tell, they're not really designed for this. You can use them to limit things to a percentage of the CPU when you're under load, but when there are spare cores they optimize for throughput, and schedule things anyway.
After digging through the Internet, I finally found a utility called cpulimit that is almost what I want. It'll let me limit the total CPU of a process tree to X% of a core, so I can limit GIMPS to 3 cores, and leave one completely spare, but I still have to manually pay adjust to it if I want to give it my whole CPU while I'm gone.
What I really want, is for it to check my load average, and automatically scale up/down the limit it's applying to GIMPS based on how busy my system is. After pulling the source for it, I think I know how to make it do that, so sometime when I get really motivated I'm going to give it a try.
Among the other fun things I've learned while setting up this machine is what happens when you don't have your static /dev setup right (the kernel can't launch init...), when you don't have a /run folder (your hostname never sets, and you can't successfully halt or reboot among other things), and Gentoo live cd's haven't setup the net.eth0 symlink by default since at least January. Oh, and don't dd your disk while you have it mounted... It doesn't end well.
I have also now become well-acquainted with NewEgg's return policies, having sent them back a bad motherboard, and a set of four 8-GB RAM chips (which I mistakenly ordered despite the fact that registered memory wasn't compatible with my motherboard). Props to NewEgg for painlessly taking it all back.  (Update: NewEgg has since refused to accept my motherboard back... :-(  )

I'd like to think this experience has made me wiser, but if nothing else, at least I now have a faster computer. :-D