Blog Moved, etc.

Well, after the incident involving a complete rebuild of Wired, I've decided to move the blog into my own home directory. It's Wordpress and should survived moving between machines better than movable type. All the old posts have been moved over.

 

This week has been a busy one. Not only have I been working on wired,  I've also been campaigning to become station manager at STAR (we still await results).

 

At work, we've got a new website as well. Check it out at http://www.perthfm.co.uk/  It's better than the old one and has some content. Talking of which, I got my photo taken and have submitted the required material for the "DJ Blog".

ResNet, FATMAN and JA.NET - A Romantic Tale

Trust me to get the title of Ian's talk wrong, eh. Well that's the correct title. And as promised I am linking to the video.

To see the embeded video, click on the read more link.

WiredSoc Presents...

... Resnet, FATMAN and JANET - (A Love Story). Or I think that's what the title of last night's talk was. Thankfully, the talk itself was more memorable than the title. There was certainly a fair amount I learned about the whole setup and history of St. Andrews connection to the world. That combined with a few pieces of kit being shown off at the end kept the attention of the audience (ooh... look at the shiny!).

A video of it has been recorded and if it makes an appearance online, I'll be sure to post.

Oh, and the talk was followed by an EGM. We now have the same person as last year looking after the website and I get a vote on the executive committee. :)

Wired Nosedives Again

The other day, wired crashed on us again. This time, no warning, some services still running. Rather odd. So I get called over to take a look at the physical box. Jaunty is there with the screen out and wired taking a rather serious hexdump:

Click here for the pic of the hexdump.

Of course, the keyboard is not responding so we cannot scroll up to get any other messages. All we could do was reboot and investigate based on this photo and the logs (which were pretty bear).

Not good and we still can't pinpoint the issue. However, today wired went down again. This time the more regular cannot access services type of going down.

However, things were not quite so straight forward. Both the physical and virtual machine were still running! It turns out that the virtual network interfaces had been dropped and came back after a networking restart.

I really think with all these issues, the migration to the other machine can only make things better.

Changing Services

We're having a bit of a changed on the wired servers. We've just picked up another physical box, allowing us to segregate the services a bit better (we've not had much luck with virtualisation).

The current plan is to move the services in the wired VM onto the new physical box. This has been completed but is not yet in use as we're testing it (OMG wired testing something!?!). This new box is also running a VM that will only provide shell services. The benefit being that if the shell system is compromised (e.g. fork bomb, root exploit) the rest of our services should remain up.

But why the move from smoked to the new box? Well, the plan is to used smoked for storage services only (e.g. databases, nfs). So we have a split between services and data.

The migration will NOT be completed until we have tested the new setup. This will involve poking the committee to use the new services. And in Kieran's words "don't just log in, type ls and exit".

We did have a bit of fun with cron jobs running across two machines. Due to one updating the pid file then the other, we ended up with a huge number of GLaDOS instances on the IRC server.

Virtually Broken

The other day saw wired's VM just decide it didn't like executing anything useful and just eat CPU (we're talking much more than usual). So, I kill the server (you can't use it) and attempt to restart it.

Unfortunately this fails. Not because somebody screwed up with tar and root like last time but because the kernel module decided to unload itself. This was actually nothing to do with the crash but did give some rather concerning error messages.

If this is a one off, we have no problems but if it happens again, we should probably consider other options to KVM. We've tried Xen which seemed to "pause" execution and resume it after an arbritary period of time.  As it didn't crash out, we had no error logs and could find no similar situations on the internet. So that option is also out.

VMWare has been toured as an option as well as OpenVZ. As we only have on VM, we could actually just run wired on the bear metal. The last option seeming the most stable as the host OS has not actually crashed out yet.

Power Goes Out... So Does wired

So, I come to St. Andrews to sort a few things out only to find a power cut when I turn up. Hmm... puts a bit of a dampener on things. Anyway, power comes back on, wired remains offline. Turns out there is a number of issues causing boot to fail (on the VM not the host). After much work from Kieran it is operational again let's hope it stays that way.

Changes Across The Board

We've had a few teething problems with out Xen based server system. We kept running into an issue where the server "pauses" for an indefinite period of time then resumes as if nothing had happened. As the system never actually crashed, there was no information in any of the logs.

Upgrading the Xen version did not fix the problem. So, we've simplified things a bit. This allows us to get a server up and providing services.

We no longer use Xen for virtualisation (though we may use it in a VM in the future). We now use KVM / Qemu. The plan is to have one VM as the main (live) system. This is backed up using LVM snapshots. The main VM "talks" to the host system holding all of the data.

Eventually further VMs will be created including Dev (testing, etc) and *possibly* a VM running Xen. Why? Well, it allows us to create more VMs in a controlled environment for users, etc.

Anyhow, service should be resumed as normal now.

Online All The Time... Hopefully!

We've had a few teething problems with out Xen based server system. We kept running into an issue where the server "pauses" for an indefinite period of time then resumes as if nothing had happened. As the system never actually crashed, there was no information in any of the logs.

Upgrading the Xen version did not fix the problem. So, we've simplified things a bit. This allows us to get a server up and providing services.

We no longer use Xen for virtualisation (though we may use it in a VM in the future). We now use KVM / Qemu. The plan is to have one VM as the main (live) system. This is backed up using LVM snapshots. The main VM "talks" to the host system holding all of the data.

Eventually further VMs will be created including Dev (testing, etc) and *possibly* a VM running Xen. Why? Well, it allows us to create more VMs in a controlled environment for users, etc.

Documentation, Documentation, Documentation

After much mumbling from other members of WiredSoc, it has been decided to create a set of detailed documentation. This project has been started by myself but I'm not very far in. A lot of the detail is missing, but it's a start and not bad for two hours.

You can read the documentation here. Only wiredsoc members can edit the content.

Busy Times Here

Thursday saw the physical installation of the new wired server. After a lot of huffing and puffing (I tell you that thing was *heavy*), it's all wired in and networked.

A few points about the new server. It's far more powerful than the current/old server. Also, we're making use of Xen virtualisation. There are many arguments for this approach, including the ability to pull a virtual machine if something goes bad. There's also the flip side - more complicated administration.

A decision has been made to make the transition between the old and new server progressive. So, we launch the new service one at a time. This should reduce the problems we run into and allows the services to continue to be used until the new ones are ready.

One example is the database migration. We run both a mysql and a postgresql server. So far only the mysql server has been migrated. The process was simple, get mysql to dump everything (that includes using the --all-databases flag) and them import it into the new server. Once this was completed, we took the old server offline and updated the hosts file on our server. This was done as the hosts file was ponting the aliases to the old server not the new server as our DNS server is. Once the entries were removed, we fell back on the DNS server and the new server was providing the data.

This kept most things working except for those pointing to the old server directly. These have now mostly been updated and are working again.

Wired is not the only server I have been bringing online. I have also been working on my own family server (steelehost.dyndns.org). This is acting as a web, file, backup and proxy server. The proxy part may sound a bit odd for a family but it is running squid and ad removal. No annoying adverts! However, the downside is the ad removal can be a little overzelous. Some corporate websites have the banners replaced with "this ad has been zapped". I'll post a screenshot soon on that one.

The plan is to eventually get subversion on the server and use it for various personal software projects.

Talking Servers

Last night saw Kieran giving the society a talk on the current server setup and the proposed setup for the new server.

Amongst the more interesting facts were:

* Wired's e-mails are spam filtered by the university mail server. Even though my inbox is full of messages about "love sticks" and pills, most of it is apparently dropped.
* I am not allowed to blacklist logcheck emails as spam... :)
* On the new server, there will be a virtual server dedicated to the resource intensive service that is.... IRC.

* Kieran is to retain root access until at least the summer is over as he has physical access working in St. Andrews. Working in Dundee, I'm still too far away to stop him.

Still, it was a very interesting presentation. I thought I'd better not write about it on the night due to the consumption of small amounts intoxicating beveridges.

Debian and OpenSSL

Well, you can usually rely on Debian to be secure and stable. It still is. But a story has been published recently stating basically the Debian edited version of OpenSSL doesn't create very random keys. Looking at the source code, it appears that they commented out a line of code that obtained a block of memory (but doesn't write to it) then reads it. This is a bug in most programs as you don't know what's there but works reasonably well as a random seed.

As we no longer have such a random seed, the keys become predicable. Just like old computer games. The reason: random number generators are not actually random. I was taught at school that they follow a list of random values that is pre-set. So the same seed will produce the same numbers. Which actually proves useful for testing. Not so much for keeping things secure.

Kieran has informed me the servers have been updated (bar starfm as it takes out the playout system when we do it).

Keeping an Eye on Things

To keep an eye on servers, systems administrators often make use of
various monitoring tools. For example wired makes use of programs such
as logcheck, integrit and tiger. These are all valuable tools checking
system logs, file changes and system changes (these can suggest an
intrusion has occurred).

Being useful tools, they also tend to
e-mail the administration team with regular updates. The down side -
they can do it several times a day. For example, my inbox currently has
various messages along the lines of

May  7 23:03:40 wired sshd[304]: error: PAM: Permission denied for illegal user root from aaa.bbb.com

Ah,
the joy of botnets. I know this is the downside of such tools but
something I will have to put up with as we look out for real problems
that are within our control.

P.S. The root account is disabled for remote login if you're feeling bored / silly enough to try it.

Forgetfulness and Last Minute

There seems to be a rumour kicking around that students always do things last minute or forget all together. I'd like to dispel this myth, but I did not get out of my pit until 0845 today therefore running late for a tutorial. Not a good move but it's the last week of classes and I've lasted out so far.

Anyway, yesterday we were asked to provide assistance for live artists on one of STAR's programs less than 24hrs before the show. Thankfully, a few of the tech team did step up to the plate and help out! But, we technically should have said it was not doable.

On the forgetfulness front... I did walk in to the studio the other day to a recorded show still running on loop from CD the night before. It looks as though they expected people to turn up at midnight / 1am to stop the CD while they were away.  No great problem fixing it... but it gave me an excuse to play Meat Loaf on the radio!

A bit of tech update as well. Wired's new server seems to be moving along nicely (according to the updates I've read from Hash9). We are going virtual with this server using Xen (the hardware supports it). Should be a fun new experience as we currently run separate boxes. However, the assistant sysadmin (Jaunty) does claim he's had training in virtualisation. A possible scape goat when things go wrong... :)

...only joking Jaunty!