Tim Bishop

Computing Officer

Electronic

Email: T.D.Bishop@kent.ac.uk
PGP Key:

Postal address

School of Computing,
University of Kent,
School of Computing, University of Kent Canterbury, Kent, CT2 7NF England

Telephone

workTel: +44 (0)1227 827237
faxFax: +44 (0)1227 762811

Blog

Only the most recent entries are included, see the full blog for more.

14:24, Wednesday, 14 October, 2009

Increasing our storage provision

During the summer we started getting tight on storage availability. It seems that usage on our home directory areas constantly increases – people never delete stuff (me included!). We were running most of our stuff through our Veritas Cluster from a pair of Sun 3511 arrays and a single 3510 array. Between them (taking mirroring in to account) we had around 3TB of space.

Now, it’s a well known fact with maintenance contracts that the cost goes up over time (parts get more scarce and more costly). So we did the sums on the cost we were paying for the old arrays and realised that over a sensible lifetime period it was cheaper to replace them. So we got a pair of Sun 2540 arrays with a 12TB capacity each.

Since our data is absolutely precious we mirror these arrays and use RAID 6. This gives us just under 10 TB of usable space, which is a fair amount more than we started with.

The next stage was bring this online. Because we use Veritas Volume Manager and the Veritas File System we were able to do this almost transparently. The new arrays were provisioned and added to the relevant diskgroups. The volumes were then mirrored on to them and then the filesystems expanded. Finally the old arrays were disconnected. All of this was done without any downtime or interruption to our users or services.

I said almost transparently though. It seems it’s not possible to change the VCS coordinator disks without taking the diskgroups offline and back online (this might be improved in VCS 5). So I rebooted the whole cluster last weekend and it was all finished.

The problem with all this clever technology? Nobody knows we’ve done it. After weeks of work we grew the filesystems just before they completely filled and without any noticable downtime. We’d probably get more gratitude if we’d let it fill up first

00:36, Thursday, 30 July, 2009

A new look CS website

A little over 3 years ago I posted about the reskin the Computer Science website at the University of Kent had just received. Well, we’re there again, although we’re now called the School of Computing.

I’ve been working on this project for the last few months. It started out with some meetings with the Information Services web team. They talked us through their current systems and then did a good job of selling their new system (which later became known as Pantheon). We decided to go with the new one, but hit a slight snag – it wasn’t quite written yet.

After a bit of waiting we finally got our hands on the code and I began the integration work. Although we’re deploying their system, we’re running our own copy of it on our servers. This is a good mix; it keeps us in sync with the central web development, but allows us to maintain the flexibility we’ve had for years. So this first integration step was a little slow, and it took some lengthy IRC conversations and a session around my desk to finally get it sorted.

Over the next month or so I spent more time tying this in to our system. I found various bugs and issues with the code and fed my fixes back to the development team where necessary. I also tried to bring across the look from our blog, and sorted out the configuration and menu structures. Then, with a working system in place, and new theme from the web team called Bacchus, I was ready to do the final integration step – tying it in to our existing build system.

At this point time was becoming more of an issue. We could have just chucked all our old ways of doing this out of the window. This might have been less effort for me, but it would have been more for everyone else. The decision was made to stick with our old infrastructure and tie the new system in to our templates. This didn’t turn out to be too hard, and only took a couple of weeks to complete.

What we’re left with is an interesting system. Our build scripts take user generated content and wrap them up with our templates to generate a HTML file. Then, at runtime, this HTML file (which contains special markup) is fed through the Pantheon system which uses the Bacchus template to render the final page. So we’ve got a baked and fried system all in one .

The final step was a week long effort to rebuild pages and tweak the various build scripts and jobs to generate reasonable looking and valid XHTML pages. This was more tedious than technical I suppose, but a satisfying finaly step to the project. Then the work passed over to our web editors to start reorganising and rebuilding the content.

And there we have it – the new Computer Science website. Just don’t dig too deep or you’ll find the pages we haven’t finished with yet .

Blog included using rawdog.

Work

I'm a Computing Officer for the Computer Science department at the University of Kent. I spend the majority of my time on Unix system administration and development. We mainly use Solaris machines, but we also have some Linux and FreeBSD servers.

Here are some of the things that I have deployed and run:

Veritas Cluster Server (VCS)

We run VCS to make our core services highly available. The system consists of four Sun Fire V240s, one Sun StorEdge 3510, two Sun StorEdge 3511s, and a bunch of network and fibre channel infrastructure. There is pretty much complete redundancy built in to this system so that in the case of hardware failure all services will continue to run unaffected.

We have the following services running within our cluster:

Caching DNS server with mirror of Kent domains. This provides resilience if the upstream Computing Service systems fail or are unreachable.
Student databases. Both MySQL and PostgreSQL databases are available for student work.
Internal databases. These back our internal systems such as our helpdesk software, website, and spam checking facilities.
Backup mail hub. Should our other mail hubs all be unavailable this service will ensure email is still delivered.
IRC. Mainly used for students to chat to each other.
Centralised filestore. This is still in development but when finished will carry filestore for all our machines.

Future plans include moving our main webserver into the cluster.

Some people might also be interested in our efforts to make VXFS quotas work over NFS on Solaris. It's fortunately a trivial fix.

Mail Hubs

We run a pair of mail hubs that receive all email for Computer Science systems. These hubs offload the work of spam checking from our end-user systems where it had an impact on other work. We currently process around 20,000 emails per day through these machines.

Apart from Exim these machines run SpamAssassin to perform the spam checking. This took some work to integrate, and some development of local tools to allow users to configure individual settings. Our user docs are available here.

Helpdesk Software (RT)

We also use RT to manage our support system. I deployed this and developed various scripts to help integrate RT with our requirements. We have improved our productivity as a result of deploying RT and we would highly recommend it to others.

Software Packaging

We run a variety of Solaris systems ranging from end-user workstations to servers with specific tasks. This requires a lot of software to be installed and distributed to our machines. We have central server which distributes its software overnight to all our machines, with specific rules and exceptions where required. This makes the effort of maintaining extra machines minimal.

For building the software we make use of the GARStow framework. This make it easy to install new software and even easier to keep things up-to-date. We're still in the progress of migrating fully to this system, but most of our core software is done. A list of packaged software can be found here.

Monitoring

We use a collection of tools for network and systems monitoring. Each has it's own role although there is some overlap. They are:

Nagios - monitors services on machines.
i-scream - monitors machine resources.
Snort - monitors network activity.

Useful Links

Computer science systems webpages
All about the Systems Group - docs, faqs, etc

Research Interests

I don't do much research that isn't directly related to whatever systems work I happen to be doing. So there isn't much here.

Access Control Lists for TCP/UDP in Java

This mini-project was an attempt to integrate some ACL functionality into the existing Java networking API. Specifically I wanted these features in the ServerSocket (for TCP) and the DatagramSocket (for UDP). The code here, integrated into the generic i-scream util package, provides a common ACL class to define an access control list, and then two replacements for the above named classes - ACLServerSocket and ACLDatagramSocket. Both of these two classes extend their non-ACL counterparts meaning they can be dropped straight in as replacements into existing code.

Documentation for the ACL classes is available here:

Source code is available here.

The compiled classes are available as part of the i-scream util package jar file, which is available from here.

You might also be interested in the Queue class which was designed to work in a multithreaded environment.

Statistics

KentIRC stats
Statistics from the local irc network.

Section navigation

School of Computing

Studying

Activities

Local users

About us

Search