Sun - Week Sixteen

OK, writing this from the future, I can’t remember the details of this week…

The main forcus for this week was playing with a blade chassis, and a load of blade servers.  I was playing with a mix of X64 and SPARC systems, I was amazed that you could mix the two in an enclosed unit.  I’m still unsure why you would choose a blade over a 1U unit…

I had a hard time because I needed to liaison with several owners of blades in the chassis to get them moved to other chassis.  After this was complete it was just a case of installing new blades and upgrading some firmware.  Mick (the engineer who I was doing the work for) then installed some new blades which added storage to the blade next to them.

Interesting stuff…But I think I’ll try to leave them alone…

Sun - Week Fifteen

Week Fifteen, subtitled:

The small things

Monday morning started with the subtitle. I saw a pigeon fall from the sky, frantically flapping it’s wings causing some leaves to scatter, it made me smile. This week I shall try to look for the small things that make me smile, and report them here with the rest of my news. The morning brought a job which I have already had too much experience with, unpacking boxes which hold products…Leekes brought me much experience in how to open a card board box, and take out products, it’s a boring task, but this morning the mundanety was lifted with the fact these were new products, and there I have to stop and tell you no more :P My lunch break ended with me setting up another monitor….A flat screen for extra windows and the like, however, I have retained my ten workspaces for various things…8 of which are still as full as ever… The afternoon was spent working on some console problems with a machine in France, hunting around GMP02 for a T3 storage array and a couple of other little tasks, as well as the weekly meeting about what happened what is happening and what is going to happen…

Tuesday started with setting up a 280r with a D2 array, these were customer systems which were to be worked on. The small thing for today, is that with swivly chairs (the ones with wheels) you can scoot around the lab quite happily with very little effort, it makes every job fun :D Late morning and the afternoon was spent documenting the partitioning and mirroring of disks at jumpstart level. I’m also testing out the steps using X86, if I remember, I shall post back to say whether it worked or not….

Wednesday saw me running around and cleaning up my queue of tickets. I cleared about four away which was good, still got a number left. I spent the afternoon troubleshooting why a customer system (a V240r) wouldn’t net boot, as well as looking at some documentation, both stuff I needed to update, and other stuff I needed to understand and implement. The small thing for Wednesday almost didn’t happen, but then, waiting for the bus, the wind picked up and swirled some leaves about in a kind of whirl wind fashion…

Thursday was the start of a two day TOI (Transfer Of Information) on X64 machines. We have so far looked at what CPU’s our servers have. Then went into details about the interconnections of CPU’s/memory/north bridges. We learned how HyperTransport worked in AMD systems, and how Intel’s FSB was being replaced with QuickPath. We moved onto looking how the BIOS is loaded in such a legacy fashion, and builds up to booting the whole machine…Then the last thing before lunch was looking at how Service Processors work, how they connect to the outside world, as well as to the host itself. After lunch we looked mainly at RAID and how the chips we use for hardware RAID don’t change much from revision to revision, so we can keep similar drivers and maintain them over several platforms easily. After the course I looked at my 880 and applied some patches, I also booked out a T5440 which I am going to use to test some features of ZFS. Hehe, my small thing for today was the fact that I half followed, half made up a recipe for lasagna, and it turned out pretty good! :D

Friday went slowly, with us looking at the theory behind IPMI tool and then some IPMI in action.

Friday night is what I want to blog about now…Bill Bailey! Me and Faye went to London to see Bill in his guide to the orchestra! It was brilliant, after leaving work early and driving through London to the Royal Albert Hall, we sat there waiting for Bill to come on stage. He finally entered and began his guide, from the 1970’s villain themes played by the trombone, to the interesting fact that all oboe players love the BeeGees…. My little thing of the day, came from the ride home, when I saw a sign for yawbus, then realised that it was actually a sign saying ‘Subway’…Made me smile…

Sun - Week Fourteen

Monday starts off with me looking into this jumpstart image. It turns out to be really easy! All I needed to do was to find the Solaris 10 Update 4 image, this had a file called setup_install_server (or similar), I just used this executable file to install the image to another directory. After this it was a case of using the patchadd command:

patchadd -C

And this installed that patch onto the image for installation at install time! A bit clever or what!? The afternoon was spent looking at mirroring a machine at install over jumpstart…I didn’t get it to work :( But I shall keep trying!

Tuesday was spent looking into why I couldn’t jumpstart an X86 box with Solaris 9 update 8…I tried various machines in our lab, as well as other machines around the world, some labs don’t even have an image that old! The V65s and the LX50s that I tried either didn’t have rpower configured, or were just being a pain. Alas, I didn’t finish the task by the end of the day :(

Wednesday morning was partially spent on yesterdays problem, but I was also asked to help set up an X2200. The afternoon was spent trying to install *Windows* onto that server :| It felt wrong and awful…

Thursday starts with me continuing the install of Windows….Apparently it doesn’t know how to do sata raid :roll: Oh well, out comes the floopy disk so I can load drivers on! The afternoon was spent doing some tickets. One involved moving some firmware versions form one place to another and reorganising some of firmware files. I also got the chance to look at my 880, which I’m using to try and practice LiveUpgrades. I’m currently learning about the JumpStart profiles..

.

Friday morning I spent installing the drivers for the Windows box that didn”t understand fibre cards. I then helped David and James set up some storage for the box. I then went over to GMP02 to receive a package, this contained some fan trays for an 880 in GMP02 , so I went upstairs to fix it. I forgot to check the console line, so shut the server down while the engineer was doing work on it…I apologised and quickly went to lunch, After a slightly elongated lunch I received another email from the engineer who was very good about the situation :) I then spent the afternoon playing with custom Jumpstarts with my 880. At the moment it’s complaining about unexpected allocated inodes….Here is my profile, it’s most probably wrong:

install_type initial_install system_type standalone partitioning explicit metadb c1t0d0s7 size 8192 count 3 metadb c1t1d0s7 size 8192 count 3 filesys mirror:d0 c1t0d0s0 c1t1d0s0 10240 / filesys mirror:d0 c1t0d0s3 c1t1d0s3 10240 /var filesys mirror:d0 c1t0d0s4 c1t1d0s4 10240 /opt filesys mirror:d0 c1t0d0s5 c1t1d0s5 10240 /mnt filesys mirror:d0 c1t0d0s6 c1t1d0s6 10240 /tmp filesys c1t0d0s1 free swap isa_bits 64 boot_device any update cluster SUNWCall

Hopefully I can get something out of it :)

Looking at this doc, it seems this should work: install_type initial_install system_type standalone partitioning explicit metadb c1t0d0s7 size 8192 count 3 metadb c1t1d0s7 size 8192 count 3 filesys mirror c1t0d0s0 c1t1d0s0 10240 / filesys mirror c1t0d0s3 c1t1d0s3 10240 /var filesys mirror c1t0d0s4 c1t1d0s4 10240 /opt filesys mirror c1t0d0s5 c1t1d0s5 10240 /mnt filesys mirror c1t0d0s6 c1t1d0s6 10240 /tmp filesys c1t0d0s1 free swap isa_bits 64 boot_device any update cluster SUNWCall

Seems to work!

Processing profile - Selecting cluster (SUNWCall) - Selecting all disks - Configuring boot device - Configuring SVM State Database Replica on (c1t0d0s7) - Configuring SVM State Database Replica on (c1t1d0s7) - Configuring SVM Mirror Volume on / (c1t0d0s0) - Configuring SVM Mirror Volume on (c1t1d0s0) - Configuring SVM Mirror Volume on /var (c1t0d0s3) - Configuring SVM Mirror Volume on (c1t1d0s3) - Configuring SVM Mirror Volume on /opt (c1t0d0s4) - Configuring SVM Mirror Volume on (c1t1d0s4) - Configuring SVM Mirror Volume on /mnt (c1t0d0s5) - Configuring SVM Mirror Volume on (c1t1d0s5) - Configuring SVM Mirror Volume on /tmp (c1t0d0s6) - Configuring SVM Mirror Volume on (c1t1d0s6) - Configuring swap (c1t0d0s1) - Deselecting unmodified disk (c1t2d0) - Deselecting unmodified disk (c1t3d0) - Deselecting unmodified disk (c1t4d0) - Deselecting unmodified disk (c1t5d0)

Verifying disk configuration - WARNING: Unused disk space (c1t1d0) - WARNING: Changing the system’s default boot device in the EEPROM

Verifying space allocation - Total software size: 3014.27 Mbytes

Preparing system for Solaris install

Configuring disk (c1t0d0) - Creating Solaris disk label (VTOC)

Configuring disk (c1t1d0) - Creating Solaris disk label (VTOC)

Creating and checking UFS file systems - Creating / (c1t0d0s0) - Creating /var (c1t0d0s3) - Creating /opt (c1t0d0s4) - Creating /mnt (c1t0d0s5) - Creating /tmp (c1t0d0s6)

Creating SVM Meta Devices - Creating SVM State Replica on disk c1t0d0s7 - metadb: v4u-880h-gmp03: network/rpc/meta:default: failed to enable/disable SVM service - Creating SVM State Replica on disk c1t1d0s7 - metadb: waiting on /etc/lvm/lock - Creating SVM Mirror Volume d0 (/) - Creating SVM Mirror Volume d10 (/var) - Creating SVM Mirror Volume d20 (/opt) - Creating SVM Mirror Volume d30 (/mnt) - Creating SVM Mirror Volume d40 (/tmp)

After this it seems to go into the installation.

Also, an alarm went off right next to where we work, so most of us had ear defenders on…

Sun - Week Thirteen

Unlucky for some, will it be for me? As you possibly guess, I write parts of entries throughout the week and publish on Fridays, so at the time of writing, I have no idea if it’s going to be a good or bad week, it’s 11:50, Monday morning :P

So Monday starts off with me cycling in, sitting down to breakfast then being interrupted because a fibre cable won’t come out of it’s socket. I went down to the lab where me and the engineer pulled it out and I took the cable upstairs to bin (must do that sometime actually). While getting ready for the day Clive King popped into the office! Not sure why he’s here, but he was checking out some of the M series machines…I’m pitted against the same problems that were pestering me last week, as well as a problem with an Ultra 10 that is pingable, but won’t net boot. I made a new group for a multihead SunRay :D utmhadm myName -d 2x1 -p 0123456789ab -l 0123456789ab,cdef01234567 I spent the afternoon fixing an Ultra 10, the problem was about lab tool ‘js_config’ not being properly configured for the machine. I also had an Ultra 80 to look at, it didn’t want to boot, so after rearranging the CPUs, I concluded it was a duff system board, so another one of them is on order… As I cycled out of the campus, a squirrel, obviously daring death, leaped out in front of me, it’s tail inches away from my front wheel!

I woke up this morning and went down to the shower finding the kitchen in a mess. While making lunch I found that my tuna had been eaten and that the foil I had bought had been all screwed up. Coupled with the fact that I had forgotten my Sun badge today, I wasn’t off to a good start. At 10:30 I had a meeting with Paul about our console server (enotty), it is still on Solaris 10, the original release! I’m about to do some testing on an 880 so I can simulate a LiveUpgrade and get to know any pitfalls I might encounter.

Wednesday morning started with me replacing the Ultra 80 system board,only to find that it was faulty, so had to replace it with the old system board, send the new one back and request a new board. The system was very different to a normal ‘PC’. The RAM was really weird, a little switch rocked left or right to release the DIMMS, the CPUs have a couple of handles on (like DIMM slots), when pulled they unlock the CPU and slide out. The new system board wasn’t powering up. I whittled this down to the bottom slot not wanting to be populated with a CPU. Alas, after taking the bottom CPU out, there was still no output on the screen. During the afternoon I tried to get to grips with the LiveUpgrade stuff, but I was asked to take up another task which involved trying to get a Minnow to work. I’ve got no idea what I’m doing, everybody seems to think I do, everyone else seems to know a little more than I do too…Didn’t get it to work, will leave until tomorrow.

Well, I think this is an unlucky week. I was woken up last night by house mates coming in at 2am, with some girl that just screeched, she ended up sleeping with one of them…They made a racket until 4am, then I finally got some sleep… The morning at work hasn’t been much better, I’ve been working with this Minnow still, I think I’m trying to trouble shoot problems with an HBA, as now I’m using a unipack and still getting errors….Just swapped the SCSI cable from the upper port on the HBA, to the lower port, and it seems to recognise the disks now! Just got to wait and see whether the engineer is happy now :P Lunch seems in order… The afternoon was spent trying to get some last few things tied up before home time! This included more work on that blasted Ultra 80! Still not fixed :( After work I met up with some of the guys who work on the floor below us, me and Tom went to their house just over the M3 for a few games of cards and some pizza, was a great night :D

Friday starts with estale, one of the Sun Ray servers, crashing…Still not sure why, but suffice to say, it dragged my session down with it…Ho hum, it’s been a very busy morning due to people trying to solve this, and people submitting more tickets than usual…I’m sure they heard we were having problems and thought they’d give us a challenge! I’ve powercycled a machine from the SP, removed a booking and tried to get a switch to work. The switch wouldn’t ping or give us a console connection, so I’ve passed it over to Tom who might be able to do a little bit more! Jobs for this afternoon include making a custom jumpstart image…Oh, I also booked tickets for me and Faye to go see Bill Bailey in a couple of weeks time :D Clicky Perhaps more importantly; we’re going to be in the Royal Albert Hall :D The afternoon has been quite full. I finally fixed that problem with the T2000 as a node, I just took out all the cards, reinserted them and voila! It worked! I have also been working on getting the customer jumpstart image to work, I’ve not advanced much…But hopefully I’ll get it done by the end of Monday. I’ve also been postbooking some machines, this has so far just been making sure hosts are alive, then jumpstarting them with Solaris 10 Update 4. Early next week I shall look at making sure they don’t have extra cards or network stuff attached to them.

Sun - Week Twelve

Monday started slowly. I cycled in and not much happened in the morning. By the afternoon I had inherited a couple of tickets which I started work on after lunch. Immediately after lunch we had the weekly staff meeting, then I got cracking on some tickets…One system wasn’t turned on, so I turned it on (very taxing) and I started to look at Liam’s ticket about remaining SAN switched.

Tuesday was really busy. I finally received the RSC for the 280r that I needed. I installed that then headed over to GMP02 to look at a cluster. I had to repair the box after James had maliciously ripped it to shreds…Or rather he had broken a switch that makes the system power off when the lid is removed. I replaced said switch and some gubbins to do with that, then popped the machine back together. Upon completion, it was lunch time :D After lunch I talked to the guy who has booked the cluster system, but didn’t get any time to continue work with it. I had to configure the RSC from the 280r platform, which was weird, and after much fiddling and replacing of cables, I finally got it to work. I have also had to explain how to jumpstart a system to an engineer after directing him to the relevant documentation several times. That’s taken a large part of the the afternoon!

Wednesday was taken up completely by the cluster I’ve inherited from James. I don’t know what I’m doing, the engineer doesn’t know what is supposed to happen either. The two nodes didn’t boot in the morning, and I spent the morning locating firmware images and working out how I could up an FTP service so I could update the firmware on node b. The afternoon was spent trying to set up the FTP server, which David Cole helped with, and the finally upgraded the firmware. Then there was the problems with node a. It just wouldn’t get to OBP, it was really annoying. I reinstalled the firmware, but to no avail. It was home time before I got it working, so more fun on Thursday….

As promised, more T2000 fun on Thursday. I ran the max verbosity, max diagnostic levels of POST, with no clue as to why this system won’t boot. While I was letting the POST run, I tidied and rearranged my desk:

The afternoon was spent trying to troubleshoot further, to no avail :(

Friday morning was spent on getting this system to work. In downgraded the firmware, upgraded the firmware and a number of other things, but nothing has happened :( It is currently mid afternoon and I have a splitting headache. I can’t wait for hometime so I can see my Faye :D And Aled is coming down too!

Sun - Week Eleven

After a wonderful weekend with Faye, Monday started off with getting up early for a call with the APAC guys (Singapore and Australia), we were talking about new development life cycles - something that bores me at the best of times, and today was no exception. We do need to re-evaluate the way we fix bugs and release new versions of our lab tools though. The rest of Monday was fairly quiet, with a second meeting in the afternoon.

Tuesday was much the same as Monday…We have a work experience lad in this week, so I took him down to the lab and tidied away loads of hard drives and PCI cards. After lunch we finished off what we hadn’t done and I took him around and showed him the insides of a range of servers we had. He seemed pretty interested, I just hope I didn’t melt his brain :?

Wednesday I finally finished off the new starters document! I’ve been working on and off on it for weeks! I finally printed it out and scanned through it, then James looked through it and pointed out one or two mistakes which I fixed. Late morning and the afternoon was filled with the moving of hard drives. We are getting rid of the present shelves and replacing them with new ones, hard drives are blinking heavy!!! And there was more than a few there!

Thursday morning went slowly, with an action on a ticket requiring me to order a new Remote Service Controller. In the afternoon I had a meeting with Paul, James and Hans to talk about a 12K in Florida. I then logged on and started to investigate how it was configured.

Friday was spent on a couple of things, mainly trying to figure out what was what with this 12K still… Last thing I did was spend some time talking to a Yankie who was quite rude over the phone. He wanted to have some things pulled out of a server, like PSUs and Fans…

SUNWlibsexy

I’m just updating packages on my OpenSolaris installation, and I glanced at the screen only to find a package called “SUNWlibsexy” sitting bang in the middle of the update screen. After a quick Google, I found this: > Name: SUNWlibsexy License: Other Group: System/Libraries Version: 0.1.11 Summary: libsexy is a collection of GTK+ widgets that extend the functionality of such standard widget. From here: clicky

Just one more thing in computer science which is…Interesting…

Sun - Week Ten

So this week I’ve been on my 15k/25k maintenance course. It’s was very boring. It was basically the serengeti training all over again.

It’s basically a big box with up to 18 systems boards in. Each system board has up to 4 CPUs and 32 sticks of RAM. These system boards can be joined together to create ‘domains’, these domains can run instances of Solaris, so you effectively get one system spread over several boards…You can even add and remove system boards while a domain is running!

It’s a remarkable piece of kit…But it was pretty boring to learn…

A new design

I have redesigned my webshite, mostly because I got bored and didn’t like the old design…

You can still find it at: users.aber.a.uk/bil6

It has been tested using a number of browsers; each work OK, Opera and Safari for iPhone both have slight issues with the CSS…

Sun - Week Nine

So, two months on and we have moved from knowing nothing, to knowing a bit more about what we’re doing.

Monday started with panic, one of our SunRay servers had been upgraded, and now things didn’t work…Though we think it was an ITOPs issue, as we didn’t fix it…The server is question has been upgraded to a later version of Navada. The afternoon was littered with tickets and went very quickly.

Tuesday morning started with looking at some tickets I picked up last thing on Monday, I had an 880 to look at, this now requires me to test 32 ports on an etherlite, and then potentially label up said 32 ports and cables, then pull them so I can reset the etherlite (just resetting the etherlite with cables plugged in will send a break to all the attached machines). The afternoon was spent trying to get a v880 to work, it would appear that the RSC is flakey…Might be a case that it needs a new one, or it may be that the firmware is incompatible with something else…I also shifted a A5200, which was really heavy, to the overflow lab.

Wednesday morning was hectic; Liam and James were both out of the office, and there was a number of tickets on the queue. We all took a number each and set to work. My first job was to talk to Paul about some of the ticket details. Then I spent 20 minutes talking to Michael Clarke about a 25k problem, which turned out to be more of a problem with NIS+. I am to order in some new parts for an ultra 24 workstation as the one in question is overheating. I have just moved the Minnows to GMP02, and am waiting for a guy to come and see me about them…

Thursday came and went, with more tickets coming and going. More looking at machines that seem dead, and rebooting them. Faye came to see me for lunch :D After work, we went to the Dolphin for a pint, and we were all sad to hear that it will be closing shortly, the to a restaurant called ‘so asia’, this was all because Michael Clarke had come back for a day. We went to ‘The four horseshoes’ and won the pub quiz after arriving late.

Friday mornign started with a conversation with Paul on the bus! He had checked his car into a garage and had caught the bus. When we got into work Liam showed me how to kill off sunray sessions as he needed to use his main card and had a duff session trapped on it. I also looked at an old 65x which seemed to have a memory problem. Looking up some docs I managed to cure this. The machine is now running Solaris 10 update 4 quite happily. I also had a v240 which didn’t appear to be alive. After a reboot and fresh install, all is good. This afternoon I spent a lot of time finding a camera. The camera was so an engineer in Australia could see some LEDs on the box he had booked. I got my Sun Ray today, however, I’ve forgotten my PIN number! Silly me! :(