Sun - Week Thirtyeight

Monday I was back in the office, Steve was out coming home from a weekend in Aber and James had started a week of touring the UK. I picked up a few tickets and knuckled down to get them done. I had one which was to simply check if some connections between a storage array and a host was OK and to provide feedback on how they were configured. Another was to do some testing on our lab tool “js_config”, the developer has updated the current version and logged about 25 tickets to test the new and old features out. I picked one which had me installing a SPARC machine with the latest Nevada build, everything seemed to go OK. I spent the afternoon moving a cryto card from a stable system and putting it in to a T1000, and back again, and then back again, finally placing it back in the stable system…I still have the original crypto card from the T1000 on my desk actually…

Tuesday came and it felt like Monday again :( I spent a lot of time in the very warm GMP02 lab working with the 15K over there. I had picked up a ticket which involved getting a starcat domain up running with similar hardware to a customer setup. I found a board with the same CPU’s, and changed the memory configuration to match too. I then had problems booting the domain, but I needed to leave it as I had other work that was more urgent. I had to detach the host from the array and put it back again. I also fixed a problem with the host which stopped it from jumpstarting.

Wednesday felt a lot like Thursday… I gave the morning to the 15K, it wasn’t booting, then when it did boot it wouldn’t plumb the network card. I figured this was because I reconfigured it to use a different IO boat (I had swapped the boats to match the customer config more closely). Firstly, I looked at the RAM, and found I had configured it wrong, I put this right and moved onto the whole network thing…I decided to net boot it as OBP could see the network. This worked and I handed the domain to the engineer, I also set up a place for him to put a Flash Archive which was taken from the customer system. I also did some more things with that host and array. There was some customer 10g fibre cards in our 25K, so I pulled them out and gave them back to the relevant engineer, I’ve still got to find some replacements… An engineer came to me asking why his desktop wouldn’t boot, it kept timing out. We finally whittled this down to the fact he was on the wrong network, he has been moved over now and all is OK. I also gave blood! The blood people were at work so went to donate.

Thursday is here and I’m sure it’s supposed to be holiday time… I woke up this morning feeling really cold and had a horrid neck ache, to make matters worse the shower was cold as someone had turned off the heating. I felt generally miserable and unmotivated all day :( I mainly did more work on the host and array, adding remote power to the array which would mean I have less to do with it later in the week/next week. I’ve got to look at James’ racks from France this afternoon, joy of joys! they aren’t full, so hopefully between today and tomorrow I’ll get some done! I’ve know I’ve really got to keep this blog more up to date, I’ve got a load of OpenSolaris stuff to blog about, some important lessons to be learned and stuff like that…Will try and do that tonight…

Well, it’s finally Friday! It’s been an odd old week. Today I spent the morning documenting some of the stuff from the French lab, I’ve moved all the machines into out lab group, and have determined whether they have asset and serial numbers, I’ve collected missing serial numbers and need to find some asset stickers… Lunch was spent updating this entry actually…And sprucing up my CV for position I’m hoping to get next year. More of that if I get the job ;) After lunch I went back to documenting the servers again, it’s a horribly boring task, and fiddly, one slip up and you’ve got to start again…Maybe that’s just my ‘accounting’ though. I also managed to get an x4100 to boot OpenSolaris 2008.11! I had to flash the firmware as when I found the machine, it had really old firmware, and no disks which was odd…Though I learned how to set up a TFTP server :D

Sun - Week Thirty Seven

This week was brilliant! I was on a training course: Solaris Internals. It was great for two reasons, firstly because I missed the French lab moving over to our lab (although now I wish I had been there), and secondly, I was the only person on the course! I had an instructor all to myself, again great, but I had to make sure I paid attention as I was the only person to answer questions :P

The course focussed on kernel structure, we did a lot of stuff about memory and virtual memory and tracing it through the kernel using the modular debugger (mdb). We also looked at file systems, including a small amount on ZFS, though it was old material as the course had been written before source code was released as part of the OpenSolaris project.

All in all, it was a great week, and made even better by Faye coming down on Wednesday evening and spending the week with me :D

Sun - Week Thirtysix

Monday started rather differently to usual. we had our lab staff meeting at 10am so we could include the APAC lab manager on the call, he had some interesting things to tell us about collecting statistics about how the lab was used, and how we are going to implant some more measures in to get more accurate results. We still had the meeting in the afternoon to talk about our own lab, after this meeting we went down and did a lab tidy in preparation for the AMER lab managers visit on Wednesday. I also helped Paul rack some new machines which will be used for bookable LDOM’s, as well as a virtual lab host. The virtual lab host has a combination of zones and VirtualBox I think, these can be provisioned as and when needed.

Tuesday configured a multi-head Sun Ray config for an engineer. I also looked at why the JBOD on the virtual lab box seemed to be winging. It would appear that the J4200 will beep at you if it has an error, and the error could be that some ports aren’t plugged into anything! Anyway, the silencer button was pressed and it seems quite happy…However the x4600 it’s attached to is complaining of a failed PSU (though no PSU claim to have failed) and a failed fan, thus causing the “system to hot” light to illuminate.

Wednesday we had the pleasure of meeting the AMER lab manager. He had come to see Paul and also interview someone for their industrial year 09-10. On another note I had a SunBlade 2000 which wasn’t pingable, and wouldn’t ping anything else…With Ali’s help I discovered that it was on the wrong network…It needed to be on the 129.156.175.X network, but it was on the 129.156.208.X network. I configured the VLAN and it all worked. After lunch some other machines were having problems, and it would appear I had knocked them onto the wrong network! I revered my changes and went to plug the SunBlade 2000 into a different switch.

Thursday and Friday was spent doing some minor things such as removing cables from machines, adding PCI cards and jumpstarting machines.

Sun - Week Thirtyfive

Well, this week really wasn’t very special.

I assigned some new IP’s to a couple of machines. Evaluated quite a few bugs which were all accepted to be fixed.

The most exciting things I did included talking to a very polite American engineer about booking a couple of systems and getting Suse Linux installed on them. I also looked at the UPS system on one of our production servers. It was complaining about the battery which had failed a test. I’ve changed the battery and I need to make sure it’s still OK.

That’s it really, a very uninteresting week!

Intel Manager 3: Unseen Forces

The Intel Manager game is on it’s third incarnation and brings a very different form of game play. I didn’t play the IMG2, but in the first version you bought more servers and upgraded network cards etc (choices from ‘no-name’ brands up to Intel’s finest).  This new version you do no buying.  You first train in various areas by completing challenges (answering an Intel related question) or taking the training course (your skill is increased over a short period of time).  Once you have learnt a skill (such as multi-core) you can apply it to the employees desktop/laptop.

I think I prefer the first version, I wish they had built apon that…

Beleg-Iâ gets an upgrade

It happened a couple of weeks ago now, I’ve upgraded the storage in beleg-iâ.  It now has 4x 500GB disks! I have them in two mirrors, one for the boot disk and one for data.

root@beleg-ia:/share/media# zpool status pool: rpool state: ONLINE scrub: none requested config:

NAME        STATE     READ WRITE CKSUM rpool       ONLINE       0     0     0 mirror    ONLINE       0     0     0 c4d0s0  ONLINE       0     0     0 c4d1s0  ONLINE       0     0     0

errors: No known data errors

pool: share state: ONLINE scrub: none requested config:

NAME        STATE     READ WRITE CKSUM share       ONLINE       0     0     0 mirror    ONLINE       0     0     0 c5d0s0  ONLINE       0     0     0 c5d1s0  ONLINE       0     0     0

errors: No known data errors

The above was achieved by doing something similar to this:

prtvtoc /dev/rdsk/c4d0s0 | fmthard -s - /dev/rdsk/c4d1s0

prtvtoc /dev/rdsk/c4d0s0 | fmthard -s - /dev/rdsk/c5d0s0

prtvtoc /dev/rdsk/c4d0s0 | fmthard -s - /dev/rdsk/c5d1s0

zpool attach -f rpool c4d0s0 c4d1s0

zpool attach -f share c5d0s0 c5d1s0

First we print the vtoc (volume table of contents) and pipe that to a format command. The -f was used to force the attachment, I followed instructions from several other sources and all has worked for me.

The ‘share’ pool is auto-mounted to /share which is cool.  Now with file systems galore using ZFS, `zfs list` give me the following output:

root@beleg-ia:/share/media# zfs list NAME                            USED  AVAIL  REFER  MOUNTPOINT rpool                          33.1G   424G    75K  /rpool rpool/ROOT                     4.79G   424G    18K  legacy rpool/ROOT/opensolaris         5.42M   424G  2.27G  / rpool/ROOT/opensolaris-1       4.79G   424G  4.55G  / rpool/dump                     1018M   424G  1018M  - rpool/export                   26.3G   424G    19K  /export rpool/export/home              26.3G   424G    21K  /export/home rpool/export/home/admin        28.8M   424G  28.8M  /export/home/admin rpool/export/home/ben          26.3G   424G  26.3G  /export/home/ben rpool/export/home/ben/backups  20.4M   424G  20.4M  /export/home/ben/backups rpool/swap                     1018M   425G  31.9M  - share                           126G   331G    19K  /share share/media                     126G   331G    27K  /share/media share/media/iTunes             5.66G   331G  5.66G  /share/media/iTunes share/media/misc                911M   331G   911M  /share/media/misc share/media/movies             61.8G   331G  61.8G  /share/media/movies share/media/pictures            425M   331G   425M  /share/media/pictures share/media/standup            5.32G   331G  5.32G  /share/media/standup share/media/torip              4.50G   331G  4.50G  /share/media/torip share/media/tv                 47.7G   331G  47.7G  /share/media/tv

So I’m pretty happy with it at the moment.  I’ll be looking at upgrading the motherboard next.  Something with 8 SATA channels so I can increase my hdd’s and potentially create a RAIDz for 4x 1TB drives…But we’ll have to see how funds go…

DynDNS and my server

I meant to blog this ages ago, and have only just remembered about it!

I wanted a way for me to access my server when I wasn’t on the local network.  A friend pointed me to DynDNS, this will give you a domain name (e.g. myname.something.com) which you can update to point to your IP.  However, you need a method for updating the DNS entry, this is needed because everytime I switch off my modem and turn it back on, Tiscali give me a new IP.

So, my OpenSolaris box will need to update DynDNS. DynDNS have a couple of scripts for doing this, and how to install/run them.  Clicky.  I chose to use inadyn (here). Only problem now is, I’ve got to make it run!  And I really didn’t want to run it at boot every time…So it was an excellent excuse to play with SMF!

I moved the script to /lib/svc/method/inadyn-exe and created a config file in /etc/inadyn.conf Here is my config file with some of the more important data spoofed:

admin@beleg-ia:~$ cat /etc/inadyn.conf --username user1 --password password --update_period 6000 --background --alias myserver.example.com

I then created a new manifest for SMF.  The manifest file is written in XML and tells SMF what the service is, what services it depends on, what services depend on it, what to do when enabling it, what to do when disabling it and much more. My manifest looks like this: > **

**




**<–! Only create 1 instance –> **

**

**

**

**



** **

**

IP updator

**


So, after placing this in /var/svc/manifest/site/ and calling it ip-update.xml I ran it through an XML checker:

xmllint ip-update.xml

Then I imported it into the SMF repository:

svccfg

svc:>  validate ip-update.xml svc:>  import ip-update.xml

After this I was able to use `svcadm enable ip-update` to start the service, though it will start up after the network service has started.  If the service goes down SMF will try to restart it for me!  Even after two months it keeps DynDNS updated and I can happily SSH in from outside my home network!

NB: You may need to set up SSH forwarding from your home router.

Sun - Week Thirtyfour

Monday was a pretty normal Monday. I finished off that ticket about putting fibre cards into machines, and fixed a few problems that we had with it. The afternoon was spent troubleshooting a SunBlade 2000, it wasn’t booting and the CPU was suspect. I had already done some work on this last week, but I think I failed to mention it. By the end of the day I had another SunBlade 2000 on the bench with me and was swapping CPU’s and DIMMs to see what was up, I ended up getting new errors…At the end of the day, I found some of the pins were bent :( These were giving the new errors, though seemed related to the previous ones.

The rest of the week was spent learning and writing SQL. I have been assigned a project to write this script which will look for bookings in the database which are over a certain duration. I will then email the owner of the booking that they still have this machine and if they are not using it, please unbook it.

I did also get a new system board the for SunBlade 2000, I have replaced it, but the pins on the CPU module are also damaged, so I will have to see if I can fix them, or if I’ll need to order new CPU’s…

Sun - Week Thirtythree

The weekend was great. Faye came down on Friday and went home on Sunday, we had a really nice, weekend doing not much. Parents came round on Saturday too to show Faye the photo scrapbook Mum had made me for my birthday.

Monday I worked on a ticket which I had picked up last thing on Friday. It was to install four network cards into a T2000. The T2000 booked was on the back bench under a 3310 array (or something similar). Since my engineer had a genuine booking, I disconnected the T2000, put it back in it’s place and set about taking out old cards and adding in the new ones. When I came across problems with the SP and asked for help, it transpired that the machine actually belonged to someone else…Tom had canceled the booking, and I had dismantled the config. I hastily set about undoing what I had done to some success, I emailed the original owner and apologised.

Tuesday I had a ticket from an engineer who wanted some extra disks in an x4100, however, these machines take two hard drives and a DVD drive, or four hard drives, but you must specify when you buy the machine. We looked for some others but he finally got a machine in the US, so it was all out of my hands. I also found out why my X2200 wasn’t booting using the internal disks! I had placed the connections from a RAID card to the system board, the card took SAS disks but the system board only takes SATA, both are electronically the same, but I guess signals are different. Anyway, one mystery solved!

Wednesday was surprisingly busy. In the morning I installed some fibre cards and cabling from a V215 and a switch. This was followed another remote engineer requesting me to pull some drives from an array. Also, I moved the array and card from that X2200 to another few machines (an X4140 and an X4600). I installed a SCSI HBA into a V250 and hooked it up to some storage.

Thursday felt like an extension of Wednesday with the frenzy never ending. I started by cabling up a couple of 280r’s to a storage array (2510). The same engineer was asking for two HBS’s in a couple of x4540’s and after lunch, asked me to uncable the origonal setup and bale both the 4540’s to the array. He later asked for two switches to be introduced and wired like so: I helped an engineer netboot a machine, the ethernet was wrong in NIS+ so the netboot timed out, I corrected the entry and it was on it’s way.

Friday morning I came in ane started work no a stable machine that was stuck in continuus netboot. I reset the BIOS to boot from the HDD and then jump-started it with Solaris as asked by the engineer. The morning also brought problems with the x4540 and 2510 setup. After some fiddling we found that the array wasn’t configured as it should have been, so we rewired the network to match. In the afternoon I spent quite a lot of time over in the 02 lab. I was inserting fibre cards into a couple of machines, an x2100 and a v120. Unfortunately there was no fibre switch over in that lab, so on Monday I’ve got to move the cards to other machines in the 03 lab.

Sun - Week Thirtytwo

After having a nice week off, I come into work on Monday to find myself with a ticket stuck on my queue and a meeting at 10am about said ticket… The ticket is to do with a new feature I’m going to be adding into our inventory database. It will basically look at how long a user has booked a system for, then email them if needs be. I spent some time looking at this project before picking up a ticket on my queue, and was also given a ticket. The first ticket was just to check to see why a machine wasn’t accessible, neither the SP or the host was pingable. I looked at the machine and the power leads had slipped out the back but just a little…But enough to puzzle me at first glance… The other ticket was given to my by Tom. An Indian engineer was asking for an array and a SCSI card to be moved from a T2000 to an X2200. I did this and got endless questions until the end of the day. After work I decided it was time to hit the gym!

Tuesday was a little more interesting. I spent the morning installing some X4250’s for an engineer, he had eight, but I only had two rack mount kits, so I racked up two of them leaving enough space to fit one between them and one could sit on top. I cabled them up and gave them customer entries into out booking system, added them to NIS+ and all the other usual stuff. Another ticket was to just power cycle a system’s SP… Another SP wasn’t accepting SSH connections. After reading the documentation for the SP, I used the serial port to access the SP, then ran: set /SP/services/ssh state=enabled | disabled Which sorted it out nicely…

Wednesday I spent working out what the X2200 I set up on Monday didn’t want to work, I didn’t dwell on it as William was in and I needed to talk to him about the inventory database and how it worked. He talked me through the structure and basic Sybase commands.

Thursday was spent tying up loose ends, and I went to the gym after work!

Friday I helped an engineer with net booting a machine remotely. I showed him how to access the KVM on the SP and net boot it. He was still having problems so I did it manually, but at least he knows now.