Mar 27, 2009 · 4 minute read · Comments
Monday I was back in the office, Steve was out coming home from a weekend in Aber and James had started a week of touring the UK.
I picked up a few tickets and knuckled down to get them done. I had one which was to simply check if some connections between a storage array and a host was OK and to provide feedback on how they were configured.
Another was to do some testing on our lab tool “js_config”, the developer has updated the current version and logged about 25 tickets to test the new and old features out. I picked one which had me installing a SPARC machine with the latest Nevada build, everything seemed to go OK.
I spent the afternoon moving a cryto card from a stable system and putting it in to a T1000, and back again, and then back again, finally placing it back in the stable system…I still have the original crypto card from the T1000 on my desk actually…
Tuesday came and it felt like Monday again :(
I spent a lot of time in the very warm GMP02 lab working with the 15K over there. I had picked up a ticket which involved getting a starcat domain up running with similar hardware to a customer setup.
I found a board with the same CPU’s, and changed the memory configuration to match too. I then had problems booting the domain, but I needed to leave it as I had other work that was more urgent.
I had to detach the host from the array and put it back again. I also fixed a problem with the host which stopped it from jumpstarting.
Wednesday felt a lot like Thursday…
I gave the morning to the 15K, it wasn’t booting, then when it did boot it wouldn’t plumb the network card. I figured this was because I reconfigured it to use a different IO boat (I had swapped the boats to match the customer config more closely). Firstly, I looked at the RAM, and found I had configured it wrong, I put this right and moved onto the whole network thing…I decided to net boot it as OBP could see the network. This worked and I handed the domain to the engineer, I also set up a place for him to put a Flash Archive which was taken from the customer system.
I also did some more things with that host and array.
There was some customer 10g fibre cards in our 25K, so I pulled them out and gave them back to the relevant engineer, I’ve still got to find some replacements…
An engineer came to me asking why his desktop wouldn’t boot, it kept timing out. We finally whittled this down to the fact he was on the wrong network, he has been moved over now and all is OK.
I also gave blood! The blood people were at work so went to donate.
Thursday is here and I’m sure it’s supposed to be holiday time…
I woke up this morning feeling really cold and had a horrid neck ache, to make matters worse the shower was cold as someone had turned off the heating. I felt generally miserable and unmotivated all day :(
I mainly did more work on the host and array, adding remote power to the array which would mean I have less to do with it later in the week/next week.
I’ve got to look at James’ racks from France this afternoon, joy of joys! they aren’t full, so hopefully between today and tomorrow I’ll get some done!
I’ve know I’ve really got to keep this blog more up to date, I’ve got a load of OpenSolaris stuff to blog about, some important lessons to be learned and stuff like that…Will try and do that tonight…
Well, it’s finally Friday! It’s been an odd old week.
Today I spent the morning documenting some of the stuff from the French lab, I’ve moved all the machines into out lab group, and have determined whether they have asset and serial numbers, I’ve collected missing serial numbers and need to find some asset stickers…
Lunch was spent updating this entry actually…And sprucing up my CV for position I’m hoping to get next year. More of that if I get the job ;)
After lunch I went back to documenting the servers again, it’s a horribly boring task, and fiddly, one slip up and you’ve got to start again…Maybe that’s just my ‘accounting’ though.
I also managed to get an x4100 to boot OpenSolaris 2008.11! I had to flash the firmware as when I found the machine, it had really old firmware, and no disks which was odd…Though I learned how to set up a TFTP server :D
Mar 20, 2009 · 1 minute read · Comments
This week was brilliant! I was on a training course: Solaris Internals.
It was great for two reasons, firstly because I missed the French lab moving over to our lab (although now I wish I had been there), and secondly, I was the only person on the course! I had an instructor all to myself, again great, but I had to make sure I paid attention as I was the only person to answer questions :P
The course focussed on kernel structure, we did a lot of stuff about memory and virtual memory and tracing it through the kernel using the modular debugger (mdb). We also looked at file systems, including a small amount on ZFS, though it was old material as the course had been written before source code was released as part of the OpenSolaris project.
All in all, it was a great week, and made even better by Faye coming down on Wednesday evening and spending the week with me :D
Mar 13, 2009 · 2 minute read · Comments
Monday started rather differently to usual. we had our lab staff meeting at 10am so we could include the APAC lab manager on the call, he had some interesting things to tell us about collecting statistics about how the lab was used, and how we are going to implant some more measures in to get more accurate results.
We still had the meeting in the afternoon to talk about our own lab, after this meeting we went down and did a lab tidy in preparation for the AMER lab managers visit on Wednesday. I also helped Paul rack some new machines which will be used for bookable LDOM’s, as well as a virtual lab host. The virtual lab host has a combination of zones and VirtualBox I think, these can be provisioned as and when needed.
Tuesday configured a multi-head Sun Ray config for an engineer. I also looked at why the JBOD on the virtual lab box seemed to be winging. It would appear that the J4200 will beep at you if it has an error, and the error could be that some ports aren’t plugged into anything! Anyway, the silencer button was pressed and it seems quite happy…However the x4600 it’s attached to is complaining of a failed PSU (though no PSU claim to have failed) and a failed fan, thus causing the “system to hot” light to illuminate.
Wednesday we had the pleasure of meeting the AMER lab manager. He had come to see Paul and also interview someone for their industrial year 09-10.
On another note I had a SunBlade 2000 which wasn’t pingable, and wouldn’t ping anything else…With Ali’s help I discovered that it was on the wrong network…It needed to be on the 129.156.175.X network, but it was on the 129.156.208.X network. I configured the VLAN and it all worked. After lunch some other machines were having problems, and it would appear I had knocked them onto the wrong network! I revered my changes and went to plug the SunBlade 2000 into a different switch.
Thursday and Friday was spent doing some minor things such as removing cables from machines, adding PCI cards and jumpstarting machines.
Mar 6, 2009 · 1 minute read · Comments
Well, this week really wasn’t very special.
I assigned some new IP’s to a couple of machines.
Evaluated quite a few bugs which were all accepted to be fixed.
The most exciting things I did included talking to a very polite American engineer about booking a couple of systems and getting Suse Linux installed on them.
I also looked at the UPS system on one of our production servers. It was complaining about the battery which had failed a test. I’ve changed the battery and I need to make sure it’s still OK.
That’s it really, a very uninteresting week!
Mar 2, 2009 · 1 minute read · Comments
The Intel Manager game is on it’s third incarnation and brings a very different form of game play.
I didn’t play the IMG2, but in the first version you bought more servers and upgraded network cards etc (choices from ‘no-name’ brands up to Intel’s finest). This new version you do no buying. You first train in various areas by completing challenges (answering an Intel related question) or taking the training course (your skill is increased over a short period of time). Once you have learnt a skill (such as multi-core) you can apply it to the employees desktop/laptop.
I think I prefer the first version, I wish they had built apon that…
Mar 1, 2009 · 2 minute read · Comments
It happened a couple of weeks ago now, I’ve upgraded the storage in beleg-iâ. It now has 4x 500GB disks!
I have them in two mirrors, one for the boot disk and one for data.
root@beleg-ia:/share/media# zpool status
scrub: none requested
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4d0s0 ONLINE 0 0 0
c4d1s0 ONLINE 0 0 0
errors: No known data errors
scrub: none requested
NAME STATE READ WRITE CKSUM
share ONLINE 0 0 0
mirror ONLINE 0 0 0
c5d0s0 ONLINE 0 0 0
c5d1s0 ONLINE 0 0 0
errors: No known data errors
The above was achieved by doing something similar to this:
prtvtoc /dev/rdsk/c4d0s0 | fmthard -s - /dev/rdsk/c4d1s0
prtvtoc /dev/rdsk/c4d0s0 | fmthard -s - /dev/rdsk/c5d0s0
prtvtoc /dev/rdsk/c4d0s0 | fmthard -s - /dev/rdsk/c5d1s0
zpool attach -f rpool c4d0s0 c4d1s0
zpool attach -f share c5d0s0 c5d1s0
First we print the vtoc (volume table of contents) and pipe that to a format command.
The -f was used to force the attachment, I followed instructions from several other sources and all has worked for me.
The ‘share’ pool is auto-mounted to /share which is cool. Now with file systems galore using ZFS, `zfs list` give me the following output:
root@beleg-ia:/share/media# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 33.1G 424G 75K /rpool
rpool/ROOT 4.79G 424G 18K legacy
rpool/ROOT/opensolaris 5.42M 424G 2.27G /
rpool/ROOT/opensolaris-1 4.79G 424G 4.55G /
rpool/dump 1018M 424G 1018M -
rpool/export 26.3G 424G 19K /export
rpool/export/home 26.3G 424G 21K /export/home
rpool/export/home/admin 28.8M 424G 28.8M /export/home/admin
rpool/export/home/ben 26.3G 424G 26.3G /export/home/ben
rpool/export/home/ben/backups 20.4M 424G 20.4M /export/home/ben/backups
rpool/swap 1018M 425G 31.9M -
share 126G 331G 19K /share
share/media 126G 331G 27K /share/media
share/media/iTunes 5.66G 331G 5.66G /share/media/iTunes
share/media/misc 911M 331G 911M /share/media/misc
share/media/movies 61.8G 331G 61.8G /share/media/movies
share/media/pictures 425M 331G 425M /share/media/pictures
share/media/standup 5.32G 331G 5.32G /share/media/standup
share/media/torip 4.50G 331G 4.50G /share/media/torip
share/media/tv 47.7G 331G 47.7G /share/media/tv
So I’m pretty happy with it at the moment. I’ll be looking at upgrading the motherboard next. Something with 8 SATA channels so I can increase my hdd’s and potentially create a RAIDz for 4x 1TB drives…But we’ll have to see how funds go…
Mar 1, 2009 · 2 minute read · Comments
I meant to blog this ages ago, and have only just remembered about it!
I wanted a way for me to access my server when I wasn’t on the local network. A friend pointed me to DynDNS, this will give you a domain name (e.g. myname.something.com) which you can update to point to your IP. However, you need a method for updating the DNS entry, this is needed because everytime I switch off my modem and turn it back on, Tiscali give me a new IP.
So, my OpenSolaris box will need to update DynDNS.
DynDNS have a couple of scripts for doing this, and how to install/run them. Clicky. I chose to use inadyn (here).
Only problem now is, I’ve got to make it run! And I really didn’t want to run it at boot every time…So it was an excellent excuse to play with SMF!
I moved the script to /lib/svc/method/inadyn-exe and created a config file in /etc/inadyn.conf
Here is my config file with some of the more important data spoofed:
admin@beleg-ia:~$ cat /etc/inadyn.conf
I then created a new manifest for SMF. The manifest file is written in XML and tells SMF what the service is, what services it depends on, what services depend on it, what to do when enabling it, what to do when disabling it and much more.
My manifest looks like this:
**<–! Only create 1 instance –>
So, after placing this in /var/svc/manifest/site/ and calling it ip-update.xml I ran it through an XML checker:
Then I imported it into the SMF repository:
svc:> validate ip-update.xml
svc:> import ip-update.xml
After this I was able to use `svcadm enable ip-update` to start the service, though it will start up after the network service has started. If the service goes down SMF will try to restart it for me! Even after two months it keeps DynDNS updated and I can happily SSH in from outside my home network!
NB: You may need to set up SSH forwarding from your home router.
Feb 27, 2009 · 2 minute read · Comments
Monday was a pretty normal Monday. I finished off that ticket about putting fibre cards into machines, and fixed a few problems that we had with it.
The afternoon was spent troubleshooting a SunBlade 2000, it wasn’t booting and the CPU was suspect. I had already done some work on this last week, but I think I failed to mention it. By the end of the day I had another SunBlade 2000 on the bench with me and was swapping CPU’s and DIMMs to see what was up, I ended up getting new errors…At the end of the day, I found some of the pins were bent :( These were giving the new errors, though seemed related to the previous ones.
The rest of the week was spent learning and writing SQL. I have been assigned a project to write this script which will look for bookings in the database which are over a certain duration. I will then email the owner of the booking that they still have this machine and if they are not using it, please unbook it.
I did also get a new system board the for SunBlade 2000, I have replaced it, but the pins on the CPU module are also damaged, so I will have to see if I can fix them, or if I’ll need to order new CPU’s…
Feb 20, 2009 · 3 minute read · Comments
The weekend was great. Faye came down on Friday and went home on Sunday, we had a really nice, weekend doing not much. Parents came round on Saturday too to show Faye the photo scrapbook Mum had made me for my birthday.
Monday I worked on a ticket which I had picked up last thing on Friday. It was to install four network cards into a T2000. The T2000 booked was on the back bench under a 3310 array (or something similar). Since my engineer had a genuine booking, I disconnected the T2000, put it back in it’s place and set about taking out old cards and adding in the new ones.
When I came across problems with the SP and asked for help, it transpired that the machine actually belonged to someone else…Tom had canceled the booking, and I had dismantled the config. I hastily set about undoing what I had done to some success, I emailed the original owner and apologised.
Tuesday I had a ticket from an engineer who wanted some extra disks in an x4100, however, these machines take two hard drives and a DVD drive, or four hard drives, but you must specify when you buy the machine. We looked for some others but he finally got a machine in the US, so it was all out of my hands.
I also found out why my X2200 wasn’t booting using the internal disks! I had placed the connections from a RAID card to the system board, the card took SAS disks but the system board only takes SATA, both are electronically the same, but I guess signals are different. Anyway, one mystery solved!
Wednesday was surprisingly busy. In the morning I installed some fibre cards and cabling from a V215 and a switch.
This was followed another remote engineer requesting me to pull some drives from an array.
Also, I moved the array and card from that X2200 to another few machines (an X4140 and an X4600).
I installed a SCSI HBA into a V250 and hooked it up to some storage.
Thursday felt like an extension of Wednesday with the frenzy never ending. I started by cabling up a couple of 280r’s to a storage array (2510). The same engineer was asking for two HBS’s in a couple of x4540’s and after lunch, asked me to uncable the origonal setup and bale both the 4540’s to the array. He later asked for two switches to be introduced and wired like so:
I helped an engineer netboot a machine, the ethernet was wrong in NIS+ so the netboot timed out, I corrected the entry and it was on it’s way.
Friday morning I came in ane started work no a stable machine that was stuck in continuus netboot. I reset the BIOS to boot from the HDD and then jump-started it with Solaris as asked by the engineer. The morning also brought problems with the x4540 and 2510 setup. After some fiddling we found that the array wasn’t configured as it should have been, so we rewired the network to match.
In the afternoon I spent quite a lot of time over in the 02 lab. I was inserting fibre cards into a couple of machines, an x2100 and a v120. Unfortunately there was no fibre switch over in that lab, so on Monday I’ve got to move the cards to other machines in the 03 lab.
Feb 13, 2009 · 2 minute read · Comments
After having a nice week off, I come into work on Monday to find myself with a ticket stuck on my queue and a meeting at 10am about said ticket…
The ticket is to do with a new feature I’m going to be adding into our inventory database. It will basically look at how long a user has booked a system for, then email them if needs be.
I spent some time looking at this project before picking up a ticket on my queue, and was also given a ticket.
The first ticket was just to check to see why a machine wasn’t accessible, neither the SP or the host was pingable. I looked at the machine and the power leads had slipped out the back but just a little…But enough to puzzle me at first glance…
The other ticket was given to my by Tom. An Indian engineer was asking for an array and a SCSI card to be moved from a T2000 to an X2200. I did this and got endless questions until the end of the day.
After work I decided it was time to hit the gym!
Tuesday was a little more interesting.
I spent the morning installing some X4250’s for an engineer, he had eight, but I only had two rack mount kits, so I racked up two of them leaving enough space to fit one between them and one could sit on top.
I cabled them up and gave them customer entries into out booking system, added them to NIS+ and all the other usual stuff.
Another ticket was to just power cycle a system’s SP…
Another SP wasn’t accepting SSH connections. After reading the documentation for the SP, I used the serial port to access the SP, then ran:
set /SP/services/ssh state=enabled | disabled
Which sorted it out nicely…
Wednesday I spent working out what the X2200 I set up on Monday didn’t want to work, I didn’t dwell on it as William was in and I needed to talk to him about the inventory database and how it worked. He talked me through the structure and basic Sybase commands.
Thursday was spent tying up loose ends, and I went to the gym after work!
Friday I helped an engineer with net booting a machine remotely. I showed him how to access the KVM on the SP and net boot it. He was still having problems so I did it manually, but at least he knows now.