DenBolle's hok

Semi Realtime

Tip : afbeeldingen staan verkleind op de site. Klik erop om ze vergroot weer te geven.
scheidslijn Think! scheidslijn

kijk ook eens hier:

scheidslijnThink! scheidslijnThink! scheidslijnMAG - Motorrijders Actie Groep scheidslijnBoshuizen motorservice raalte scheidslijnLimes ICT scheidslijn

An die Arbeit.

Ondanks de voorbereidingen liep het allemaal net wat anders als gepland. Zo waren er eigenlijk maar drie problemen die impact hadden.

We startten wat later dan gepland door een uitgelopen backup. Beter safe then sorry. Tijdens het omprikken van de kabels bleek dat de kast waarin de apparatuur stond precies met een poot op de vloertegel stond die ik er uit moest werken. Om dat goed voor elkaar te krijgen moest er om de kast heen extra tegels worden verwijderd zodat de eigenlijke tegel die ik er uit moest halen vrij kwam te liggen. Op zich niet erg maar ..
Door dat gebouw en geheister zat een kabel me hinderlijk in de weg. Dat ding hing niet weggewerkt vervelend te wezen. Omdat nu toch alles plat lag mooi een moment om de kabel door een ander te vervangen en weg te werken. Kabel opgesnort, ingeprikt, mooi weggewerkt en verder gaan.
En toen. tja.. toen bleek ineens dat dat rare roze kleurtje van die kabel niet voor niets was. Het duurde even voor ik me realiseerde dat die kabel een crosslink was. De oude kabel weer op de plek en alles werkte weer. (va. 11:51 uur)

Althans.. Op de een of ander manier ging het kopiëren van de data niet zo snel als we dachten. Waar een beetje snelheid nodig was bleek de copy slag traag te zijn, en waar we op een paar uur data pompen hadden gerekend werd het al snel een dikke 40 uur (is dus ergens maandag ochtend). Daarom op een noodplan overgestapt, en maken we de laatste stappen volgende week af.

En op het laatst (ik ben bijna aan bewaker 3 toe, ze werden telkens afgewisseld) bleek dat het hele printer systeem plat lag. De printerserver had een schop gekregen en aangezien onze printers hun configuraties uit de printerserver trekken ging dat fout. Na een kort telefoon gesprek met een van de locale jongens die dit soort dingen vaker bij de veter had gehad besluiten we alsnog om de printeromgeving een keer plat te gooien en dan op te starten. Langzaam aan (zeg een 5-10 min/ printer, we hebben er bijna 20 staan) kregen we ze stuk voor stuk weer aan de praat en aan het printen.

Dus de ingeschatte eindtijd van 1800 hrs. werd wat overtrokken…

Voor de geïnteresseerde, hieronder staat de hele listing (geanonimiseerd en ontdaan van specifieke tech info) van een hele dag waanzinnig lekker samenwerken op een site, met 4 landen (Nederland, Polen, België, Italië), serieus en minder serieus gegein.

een lange dag, maar met plezier gedaan. En… als het goed is merken de gebruikers er niets van.

De spelers hieronder:  Italian (in de gesprekken Ig-1) : een van de project managers werkend vanuit Itailië,  Polish-1 , Polish-2 en Polish-3 (respectievelijk Pg-1, Pg-2 en Pg-3 in de gesprekken): de poolse jongens in het poolse datacenter, werken met de VM ware machines, unix storage en Backup omgeving. C, K, A zijn testers/adviseurs die op afroep voor ons klaar stonden.

8:26 [ Italian  ] Morning Den
8:26 [ Polish-1 ] Hi Den!
8:26 [ Polish-2 ] Hi Den
8:27 [ DenBolle ] Goodmorning everyone!
                  [Ig-1] is generous : he treats us all in
                  coffee!
8:27 [ Italian  ] yes this for [Pg-2]
                  for [Pg-1]
8:28 [ Polish-1 ] thx
8:28 [ Italian  ] for Den that has to wake up
8:28 [ DenBolle ] Gracie!
8:28 [ Italian  ] yw
8:29 [ Polish-2 ] Thank you!
9:00 [ Italian  ] Hi [Pg-3]
9:00 [ Polish-3 ] Hello
9:00 [ Italian  ] please advise us when done
9:00 [ Polish-3 ] around 10 minutes left I think
9:01 [ DenBolle ]  ah ok
9:01 [ Polish-3 ] blowing from home will not speed it up,
                  sorry
9:02 [ Polish-2 ] ;-)
9:02 [ Italian  ] 9:02
9:02 [ DenBolle ] I'm in the office, I could spin up the
                  tape?
9:02 [ Italian  ] good idea
9:02 [ DenBolle ] use a microwave to compress the data on
                  the tape?
9:03 [ Polish-3 ] Den, you are the biggest of us /except
                  [Pg-1]/ could you tell few words to the
                  Adic Lib to write a little bit faster ?
9:04 [ DenBolle ] sure..
                  [walks into server room with a stick and
                  blowtorch]
9:04 [ Polish-3 ] ;-)
9:04 [ Italian  ] roftl
                  better ..... grin grin
9:06 [ DenBolle ] ok, I've discussed it with the lib, he's
                  appologizing and will work harder.
                  btw I never realized that they actually
                  can scream if you put them on fire.
9:06 [ Polish-3 ] buahahahaha
9:08 [ Polish-3 ] I though Adic is a woman..strange world..
9:10 [ DenBolle ] It screamed like a woman though..
9:14 [ Polish-3 ] btw started to write system state...
                  she is coming or he...
9:15 [ DenBolle ] she is coming..  this story get's a nice twist.
                  Go on [Pg-3]
9:16 [ Polish-3 ] heheh who will make some pictures - ->Den?
                  It is done gents!
9:16 [ DenBolle ] and put them on a site " www.hornytapelibs.com" ?
                  ok, let go on then
9:16 [ Polish-3 ] yep
9:17 [ Polish-2 ] it looks like I've missed some nice stories here
                  ok, I'm proceeding with stopping the BU1SMLXV0001
9:18 [ Polish-3 ] well..it will not be published on our webside
9:18 [ Polish-1 ] Can I start stopping VMs now?
9:18 [ Polish-3 ] logged off
9:19 [ DenBolle ] ok, I'm over to wireless to be safe.. just a se
9:19 [ DenBolle ] and back
9:20 [ Polish-3 ] [Pg-1] could you call me when the servers will be
                  back - I will start backups - I mean afternoon
9:20 [ Polish-1 ] Sure.
9:20 [ Polish-3 ] 504 110 321
                  cool,
9:31 [ DenBolle ] is this working
9:31 [ Polish-2 ] yep, now it's working
9:31 [ DenBolle ] mhm. glitch..
                  what did I miss?
9:31 [ Italian  ] ;-)
9:31 [ Polish-1 ] I know. Some functional accounts logged
                  on the console. I need to power them down
                  manually
9:32 [ DenBolle ] last remark i've seen is froim [Pg-3] :
                  9:32 [ Polish-3 ] 504 110 321
                  cool,
9:35 [ Polish-2 ] I can't paste the OCS chat, because it
                  says that it is too large..
                  So to summarize it quickly: [Pg-3] logged off,
                  because his wife was chasing him. I've stopped
                  the BU1SMLXV0001 and [Pg-1] is stopping now
                  other VMs and ESX hosts.
9:36 [ Italian  ] All Vms stopped ...great job [Pg-1]
9:36 [ DenBolle ] no problem..  So no strange things yet and
                  all goes as planned. Besides the huntdown of
                  [Pg-3]'s wife.
9:36 [ Italian  ] ;-)
9:36 [ Polish-2 ] Maybe she heard the library's screams...
9:37 [ DenBolle ] tsk.. women united...  If you hurt one ...
9:37 [ Polish-2 ] ;-))
9:38 [ Italian  ] when finished I'll copy this chat and send
                  to Manager1, Manager2 and Manager3
                  with title "drunk men at work"
9:38 [ DenBolle ] grin grin.. and we never are allowed to
                  work on saterday..
9:44 [ Polish-1 ] @all: All VMs are powered down now. All
                  ESX hosts are shutted down and powered off now.
9:44 [ DenBolle ] ok, thx
9:44 [ Italian  ] Den ..... go there and do your best
9:45 [ Polish-2 ] no no
                  now it's my part
                  please hold on
9:45 [ DenBolle ] [stopping]
9:45 [ Italian  ] sorrrrrrryyyyyy
9:45 [ Polish-2 ] Den, I'll let you know
                  It will take approximately 15 mins
9:45 [ DenBolle ] I'll be waiting for you
                  [not to self : this chat's getting stranger
                  by the sentence]
10:01 [ DenBolle ] I've got screaming equipment
10:01 [ Polish-2 ] OK, all FC switches and FAS270`s are going down
                   or are already down.
                   This is the reason of those screams.
10:02 [ DenBolle ] ok, let me check in the room when they actually
                   die..
10:02 [ Polish-2 ] Let's wait now 2-3 min, just to be sure that all
                   went down.
10:03 [ DenBolle ] sure
10:03 [ Polish-2 ] After this 2-3 mins, I'd like to ask you, Den,
                   to power off the following devices:
                   BU2DSBS001
                   BU2DSBS002
                   BU1DSBS001
                   BU1DSBS002
                   BU2DSNTP001
                   BU2DSNTP002
                   BU1DSNTP001
                   BU1DSNTP002
                   so - all 4 FAS270's and all 4 FC switches.
10:04 [ DenBolle ] I'll do. I'll start at 10.05
10:05 [ Polish-2 ] ok
10:08 [ DenBolle ] ok, all shelfs are down (no more hd activity) :
                   I'll go shutdown the devices.
                   Do you want me to reconnect the fibers straight
                   away?
10:09 [ Polish-2 ] Did you power off the additional shelves as well?
10:09 [ DenBolle ] no, thats going to mbe done now
10:10 [ Polish-2 ] To be honest - it's probably not needed.
10:10 [ DenBolle ] if you feel safer when I do: no prblem.
10:10 [ Polish-2 ] OK - If you want to do so, we can do it. Just to
                   remember: later, when we will be powering it on,
                   we need to start them in reverse order.
                   So first of all: the additional disk shelves,
                   after that 2-3 min break to let the disks spin up
                   and afterwards the FAS270 controllerrs
10:12 [ DenBolle ] ok,  I'm in, back in 10 -15 mins
10:12 [ Polish-2 ] one additional thing
10:12 [ DenBolle ] yes
10:12 [ Polish-2 ] I've set new IPs for BU2DSNTP001 & BU2DSNTP002.
                   So please reconnect also their ethernet cable to
                   the new subnet.
                   ..and you have to reconnect the FC cable as well
                   as we discussed on the diagram.
10:13 [ DenBolle ] I'll do, stay tuned
10:13 [ Polish-2 ] th
                   thx
10:40 [ DenBolle ] ok, almost done, just one issue.
                   the 4 utp cables for the 270's  : the core switch
                   is full, give me 20 mins to isolate cables we can
                   disconnect
10:41 [ Polish-2 ] 4 utp cables?
10:42 [ DenBolle ] yes the ethernet cables
10:42 [ Polish-2 ] to the BU2DSNTP001 & BU2DSNTP002 4 cables are
                   connected ?
10:42 [ DenBolle ] yes
10:42 [ Polish-2 ] they have dual ethernet ports, however only one port
                  (e0a) is used
10:43 [ DenBolle ] ok, in that case I'm still one port short : got one
                   emtpy one..
10:43 [ Polish-2 ] ok, I see
                   ok, we will wait
                   when you finish
                   you can power on following devices:

                   BU1DSBS001
                   BU1DSBS002
                   BU2DSNTP001
                   BU2DSNTP002
                   BU1DSNTP001
                   BU1DSNTP002
                   and please do not power on ESX servers
10:44 [ DenBolle ] No, I;ll do.  ok, [Ig-1], in that case could you
                   serve all once more a cup of coffe in the mean
                   time?
10:46 [ Italian  ] here they are ......
                   I'm boiling water
                   .........
                   and ..... that's it
11:03 [ DenBolle ] ok, done with the cables, powered up shelfs
                   (not FAS270's), and BU1DSB001 and BU1DSB002 are
                   starting. When these are finshed I'll boot
                   BU2NSNTP001/2 and BU1DSNTP001/2
11:03 [ Polish-2 ] great news
11:03 [ Italian  ] [happy]
11:03 [ DenBolle ] at .05 I'll start
                   BU2NSNTP001/2 and BU1DSNTP001/2
11:04 [ Polish-2 ] FC switches haven't started yet.
11:04 [ DenBolle ] ok, when they have, ping me, then I'll boot the
                   FAS270's
11:04 [ Polish-2 ] ok
                   Still no progress.. usually, they are starting
                   quite fast.
11:06 [ DenBolle ] mhm.. what Can I check now?
11:07 [ Polish-2 ] Did you do the power-cycle of those switches?
                   I mean - were they powered off?
11:07 [ DenBolle ] they were
                   and disconnected power cables / reconnected
                   when I was done.
11:07 [ Polish-2 ] ok let's wait additional 1-2 mins
11:07 [ DenBolle ] ok
                   and if they aren't up.. that's were the terminal
                   connection kicks in?
11:09 [ Polish-2 ] I think that before that, we will do do the power
                   cycle once again.
11:09 [ DenBolle ] ok, I'll wait for your instrcutions
11:09 [ Polish-2 ] terminal connection was meant for BU2DSNTP001 & 002
                   - if they would be problems after changing IPs
11:10 [ DenBolle ] ah.. I understand
11:11 [ Polish-2 ] this is really wird
                   weird*
11:11 [ DenBolle ] eh....
                   don't go there..
11:11 [ Polish-2 ] Den, could you check ping to any other device from
                   affected subnet (192.46.157.*)?
11:11 [ DenBolle ] sure, let me reconnect to wired network, I'll ping
                   you when I'm in
11:12 [ Polish-2 ] ok
11:14 [ DenBolle ] ah.. not the cleverest move..
                   no dhcp server
11:14 [ DenBolle ] let me fetch it in a other way'
11:14 [ Polish-2 ] ok
11:16 [ Italian  ] even without dhcp
                   192.46.157.* switched should ping,
                   or am I wrong?
                   switches
11:17 [ Polish-2 ] yes, you're correct
                   I'm trying to investigate the problem starting from
                   their internal network.
11:17 [ DenBolle ] ok, isolated the problem..
                   pinging internally works..
11:18 [ Polish-2 ] to FC switches as well?
11:18 [ DenBolle ] pinging outside (eg. my own site) no go...
                   ips for those?
11:18 [ Polish-2 ] 192.46.157.133
                   f.e.
11:18 [ DenBolle ] jep, no problem...
11:19 [ Polish-2 ] ok, so we have some network problem..
11:19 [ DenBolle ] and default gateway also replies : .213
                   so let me rerun my cabling and I'll get
                   back in say 10 mins
11:19 [ Polish-2 ] ok, thx
11:30 [ DenBolle ] cables seems in order, the riverbed device seems to
                   have an issue. So I rebooted it, but it doesn't show
                   links...  what IP's did we use?  perhaps a msitake
                   and used a ip from the riverbed?
11:31 [ Polish-2 ] I've used following IPs for BU2DSNTP001 & 002:
                   192.46.157.166
                   192.46.157.167
                   However, they aren't even up.. So this is probably
                   not the issue.
11:31 [ DenBolle ] agree.
                   I can ping internally, no issues.
                   outside : doesn't work
                   how can we verify the correct function on the
                   riverbed?
11:33 [ Polish-2 ] Unfortunately, I am not a network specialist and do
                   not have any knowledge about that...
                   [Pg-1], [Ig-1] - any ideas?
11:33 [ Polish-1 ] No idea at all.
11:34 [ Italian  ] I'm looking for someone from network team ......
11:34 [ DenBolle ] ok
11:34 [ Polish-2 ] from Poznan, they are all offline.
11:35 [ DenBolle ] wait just a second
                   to get the cable truogh I had to reconnect one cable
                   on the back of the riverbed, let me fetch the cabling
                   diagram
11:37 [ Polish-2 ] ok
11:50 [ Italian  ] I don't understand
                   we don't touch any netw appliance
                   only FC switches ....
11:51 [ Polish-2 ] Den mentioned that he was doing reorganization
                   in LAN switches (and/or Riverbed device) -
                   this probably is causing problem
                   YES!!!
                   FC switch pingable!
11:51 [ Italian  ] it works
                   Great job Den
                   !!!!
11:52 [ DenBolle ] pfoe..
                   my mistake!
11:52 [ Polish-2 ] :-))
11:52 [ DenBolle ] ok, back to business.. : I'll power up the FAs270?
11:53 [ Polish-2 ] yes, please
11:53 [ DenBolle ] ok, just a sec..
                   [notes to self to stay the hell away of those
                   cables.. ]
                   done, bootup in progress
11:54 [ Polish-2 ] great, I'm monitoring the.. ping to them.
                   let's hope that here we won't have problems...
11:54 [ DenBolle ] don't worry.. I know what went wrong..
                   network 101
11:54 [ Polish-2 ] hell yeah!
                   BU2dsntp001 and BU1dsntp001
                   up
11:55 [ Italian  ] 2 good news in 2 mins .... my heart could stop
11:55 [ DenBolle ] har har..
                   we're a good team, it's dun to work like thois
                   this
                   dun = fun..
12:00 [ Polish-2 ] ok, the whole storage booted just fine..
                   now I'm going to make mess with it
12:00 [ Italian  ] ok
12:02 [ DenBolle ] so at this point, we about 2,5 hrs behind on
                   schedule..
12:02 [ Italian  ] yes
12:02 [ DenBolle ] sorry.
12:02 [ Italian  ] np
                   How long [Pg-2]?
12:02 [ Polish-2 ] i think max. 30 min
12:03 [ Italian  ] great
12:03 [ DenBolle ] So you're making up for my mistake, and then
                   we're back on schedule..
                   cool
12:03 [ Italian  ] don't worry [Pg-2] even 1 hour
12:03 [ DenBolle ] no, I'll stay onsite as long as is needed.
12:29 [ Polish-2 ] OK - seems to be done.
                   LUNs from BU2 storage are remapped to the new ESX
                   hosts with IDs starting from 10.
                   Zoning on the FC switches is also reconfigured.
12:31 [ DenBolle ] ok, so next steps are powering up ESX hosts?
12:31 [ Polish-2 ] [Pg-1] - I think that you can start ESX hosts
                   now. Please check the connection to the storage
                   before starting up the VMs.
12:31 [ Polish-1 ] OK. Please let me celebrate for a while this moment
                   when world's eyes are on me.
12:31 [ Polish-2 ] haha
12:32 [ DenBolle ] hahaha.  Big aplause for you !
12:34 [ Polish-2 ] I've made it in 30 mins - I can afford to eat
                   breakfast now.
12:34 [ Polish-1 ] I'll power them up one by one.
                   BU1sxve001 goes first
12:34 [ Polish-2 ] OK, I will be watching.. on the FC switches.
                   As per FC point of view - it's up. And each Netapp
                   can see it.
12:59 [ Polish-1 ] all 3 ESX host are up but they can't see new LUNs.
                   I performed rescan onstorage but it not helped.
                   I need to check this.
13:01 [ Polish-2 ] I can only say that from my side all seems to be
                   correct. Netapps see each ESX and those ESX are logged
                   in on Netapp adapters.
                   FYI: There should be 4 new LUNs visible,
                   with IDs 10,11,12,13.
13:03 [ DenBolle ] ok, np. take your time to find out whats going on.
14:05 [ Polish-1 ] OK. Now I can see all LUNs on all 3 ESX hosts.
                   @[Pg-2]: We need to verify paths now.
                   Please provide me paths for LUN S1V1 (108,5GB)
14:07 [ Polish-2 ] Sure thing.
                   You have to to tell me this LUN id.
                   Because we have now double pairs of S1V1,
                   S1V2, and so on...
14:08 [ Polish-1 ] LUNs from BU2echt are described as:
                   S1V1_BU2
                   S1V2_BU2 and so on
14:09 [ Polish-2 ] ok, so first the LUNs from BU1ST Netapps:
14:09 [ Polish-1 ] S1V1
14:10 [ Polish-2 ] S1V1 (880.1g, LUN id: 0)
                   S1V2 (805g, LUN id: 1)
                   S1V3 (563.1g, LUN id: 4)
                   Correct path for them is: 500a098195065a78
14:10 [ Polish-1 ] This should be set as prefferd right?
                   and active
14:11 [ Polish-2 ] S2V1 (955.1g, LUN id: 2)
                   S2V2 (730.1g, LUN id: 3)
                   S2V3 (563.1g, LUN id: 5)
                   Correct path for these is: 500a098185065a78
                   Correct. It should be considered as preferred
                   (primary) path.
                   BU2 LUNs:
                   S1V1 (840g, LUN id: 10)
                   S1V2 (840g, LUN id: 11)
                   Correct path: 500a098195364ca3
                   S2V1 (840g, LUN id: 12)
                   S2V2 (840g, LUN id: 13)
                   Correct path: 500a098185364ca3
                   Uff.. that's all.
14:14 [ Polish-1 ] ok please give me a few minutes to check all hosts
14:29 [ Polish-1 ] Ok. I verified everything and corrected paths.
                   Now I suggest to power on domain controller.
                   VM in BU1.
14:29 [ Italian  ] ok for me
14:30 [ Polish-1 ] then I'll add to inventorythose machines which were
                   migrated.
14:31 [ DenBolle ] ok, well, so far all goes well.
14:32 [ Polish-1 ] Den, an all BU1st machines the second DNS entry is
                   set to the BU2 DC. I suggest to change it to
                   STVSXDMC018 147.184.54.199 and the primary DNS leave as
                   is (BU1SXDMC002).
14:33 [ DenBolle ] ok, do so, do you want me to submit a ticet for it?
14:33 [ Polish-1 ] no.
14:36 [ Italian  ] [Pg-1], about BU2SDWN001 (FV) how long does it take
                   to add to inventory ?
14:36 [ Polish-1 ] it should be one click.
                   I'll do it now.
14:37 [ Italian  ] thks
14:40 [ Italian  ] I see it
14:43 [ Italian  ] please remember to change IP
14:50 [ Polish-1 ] BU2SDWN001 is working under new IP now.
                   Now I will add BU2SXFILE001 to the inventory
                   and change the IP.
                   Then I'll call DNS team to perform DNS changes. OK?
14:51 [ DenBolle ] ok
14:51 [ Italian  ] moment
                   DNS is already updated ?
14:51 [ Polish-1 ] no
14:51 [ Italian  ] ok
                   go ahead
                   and I told C. to wait
                   for tests
14:51 [ Polish-1 ] ok
15:23 [ Polish-1 ] I powered on BU2SXFILE001 and changed IP. Then I added
                   a new disk to BU1SXFILE001 and powered it up.
                   Now I catch up with network team for DNS changes.
15:23 [ Italian  ] ok
15:25 [ Italian  ] ANy idea about new ip propagation time ?
15:29 [ Polish-1 ] [Pg-2]osz Ostrowski is workin on it now. He does not
                   know how log it may take as he's not doing it usually.
                   Only today was caought by me.
                   cought*
15:31 [ Italian  ] because I suppose you already started DNS change
                   activity, from my experience it could take also
                   24 hours to propagate
                   I don't know if is possible to force it
15:31 [ Polish-1 ] I provided him numbers of those 2 requests.
15:33 [ DenBolle ] and a forced propagation isn't a option?
15:33 [ Italian  ] Den What's your idea about FV tests, in the worst case
                   can we done them tomorrow ?
15:35 [ DenBolle ] I think we can, let me verifyt with C. on this one.
                   Its's possible to do it from home for him
15:39 [ Polish-1 ] So far from Poznan wi have this:
                   C:\>ping BU2sdwn001
                   Pinging BU2sdwn001.lan01.com [192.46.157.162] with
                   32 bytes of data:
                   Reply from 192.46.157.162: bytes=32 time=63ms TTL=119
                   Reply from 192.46.157.162: bytes=32 time=63ms TTL=119
                   Reply from 192.46.157.162: bytes=32 time=64ms TTL=119
                   Reply from 192.46.157.162: bytes=32 time=63ms TTL=119
                   Ping statistics for 192.46.157.162:
                   Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
                   Approximate round trip times in milli-seconds:
                   Minimum = 63ms, Maximum = 64ms, Average = 63ms
15:41 [ Italian  ] the same from Italy ....
15:42 [ DenBolle ] I've talked with C., tomorrow testing is possible,
                   but before 1000 hrs, he has other apointments
15:42 [ Italian  ] I think he caould start test now
                   DNS looks updated
15:43 [ DenBolle ] ok, I'll inform him.
15:43 [ Italian  ] thks
15:43 [ Polish-1 ] I'm gonna start next VMS
                   ok?
15:43 [ Italian  ] yes please
15:44 [ DenBolle ] C. wil start testing right away.
16:00 [ Polish-2 ] [Pg-1], are you starting the other VMs as well?
                   I need to check the BU1SMLXV0001 server.
16:01 [ DenBolle ] C. reports an issue : mq service isn't running,
                   I'm working on that.
16:02 [ Italian  ] ok in case we can open a ticket to GSA-WW-MIDDLEWARE
16:07 [ DenBolle ] ok, problem killed.. C. tests further.
16:07 [ Italian  ] good job "terminator"
16:16 [ Polish-1 ] I'm starting VMs one by one.
16:16 [ DenBolle ] ok, al seems in good working order ?
16:17 [ Polish-1 ] so far so good
16:17 [ Italian  ] [happy]
16:17 [ DenBolle ] you optimistic guy
16:32 [ Polish-1 ] BU1SMLXV0001 is up now
16:32 [ Polish-2 ] thanks, I've just finished checking it
                   all seems to be fine, all filesystems mouted,
                   oracle processes started
                   By the way, after the action, we need to raise a high
                   priority ticket to DBA team with request to check
                   databases on affected servers. I discussed this
                   with [naam] from DBA, and he asked me to do so.
16:34 [ Italian  ] good idea
16:35 [ Polish-2 ] How about other VMs? Did they started properly?
                   No issues yet?
16:35 [ DenBolle ] C. reports no issues on FV and MQ series, [Ig-1],
                   he'll sent you a test report.
16:35 [ Italian  ] fantastic
16:36 [ Polish-1 ] I'm checking them one by one.
                   We'll need to ask [Pg-3] to check if backup works
                   fine now.
16:36 [ Italian  ] [Pg-1] did you start data copy from BU2SXFILE001 ?
16:37 [ DenBolle ] should his wife be done with hunting now?
16:37 [ Polish-2 ] We'll see.. ;
16:37 [ Polish-1 ] not yet. I need to prepare jobs for SecureCopy.
                   Fortunatelly I installed it yesterday.
                   My wife?
16:38 [ Italian  ] [Pg-3]'s wife
16:38 [ DenBolle ] No, [Pg-3]'s wife
16:38 [ Polish-1 ] Also Lotus team could be informed to check Domino
                   server.
16:38 [ Italian  ] Den
                   Can I promote you leader?
16:39 [ DenBolle ] Sure, does it pay more?
16:39 [ Italian  ] 65% more
16:39 [ DenBolle ] Give it to me!
16:39 [ Italian  ] or 67% don't remember
                   I really have to go ........
                   but I'm available on call (if necessary , but i
                   don't think so, your are in good hands )
16:41 [ DenBolle ] No problem.  I have your mobile.
                   [realises I'm the guy for the coffee now.. ]
16:42 [ Italian  ] ehehehe
                   I'm happy that FV tests were ok
16:43 [ DenBolle ] me to, specially beacuse that's something automated
                   outside us, and BU2 depend on it..
16:43 [ Italian  ] yes
                   Thanks a lot [Pg-1] and [Pg-2], I didn't do nothing
                   "today" but  I wish you feel my support remotly
16:45 [ Polish-1 ] Thank you too.
16:45 [ DenBolle ] [Ig-1], thx sofar for your help and coordination.
                   When we're done I'll sent you a txt message to inform
                   you.
16:45 [ Polish-2 ] Sure, [Ig-1]. It was good to have you here. Thank you!
16:45 [ Italian  ] too good guys
                   Thanks Den a call or a message will be really appriaciate
16:46 [ DenBolle ] I'll do, count on it.
16:46 [ Italian  ] grazie
                   bye bye
16:46 [ Polish-2 ] bye
16:46 [ DenBolle ] bye
16:47 [ Polish-2 ] [Pg-1], are all VMs up by now?
                   If yes, I would raise ticket to DBA team.
16:48 [ DenBolle ] ok, just a short next-steps overvieuw : after al VMS are
                   up, db checks,  inform notes team, and run a test backup.
                   Anything else?
16:49 [ Polish-2 ] from my side, nothing to add
16:49 [ DenBolle ] ok, [Pg-1]?
16:51 [ Polish-1 ] and copy BU2SXFILE001 to BU1SXFILE001
                   this can last few hours
16:52 [ DenBolle ] ok, and all would take about say 2-3 hrs from now?
                   (I've got to inform security here that it's going to
                   be a little later than 1800 hrs)
16:52 [ Polish-1 ] no idea. It's 500GB to copy.
16:53 [ DenBolle ] thats (500 diveded by 16M  memory sticks)
                   say quite some later than 1800 hrs. I'll keep it
                   at 2000 hrs ok?
16:54 [ Italian  ] [Pg-1] could you be there unil the end of the copy ?
16:54 [ Polish-1 ] ok. If it's not finished until that time you can go
                   home and I'll let you know via sms if all data is copied.
                   I'll be here until the end of the copy.
16:55 [ DenBolle ] No, I can stay as long as needed, but I would like to
                   inform the security guy before he's  disapointed.
16:55 [ Italian  ] because last step is to stop BU2SXFILE001
16:55 [ DenBolle ] /scared because [Ig-1]'s ghost is here..
16:55 [ Italian  ] eheheheh
                   I'm waiting for my wife
16:56 [ Polish-1 ] I know but I have no info from K. if he checked if
                   we have access to every file. If I need correct
                   something manually it may last longer.
16:56 [ Italian  ] she has to drive me home
16:57 [ DenBolle ] ok, for that lets see what happens when the copy
                   part is starting.  And have a chat about that then.
                   First lets see that we have the DB's checked and
                   Domino team informed.
16:57 [ Polish-1 ] one moment. I have an urgent reboot in Canada.
16:58 [ Polish-2 ] Den, I am wondering whether you could raise a ticket
                   to DBA team when all VMs are up?
16:58 [ DenBolle ] sure, np,  give me the RA and I'll submit one
16:58 [ Polish-1 ] in a minute or two
16:59 [ Italian  ] TECH-CENTRE-DATABASE-SUPPORT
16:59 [ Polish-1 ] right
16:59 [ Polish-2 ] if this is a minute or two, I think that I will
                   manage to do that by myself.
17:00 [ Polish-1 ] ok
                   all VMs are up now
17:02 [ DenBolle ] allright, who's going to submit the ticket?
                   to prefent a double entry.
17:02 [ Polish-2 ] I am going to submit the ticket to DBA team.
17:02 [ DenBolle ] ok
17:05 [ Italian  ] good luck and hope you finish asap
17:05 [ DenBolle ] Thx, and snjeoy your evening!
                   snjeoy = enjoy
17:06 [ Italian  ] :-)
17:06 [ Polish-2 ] Ticket to DBA: UKIM20002972635
17:07 [ DenBolle ] ok, let see if they pcik it up quickly
17:08 [ Polish-1 ] I talked to [Pg-3]. He'll check backups about * PM
                   today and he'll send us an email with update.
17:08 [ DenBolle ] ok, so that's covered. What about Domino team?
17:08 [ Polish-1 ] I'm creating a ticket now
17:09 [ DenBolle ] Great, this is going well guys, thx a lot for
                   your effort on this one.
17:11 [ Polish-1 ] UKIM20002972653
17:11 [ DenBolle ] ok, thx
17:32 [ Polish-1 ] OK I started copying data from BU2SXFILE001
                   to BU1SXFILE001
                   1 min 200 MB
                   2 min 410 MB
                   3.73 MB/s
17:34 [ DenBolle ] mhm..
                   if I calculate right 200 mb / min  = 1Gb/ 5 min
                   --> 12Gb / hrs..
17:35 [ Polish-1 ] looks like this
17:35 [ DenBolle ] ie.. whole copy will take approx 40 hrs..
                   and not ready monday morning..
17:36 [ Polish-1 ] 5 min and 1.16GB
                   now performance is about 4 MB/s
17:38 [ DenBolle ] A. (messaging team) tells that domino is working fine..
17:38 [ Polish-1 ] ok
17:39 [ Polish-2 ] No one from DBA haven't picked up the ticket yet.
                   I'll try to contact them now.
17:40 [ DenBolle ] ok.
                   [Pg-1] : lets wait a litte to see if it speeds up,
                   but 4mb/sec is to slow to get ready
17:41 [ Polish-1 ] ok now I have 3.61MB /s
                   and 2.17GB copied
                   we can leave BU2SXFILE001 working and correct DNS entry.
                   The migration can be done later if the server is
                   accessable for users after migration. What you think?
17:43 [ DenBolle ] like having a shadowcopy in place and then just a smal
                   migration?
17:44 [ Polish-1 ] migrate files during the week and during the weekend
                   copy only files which changed.
17:44 [ DenBolle ] yeah, that's the best approach I think,
                   sync the files, and switch in the next weekend.
                   how does that goes with the change we requested?
                   can we fit it in?
17:45 [ Polish-1 ] We can do an exception. it's always possible.
                   3.34MB/s now
17:46 [ DenBolle ] ok, so the chance that it will speed up significant
                   is low,  let's go for the 'hidden sync' and migrate
                   next weekend then.
                   in this speed we're doomed
17:47 [ Polish-1 ] ok so I need to rise a ticket for a DNS entry change.
                   Then we will check if it works. I'll keep copying task
                   running during the weekend to have the main part
                   already copied.
                   A new change request will be required but I'll
                   prepare it.
17:48 [ DenBolle ] ok, and the DNS is only for the BU2SXFILE001 right?
17:48 [ Polish-1 ] right
                   this will point now to the new address
                   192.46.157.168 right/
17:49 [ DenBolle ] let me fetch that one, it...
17:56 [ DenBolle ] just a sec again.. I've got .164 ..
                   let me fect the documentation to [Ig-1]
17:56 [ Polish-1 ] ok
17:57 [ DenBolle ] The last IP address 192.46.157.168 has been required as
                   a temporary address in case of any issue on migrated VMs.
                   (Temporary emergency Vms component)
17:58 [ Polish-1 ] From changerequest:
                   3. 	15 min. 	Modify the IP address and default
                                        gateway of BU2SDWN001 as per
                                        configuration items changes.
                                        Change the IP of BU2SXFILE001 to
                                        192.46.157.168 and apply BU1 network
                                        settings.
                                        To provide screen shot of new IP
                                        config.
17:58 [ DenBolle ] yeah, it's ok.  use it
17:59 [ Polish-1 ] ok so I'm rising a ticket for DNS change.
17:59 [ DenBolle ] wait..
17:59 [ Polish-1 ] ok
18:03 [ DenBolle ] mhm.. this one isn't clear. the doc from [Ig-1]
                   (23/12/2010) states .164 as FS ip.
                   and the 168 as emergency...
                   Go ahead: use .168: this is an emergency.
18:03 [ Polish-1 ] ok
18:04 [ DenBolle ] I'll update [Ig-1] on this per email.
18:08 [ Polish-1 ] 7002306
18:10 [ DenBolle ] ?
18:10 [ Polish-1 ] request number
                   to network for DNS change
18:10 [ DenBolle ] ah ok
                   [Pg-2], any luck on the DBA team?
18:13 [ Polish-2 ] not yet, I wrote a SMS to person on-duty,
                   without any luck yet
                   I'll wait additional 5-10 mins and I will call them
18:13 [ DenBolle ] ok, in that case I'll hand out the coffee
18:13 [ Polish-2 ] ok
18:14 [ DenBolle ]  for [Pg-1], and  this one for you [Pg-2]osz
18:14 [ Polish-2 ] thx a lot
18:14 [ DenBolle ] mail from [Pg-3] :
                   Hello All,
                   Just FYI. The backup service is OK after the migration.
                   I have just started all quarter jobs except BU1SXFILE001
                   that is during data copy. So far so good for the backup
                   side.

                   Regards,

                   [Pg-3]
18:14 [ Polish-2 ] good
18:26 [ Polish-2 ] weird.. the guy who is supposed to be on-duty from DBA
                   team is not answering neither the SMS and the phone calls..
18:26 [ DenBolle ] and a 'second' one isn't available?
18:27 [ Polish-2 ] I will try to call the 'second' one
18:31 [ Polish-1 ] 21,72Gb in 1 hour
18:31 [ DenBolle ] hey..
18:31 [ Polish-2 ] so it seems that speeded up?
18:32 [ DenBolle ] thats twice what we expected..
18:32 [ Polish-2 ] 25h for 500GB
18:32 [ DenBolle ] ie.. tomorrow al is done?
18:32 [ Polish-1 ] now 6.17 but we don't know how many small files are befor
                   us
18:32 [ DenBolle ] if no files have issues that is.
18:32 [ Polish-1 ] no it's about 26 hours
                   it's still risky
18:33 [ DenBolle ] lets keep the BU2SXFILE001 alive.. to be save.
18:33 [ Polish-1 ] yes
18:34 [ DenBolle ] and stick to the plan. A dns change is easy.
                   If all is save we can do it perhaps earlier then next
                   weekend.
18:36 [ Polish-1 ] sure
18:49 [ Polish-2 ] I've contacted [naam] from DBA at last..
                   He has muted his phone and left it in thie jacket's
                   pocket.. and didn't hear that it was ringing.
                   He will be on-line in a minute and check DBs.
18:50 [ DenBolle ] pfoe..  I assume we can recognize [naam] at his black
                   eyes monday?
19:00 [ Polish-1 ] C:\>ping BU2sxfile001
                   Pinging BU2sxfile001.lan01.com [192.46.157.168] with
                   32 bytes of data:
                   Reply from 192.46.157.168: bytes=32 time=95ms TTL=119
                   Reply from 192.46.157.168: bytes=32 time=92ms TTL=119
                   Reply from 192.46.157.168: bytes=32 time=77ms TTL=119
                   Reply from 192.46.157.168: bytes=32 time=75ms TTL=120
                   Ping statistics for 192.46.157.168:
                   Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
                   Approximate round trip times in milli-seconds:
                   Minimum = 75ms, Maximum = 95ms, Average = 84ms
                   C:\>
19:01 [ DenBolle ] great thx. Solved this one.  After DBA verification we're
                   done for today ?
19:02 [ Polish-1 ] Please check if you can map shares without problems.
                   If yes, we can say that's all for you today.
19:07 [ DenBolle ] BU2sxfile001 resolves to 192.46.157.160 --> BU1sxfile001
                   when was the dns set?
19:08 [ Polish-1 ] I talked to this guy. This will be set in a minute.
                   let's wait for a while. They also have a tough day.
19:08 [ DenBolle ] sure, np..
19:23 [ DenBolle ] on the BU1SXFILE001 the BU2SXFILe001 resolves to
                   192.46.157.168, but on the network it doesn't,
                   I'll wait some more.
19:24 [ Polish-1 ] [Pg-2] is working on it now.
19:25 [ DenBolle ] sure, no hurries on this one
19:25 [ Polish-1 ] He had a call to Riga
                   network outage till monday
19:25 [ DenBolle ] in Riga?
19:25 [ Polish-1 ] yes. Packets get lost from time to time.
                   my remedy queue is blue
19:26 [ DenBolle ] because you're working all day allready ?
19:26 [ Polish-1 ] yes
19:27 [ DenBolle ] grin grin..
                   just keep up a little.. we're almost done for today
19:27 [ Polish-1 ] you are almost done
                   I'm till midnight
19:28 [ DenBolle ] wow.. thats 18 hrs ?
19:28 [ Polish-1 ] 17
19:28 [ DenBolle ] what happend?
19:28 [ Polish-1 ] when we are on remedy during weekend we have to proceed
                   all weekend's tasks that's all.
                   my weekend shift is between 12PM - 24
                   because today we had BU2migration I had to start earlier.
                   There was an option
                   KL...
19:29 [ DenBolle ] in that case : my compliments ! I appriciate it that
                   you supported us outside your regular shift times.
19:30 [ Polish-1 ] It's always better to have work in one heands that
                   pass it on and on from person to person.
19:31 [ DenBolle ] I know what you mean.. losing that one detail and then
                   no one knows what happened.
19:31 [ Polish-1 ] eg.
19:39 [ DenBolle ] allright : in the network the BU2sxfile001 in the
                   bu1 subnet works at 192.46.157.168 and mapping is
                   possible. I'm now going to the bu2 network to verify a login
19:40 [ Polish-2 ] A. just sent the info about DBs: all is fine.
                   In that case, I think that my work for today is done.
19:42 [ Polish-1 ] ok. thank you for your assistance. Have a nice sleep.
19:44 [ Polish-2 ] Thank you for your support. I wish you [Pg-1] a calm
                   on-duty time. I hope that you'll be able to relax tomorrow.
19:44 [ Polish-1 ] thanks, I hope so.
20:19 [ DenBolle ] ok, the followme system has issues.  I've shut down
                   all printers and will restart them one by one
20:19 [ Polish-1 ] ok and now?
20:20 [ DenBolle ] I wait a few mins to calm the server.
                   It can't handle the amount of config requests,
                   and  then I'll boot up the printers again
20:20 [ Polish-1 ] ok
20:22 [ Polish-2 ] OK, I'm logging off now.
                   @Den: Thanks for cooperation. I hope that all remaining
                   issues will be resolved soon.
                   Thank you guys, have a nice Sunday!
                   bye
20:22 [ Polish-1 ] bye! Thank you!
20:43 [ DenBolle ] [Pg-1], I need a reboot on the BU1SPWN001 ...
                   the priner server isn't acting wel
20:46 [ Polish-1 ] ok I'll do it right now.
20:46 [ DenBolle ] great
20:50 [ DenBolle ] and?
20:50 [ Polish-1 ] server is booting up now
20:50 [ DenBolle ] top, thx
20:53 [ DenBolle ] ah.. it's also the DHCP server...
20:53 [ Polish-1 ] yes
20:53 [ DenBolle ] lost my connection
                   I'm on wifi, is it back on ?
20:55 [ Polish-1 ] yes the server is back on
20:56 [ DenBolle ] ok, let me see that I'm getting the printer back on
20:56 [ Polish-1 ] ok
21:48 [ DenBolle ] ok, I'm back.. All printers tested and are working.
21:48 [ Polish-1 ] ok good.
                   what next?
21:48 [ DenBolle ] beer?
21:48 [ Polish-1 ] I'm at work
21:49 [ DenBolle ] I've got to drive the motor for 1,5 hrs...
                   then beer
                   We're done I think. Db's checked, mappings checked,
                   printers checked,  all works, only thing now is
                   copying the data
21:50 [ Polish-1 ] 83Gb so far and 5.52 MB/s
21:50 [ DenBolle ] so we're done in the majority of task
21:50 [ Polish-1 ] I think yes. I'm happy it worked.
21:54 [ DenBolle ] me to.
                   Thx a lot for all. I'll inform the users.
                   Next weekend the last step ?
21:55 [ Polish-1 ] I think yes. I'll contact you in the week.
                   Thanks you. Have a nice end of weekend.
21:55 [ DenBolle ] ok, excellent.  Thanks for your help and
                   patience.  Hear you next weekend.

Eén commentaar op: An die Arbeit.

  • Daar zaten wel een paar momenten tussen waar je het zweet van in de handen krijgt 🙂
    Vooral vm’s met multipathing, dan moet je de kop er wel even rustig bij houden.

    Onder het mom van : als je het niet wil dan wel en als je het wel wil dan niet…
    (Heuh? : welnu komtie:)
    We hebben hier een 4-tal oude ml350 (proliants) staan, met allerlei raid combinaties, 78 en 36 GB verdeeld over verschillende raid sets op verschillende servers. Die disken moeten leeg, dus ik ‘schud’ ze even. (Alles eruit, door elkaar husselen en hop, random terug duwen en laten initialiseren…) Hoe het kan weet ik niet, maar toch mooi dat ik stomverbaasd 3 minuten later naar een windows 2003 server inlogscherm stond te kijken.
    Wel een Blue screen natuurlijk na het inloggen, maar toch! (zie je wel,ik doe ook spannende dingen 🙂 )

Commenteer