Ondanks de voorbereidingen liep het allemaal net wat anders als gepland. Zo waren er eigenlijk maar drie problemen die impact hadden.
We startten wat later dan gepland door een uitgelopen backup. Beter safe then sorry. Tijdens het omprikken van de kabels bleek dat de kast waarin de apparatuur stond precies met een poot op de vloertegel stond die ik er uit moest werken. Om dat goed voor elkaar te krijgen moest er om de kast heen extra tegels worden verwijderd zodat de eigenlijke tegel die ik er uit moest halen vrij kwam te liggen. Op zich niet erg maar ..
Door dat gebouw en geheister zat een kabel me hinderlijk in de weg. Dat ding hing niet weggewerkt vervelend te wezen. Omdat nu toch alles plat lag mooi een moment om de kabel door een ander te vervangen en weg te werken. Kabel opgesnort, ingeprikt, mooi weggewerkt en verder gaan.
En toen. tja.. toen bleek ineens dat dat rare roze kleurtje van die kabel niet voor niets was. Het duurde even voor ik me realiseerde dat die kabel een crosslink was. De oude kabel weer op de plek en alles werkte weer. (va. 11:51 uur)
Althans.. Op de een of ander manier ging het kopiëren van de data niet zo snel als we dachten. Waar een beetje snelheid nodig was bleek de copy slag traag te zijn, en waar we op een paar uur data pompen hadden gerekend werd het al snel een dikke 40 uur (is dus ergens maandag ochtend). Daarom op een noodplan overgestapt, en maken we de laatste stappen volgende week af.
En op het laatst (ik ben bijna aan bewaker 3 toe, ze werden telkens afgewisseld) bleek dat het hele printer systeem plat lag. De printerserver had een schop gekregen en aangezien onze printers hun configuraties uit de printerserver trekken ging dat fout. Na een kort telefoon gesprek met een van de locale jongens die dit soort dingen vaker bij de veter had gehad besluiten we alsnog om de printeromgeving een keer plat te gooien en dan op te starten. Langzaam aan (zeg een 5-10 min/ printer, we hebben er bijna 20 staan) kregen we ze stuk voor stuk weer aan de praat en aan het printen.
Dus de ingeschatte eindtijd van 1800 hrs. werd wat overtrokken…
Voor de geïnteresseerde, hieronder staat de hele listing (geanonimiseerd en ontdaan van specifieke tech info) van een hele dag waanzinnig lekker samenwerken op een site, met 4 landen (Nederland, Polen, België, Italië), serieus en minder serieus gegein.
een lange dag, maar met plezier gedaan. En… als het goed is merken de gebruikers er niets van.
De spelers hieronder: Italian (in de gesprekken Ig-1) : een van de project managers werkend vanuit Itailië, Polish-1 , Polish-2 en Polish-3 (respectievelijk Pg-1, Pg-2 en Pg-3 in de gesprekken): de poolse jongens in het poolse datacenter, werken met de VM ware machines, unix storage en Backup omgeving. C, K, A zijn testers/adviseurs die op afroep voor ons klaar stonden.
8:26 [ Italian ] Morning Den 8:26 [ Polish-1 ] Hi Den! 8:26 [ Polish-2 ] Hi Den 8:27 [ DenBolle ] Goodmorning everyone! [Ig-1] is generous : he treats us all in coffee! 8:27 [ Italian ] yes this for [Pg-2] for [Pg-1] 8:28 [ Polish-1 ] thx 8:28 [ Italian ] for Den that has to wake up 8:28 [ DenBolle ] Gracie! 8:28 [ Italian ] yw 8:29 [ Polish-2 ] Thank you! 9:00 [ Italian ] Hi [Pg-3] 9:00 [ Polish-3 ] Hello 9:00 [ Italian ] please advise us when done 9:00 [ Polish-3 ] around 10 minutes left I think 9:01 [ DenBolle ] ah ok 9:01 [ Polish-3 ] blowing from home will not speed it up, sorry 9:02 [ Polish-2 ] ;-) 9:02 [ Italian ] 9:02 9:02 [ DenBolle ] I'm in the office, I could spin up the tape? 9:02 [ Italian ] good idea 9:02 [ DenBolle ] use a microwave to compress the data on the tape? 9:03 [ Polish-3 ] Den, you are the biggest of us /except [Pg-1]/ could you tell few words to the Adic Lib to write a little bit faster ? 9:04 [ DenBolle ] sure.. [walks into server room with a stick and blowtorch] 9:04 [ Polish-3 ] ;-) 9:04 [ Italian ] roftl better ..... grin grin 9:06 [ DenBolle ] ok, I've discussed it with the lib, he's appologizing and will work harder. btw I never realized that they actually can scream if you put them on fire. 9:06 [ Polish-3 ] buahahahaha 9:08 [ Polish-3 ] I though Adic is a woman..strange world.. 9:10 [ DenBolle ] It screamed like a woman though.. 9:14 [ Polish-3 ] btw started to write system state... she is coming or he... 9:15 [ DenBolle ] she is coming.. this story get's a nice twist. Go on [Pg-3] 9:16 [ Polish-3 ] heheh who will make some pictures - ->Den? It is done gents! 9:16 [ DenBolle ] and put them on a site " www.hornytapelibs.com" ? ok, let go on then 9:16 [ Polish-3 ] yep 9:17 [ Polish-2 ] it looks like I've missed some nice stories here ok, I'm proceeding with stopping the BU1SMLXV0001 9:18 [ Polish-3 ] well..it will not be published on our webside 9:18 [ Polish-1 ] Can I start stopping VMs now? 9:18 [ Polish-3 ] logged off 9:19 [ DenBolle ] ok, I'm over to wireless to be safe.. just a se 9:19 [ DenBolle ] and back 9:20 [ Polish-3 ] [Pg-1] could you call me when the servers will be back - I will start backups - I mean afternoon 9:20 [ Polish-1 ] Sure. 9:20 [ Polish-3 ] 504 110 321 cool, 9:31 [ DenBolle ] is this working 9:31 [ Polish-2 ] yep, now it's working 9:31 [ DenBolle ] mhm. glitch.. what did I miss? 9:31 [ Italian ] ;-) 9:31 [ Polish-1 ] I know. Some functional accounts logged on the console. I need to power them down manually 9:32 [ DenBolle ] last remark i've seen is froim [Pg-3] : 9:32 [ Polish-3 ] 504 110 321 cool, 9:35 [ Polish-2 ] I can't paste the OCS chat, because it says that it is too large.. So to summarize it quickly: [Pg-3] logged off, because his wife was chasing him. I've stopped the BU1SMLXV0001 and [Pg-1] is stopping now other VMs and ESX hosts. 9:36 [ Italian ] All Vms stopped ...great job [Pg-1] 9:36 [ DenBolle ] no problem.. So no strange things yet and all goes as planned. Besides the huntdown of [Pg-3]'s wife. 9:36 [ Italian ] ;-) 9:36 [ Polish-2 ] Maybe she heard the library's screams... 9:37 [ DenBolle ] tsk.. women united... If you hurt one ... 9:37 [ Polish-2 ] ;-)) 9:38 [ Italian ] when finished I'll copy this chat and send to Manager1, Manager2 and Manager3 with title "drunk men at work" 9:38 [ DenBolle ] grin grin.. and we never are allowed to work on saterday.. 9:44 [ Polish-1 ] @all: All VMs are powered down now. All ESX hosts are shutted down and powered off now. 9:44 [ DenBolle ] ok, thx 9:44 [ Italian ] Den ..... go there and do your best 9:45 [ Polish-2 ] no no now it's my part please hold on 9:45 [ DenBolle ] [stopping] 9:45 [ Italian ] sorrrrrrryyyyyy 9:45 [ Polish-2 ] Den, I'll let you know It will take approximately 15 mins 9:45 [ DenBolle ] I'll be waiting for you [not to self : this chat's getting stranger by the sentence] 10:01 [ DenBolle ] I've got screaming equipment 10:01 [ Polish-2 ] OK, all FC switches and FAS270`s are going down or are already down. This is the reason of those screams. 10:02 [ DenBolle ] ok, let me check in the room when they actually die.. 10:02 [ Polish-2 ] Let's wait now 2-3 min, just to be sure that all went down. 10:03 [ DenBolle ] sure 10:03 [ Polish-2 ] After this 2-3 mins, I'd like to ask you, Den, to power off the following devices: BU2DSBS001 BU2DSBS002 BU1DSBS001 BU1DSBS002 BU2DSNTP001 BU2DSNTP002 BU1DSNTP001 BU1DSNTP002 so - all 4 FAS270's and all 4 FC switches. 10:04 [ DenBolle ] I'll do. I'll start at 10.05 10:05 [ Polish-2 ] ok 10:08 [ DenBolle ] ok, all shelfs are down (no more hd activity) : I'll go shutdown the devices. Do you want me to reconnect the fibers straight away? 10:09 [ Polish-2 ] Did you power off the additional shelves as well? 10:09 [ DenBolle ] no, thats going to mbe done now 10:10 [ Polish-2 ] To be honest - it's probably not needed. 10:10 [ DenBolle ] if you feel safer when I do: no prblem. 10:10 [ Polish-2 ] OK - If you want to do so, we can do it. Just to remember: later, when we will be powering it on, we need to start them in reverse order. So first of all: the additional disk shelves, after that 2-3 min break to let the disks spin up and afterwards the FAS270 controllerrs 10:12 [ DenBolle ] ok, I'm in, back in 10 -15 mins 10:12 [ Polish-2 ] one additional thing 10:12 [ DenBolle ] yes 10:12 [ Polish-2 ] I've set new IPs for BU2DSNTP001 & BU2DSNTP002. So please reconnect also their ethernet cable to the new subnet. ..and you have to reconnect the FC cable as well as we discussed on the diagram. 10:13 [ DenBolle ] I'll do, stay tuned 10:13 [ Polish-2 ] th thx 10:40 [ DenBolle ] ok, almost done, just one issue. the 4 utp cables for the 270's : the core switch is full, give me 20 mins to isolate cables we can disconnect 10:41 [ Polish-2 ] 4 utp cables? 10:42 [ DenBolle ] yes the ethernet cables 10:42 [ Polish-2 ] to the BU2DSNTP001 & BU2DSNTP002 4 cables are connected ? 10:42 [ DenBolle ] yes 10:42 [ Polish-2 ] they have dual ethernet ports, however only one port (e0a) is used 10:43 [ DenBolle ] ok, in that case I'm still one port short : got one emtpy one.. 10:43 [ Polish-2 ] ok, I see ok, we will wait when you finish you can power on following devices: BU1DSBS001 BU1DSBS002 BU2DSNTP001 BU2DSNTP002 BU1DSNTP001 BU1DSNTP002 and please do not power on ESX servers 10:44 [ DenBolle ] No, I;ll do. ok, [Ig-1], in that case could you serve all once more a cup of coffe in the mean time? 10:46 [ Italian ] here they are ...... I'm boiling water ......... and ..... that's it 11:03 [ DenBolle ] ok, done with the cables, powered up shelfs (not FAS270's), and BU1DSB001 and BU1DSB002 are starting. When these are finshed I'll boot BU2NSNTP001/2 and BU1DSNTP001/2 11:03 [ Polish-2 ] great news 11:03 [ Italian ] [happy] 11:03 [ DenBolle ] at .05 I'll start BU2NSNTP001/2 and BU1DSNTP001/2 11:04 [ Polish-2 ] FC switches haven't started yet. 11:04 [ DenBolle ] ok, when they have, ping me, then I'll boot the FAS270's 11:04 [ Polish-2 ] ok Still no progress.. usually, they are starting quite fast. 11:06 [ DenBolle ] mhm.. what Can I check now? 11:07 [ Polish-2 ] Did you do the power-cycle of those switches? I mean - were they powered off? 11:07 [ DenBolle ] they were and disconnected power cables / reconnected when I was done. 11:07 [ Polish-2 ] ok let's wait additional 1-2 mins 11:07 [ DenBolle ] ok and if they aren't up.. that's were the terminal connection kicks in? 11:09 [ Polish-2 ] I think that before that, we will do do the power cycle once again. 11:09 [ DenBolle ] ok, I'll wait for your instrcutions 11:09 [ Polish-2 ] terminal connection was meant for BU2DSNTP001 & 002 - if they would be problems after changing IPs 11:10 [ DenBolle ] ah.. I understand 11:11 [ Polish-2 ] this is really wird weird* 11:11 [ DenBolle ] eh.... don't go there.. 11:11 [ Polish-2 ] Den, could you check ping to any other device from affected subnet (192.46.157.*)? 11:11 [ DenBolle ] sure, let me reconnect to wired network, I'll ping you when I'm in 11:12 [ Polish-2 ] ok 11:14 [ DenBolle ] ah.. not the cleverest move.. no dhcp server 11:14 [ DenBolle ] let me fetch it in a other way' 11:14 [ Polish-2 ] ok 11:16 [ Italian ] even without dhcp 192.46.157.* switched should ping, or am I wrong? switches 11:17 [ Polish-2 ] yes, you're correct I'm trying to investigate the problem starting from their internal network. 11:17 [ DenBolle ] ok, isolated the problem.. pinging internally works.. 11:18 [ Polish-2 ] to FC switches as well? 11:18 [ DenBolle ] pinging outside (eg. my own site) no go... ips for those? 11:18 [ Polish-2 ] 184.108.40.206 f.e. 11:18 [ DenBolle ] jep, no problem... 11:19 [ Polish-2 ] ok, so we have some network problem.. 11:19 [ DenBolle ] and default gateway also replies : .213 so let me rerun my cabling and I'll get back in say 10 mins 11:19 [ Polish-2 ] ok, thx 11:30 [ DenBolle ] cables seems in order, the riverbed device seems to have an issue. So I rebooted it, but it doesn't show links... what IP's did we use? perhaps a msitake and used a ip from the riverbed? 11:31 [ Polish-2 ] I've used following IPs for BU2DSNTP001 & 002: 220.127.116.11 18.104.22.168 However, they aren't even up.. So this is probably not the issue. 11:31 [ DenBolle ] agree. I can ping internally, no issues. outside : doesn't work how can we verify the correct function on the riverbed? 11:33 [ Polish-2 ] Unfortunately, I am not a network specialist and do not have any knowledge about that... [Pg-1], [Ig-1] - any ideas? 11:33 [ Polish-1 ] No idea at all. 11:34 [ Italian ] I'm looking for someone from network team ...... 11:34 [ DenBolle ] ok 11:34 [ Polish-2 ] from Poznan, they are all offline. 11:35 [ DenBolle ] wait just a second to get the cable truogh I had to reconnect one cable on the back of the riverbed, let me fetch the cabling diagram 11:37 [ Polish-2 ] ok 11:50 [ Italian ] I don't understand we don't touch any netw appliance only FC switches .... 11:51 [ Polish-2 ] Den mentioned that he was doing reorganization in LAN switches (and/or Riverbed device) - this probably is causing problem YES!!! FC switch pingable! 11:51 [ Italian ] it works Great job Den !!!! 11:52 [ DenBolle ] pfoe.. my mistake! 11:52 [ Polish-2 ] :-)) 11:52 [ DenBolle ] ok, back to business.. : I'll power up the FAs270? 11:53 [ Polish-2 ] yes, please 11:53 [ DenBolle ] ok, just a sec.. [notes to self to stay the hell away of those cables.. ] done, bootup in progress 11:54 [ Polish-2 ] great, I'm monitoring the.. ping to them. let's hope that here we won't have problems... 11:54 [ DenBolle ] don't worry.. I know what went wrong.. network 101 11:54 [ Polish-2 ] hell yeah! BU2dsntp001 and BU1dsntp001 up 11:55 [ Italian ] 2 good news in 2 mins .... my heart could stop 11:55 [ DenBolle ] har har.. we're a good team, it's dun to work like thois this dun = fun.. 12:00 [ Polish-2 ] ok, the whole storage booted just fine.. now I'm going to make mess with it 12:00 [ Italian ] ok 12:02 [ DenBolle ] so at this point, we about 2,5 hrs behind on schedule.. 12:02 [ Italian ] yes 12:02 [ DenBolle ] sorry. 12:02 [ Italian ] np How long [Pg-2]? 12:02 [ Polish-2 ] i think max. 30 min 12:03 [ Italian ] great 12:03 [ DenBolle ] So you're making up for my mistake, and then we're back on schedule.. cool 12:03 [ Italian ] don't worry [Pg-2] even 1 hour 12:03 [ DenBolle ] no, I'll stay onsite as long as is needed. 12:29 [ Polish-2 ] OK - seems to be done. LUNs from BU2 storage are remapped to the new ESX hosts with IDs starting from 10. Zoning on the FC switches is also reconfigured. 12:31 [ DenBolle ] ok, so next steps are powering up ESX hosts? 12:31 [ Polish-2 ] [Pg-1] - I think that you can start ESX hosts now. Please check the connection to the storage before starting up the VMs. 12:31 [ Polish-1 ] OK. Please let me celebrate for a while this moment when world's eyes are on me. 12:31 [ Polish-2 ] haha 12:32 [ DenBolle ] hahaha. Big aplause for you ! 12:34 [ Polish-2 ] I've made it in 30 mins - I can afford to eat breakfast now. 12:34 [ Polish-1 ] I'll power them up one by one. BU1sxve001 goes first 12:34 [ Polish-2 ] OK, I will be watching.. on the FC switches. As per FC point of view - it's up. And each Netapp can see it. 12:59 [ Polish-1 ] all 3 ESX host are up but they can't see new LUNs. I performed rescan onstorage but it not helped. I need to check this. 13:01 [ Polish-2 ] I can only say that from my side all seems to be correct. Netapps see each ESX and those ESX are logged in on Netapp adapters. FYI: There should be 4 new LUNs visible, with IDs 10,11,12,13. 13:03 [ DenBolle ] ok, np. take your time to find out whats going on. 14:05 [ Polish-1 ] OK. Now I can see all LUNs on all 3 ESX hosts. @[Pg-2]: We need to verify paths now. Please provide me paths for LUN S1V1 (108,5GB) 14:07 [ Polish-2 ] Sure thing. You have to to tell me this LUN id. Because we have now double pairs of S1V1, S1V2, and so on... 14:08 [ Polish-1 ] LUNs from BU2echt are described as: S1V1_BU2 S1V2_BU2 and so on 14:09 [ Polish-2 ] ok, so first the LUNs from BU1ST Netapps: 14:09 [ Polish-1 ] S1V1 14:10 [ Polish-2 ] S1V1 (880.1g, LUN id: 0) S1V2 (805g, LUN id: 1) S1V3 (563.1g, LUN id: 4) Correct path for them is: 500a098195065a78 14:10 [ Polish-1 ] This should be set as prefferd right? and active 14:11 [ Polish-2 ] S2V1 (955.1g, LUN id: 2) S2V2 (730.1g, LUN id: 3) S2V3 (563.1g, LUN id: 5) Correct path for these is: 500a098185065a78 Correct. It should be considered as preferred (primary) path. BU2 LUNs: S1V1 (840g, LUN id: 10) S1V2 (840g, LUN id: 11) Correct path: 500a098195364ca3 S2V1 (840g, LUN id: 12) S2V2 (840g, LUN id: 13) Correct path: 500a098185364ca3 Uff.. that's all. 14:14 [ Polish-1 ] ok please give me a few minutes to check all hosts 14:29 [ Polish-1 ] Ok. I verified everything and corrected paths. Now I suggest to power on domain controller. VM in BU1. 14:29 [ Italian ] ok for me 14:30 [ Polish-1 ] then I'll add to inventorythose machines which were migrated. 14:31 [ DenBolle ] ok, well, so far all goes well. 14:32 [ Polish-1 ] Den, an all BU1st machines the second DNS entry is set to the BU2 DC. I suggest to change it to STVSXDMC018 22.214.171.124 and the primary DNS leave as is (BU1SXDMC002). 14:33 [ DenBolle ] ok, do so, do you want me to submit a ticet for it? 14:33 [ Polish-1 ] no. 14:36 [ Italian ] [Pg-1], about BU2SDWN001 (FV) how long does it take to add to inventory ? 14:36 [ Polish-1 ] it should be one click. I'll do it now. 14:37 [ Italian ] thks 14:40 [ Italian ] I see it 14:43 [ Italian ] please remember to change IP 14:50 [ Polish-1 ] BU2SDWN001 is working under new IP now. Now I will add BU2SXFILE001 to the inventory and change the IP. Then I'll call DNS team to perform DNS changes. OK? 14:51 [ DenBolle ] ok 14:51 [ Italian ] moment DNS is already updated ? 14:51 [ Polish-1 ] no 14:51 [ Italian ] ok go ahead and I told C. to wait for tests 14:51 [ Polish-1 ] ok 15:23 [ Polish-1 ] I powered on BU2SXFILE001 and changed IP. Then I added a new disk to BU1SXFILE001 and powered it up. Now I catch up with network team for DNS changes. 15:23 [ Italian ] ok 15:25 [ Italian ] ANy idea about new ip propagation time ? 15:29 [ Polish-1 ] [Pg-2]osz Ostrowski is workin on it now. He does not know how log it may take as he's not doing it usually. Only today was caought by me. cought* 15:31 [ Italian ] because I suppose you already started DNS change activity, from my experience it could take also 24 hours to propagate I don't know if is possible to force it 15:31 [ Polish-1 ] I provided him numbers of those 2 requests. 15:33 [ DenBolle ] and a forced propagation isn't a option? 15:33 [ Italian ] Den What's your idea about FV tests, in the worst case can we done them tomorrow ? 15:35 [ DenBolle ] I think we can, let me verifyt with C. on this one. Its's possible to do it from home for him 15:39 [ Polish-1 ] So far from Poznan wi have this: C:\>ping BU2sdwn001 Pinging BU2sdwn001.lan01.com [126.96.36.199] with 32 bytes of data: Reply from 188.8.131.52: bytes=32 time=63ms TTL=119 Reply from 184.108.40.206: bytes=32 time=63ms TTL=119 Reply from 220.127.116.11: bytes=32 time=64ms TTL=119 Reply from 18.104.22.168: bytes=32 time=63ms TTL=119 Ping statistics for 22.214.171.124: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 63ms, Maximum = 64ms, Average = 63ms 15:41 [ Italian ] the same from Italy .... 15:42 [ DenBolle ] I've talked with C., tomorrow testing is possible, but before 1000 hrs, he has other apointments 15:42 [ Italian ] I think he caould start test now DNS looks updated 15:43 [ DenBolle ] ok, I'll inform him. 15:43 [ Italian ] thks 15:43 [ Polish-1 ] I'm gonna start next VMS ok? 15:43 [ Italian ] yes please 15:44 [ DenBolle ] C. wil start testing right away. 16:00 [ Polish-2 ] [Pg-1], are you starting the other VMs as well? I need to check the BU1SMLXV0001 server. 16:01 [ DenBolle ] C. reports an issue : mq service isn't running, I'm working on that. 16:02 [ Italian ] ok in case we can open a ticket to GSA-WW-MIDDLEWARE 16:07 [ DenBolle ] ok, problem killed.. C. tests further. 16:07 [ Italian ] good job "terminator" 16:16 [ Polish-1 ] I'm starting VMs one by one. 16:16 [ DenBolle ] ok, al seems in good working order ? 16:17 [ Polish-1 ] so far so good 16:17 [ Italian ] [happy] 16:17 [ DenBolle ] you optimistic guy 16:32 [ Polish-1 ] BU1SMLXV0001 is up now 16:32 [ Polish-2 ] thanks, I've just finished checking it all seems to be fine, all filesystems mouted, oracle processes started By the way, after the action, we need to raise a high priority ticket to DBA team with request to check databases on affected servers. I discussed this with [naam] from DBA, and he asked me to do so. 16:34 [ Italian ] good idea 16:35 [ Polish-2 ] How about other VMs? Did they started properly? No issues yet? 16:35 [ DenBolle ] C. reports no issues on FV and MQ series, [Ig-1], he'll sent you a test report. 16:35 [ Italian ] fantastic 16:36 [ Polish-1 ] I'm checking them one by one. We'll need to ask [Pg-3] to check if backup works fine now. 16:36 [ Italian ] [Pg-1] did you start data copy from BU2SXFILE001 ? 16:37 [ DenBolle ] should his wife be done with hunting now? 16:37 [ Polish-2 ] We'll see.. ; 16:37 [ Polish-1 ] not yet. I need to prepare jobs for SecureCopy. Fortunatelly I installed it yesterday. My wife? 16:38 [ Italian ] [Pg-3]'s wife 16:38 [ DenBolle ] No, [Pg-3]'s wife 16:38 [ Polish-1 ] Also Lotus team could be informed to check Domino server. 16:38 [ Italian ] Den Can I promote you leader? 16:39 [ DenBolle ] Sure, does it pay more? 16:39 [ Italian ] 65% more 16:39 [ DenBolle ] Give it to me! 16:39 [ Italian ] or 67% don't remember I really have to go ........ but I'm available on call (if necessary , but i don't think so, your are in good hands ) 16:41 [ DenBolle ] No problem. I have your mobile. [realises I'm the guy for the coffee now.. ] 16:42 [ Italian ] ehehehe I'm happy that FV tests were ok 16:43 [ DenBolle ] me to, specially beacuse that's something automated outside us, and BU2 depend on it.. 16:43 [ Italian ] yes Thanks a lot [Pg-1] and [Pg-2], I didn't do nothing "today" but I wish you feel my support remotly 16:45 [ Polish-1 ] Thank you too. 16:45 [ DenBolle ] [Ig-1], thx sofar for your help and coordination. When we're done I'll sent you a txt message to inform you. 16:45 [ Polish-2 ] Sure, [Ig-1]. It was good to have you here. Thank you! 16:45 [ Italian ] too good guys Thanks Den a call or a message will be really appriaciate 16:46 [ DenBolle ] I'll do, count on it. 16:46 [ Italian ] grazie bye bye 16:46 [ Polish-2 ] bye 16:46 [ DenBolle ] bye 16:47 [ Polish-2 ] [Pg-1], are all VMs up by now? If yes, I would raise ticket to DBA team. 16:48 [ DenBolle ] ok, just a short next-steps overvieuw : after al VMS are up, db checks, inform notes team, and run a test backup. Anything else? 16:49 [ Polish-2 ] from my side, nothing to add 16:49 [ DenBolle ] ok, [Pg-1]? 16:51 [ Polish-1 ] and copy BU2SXFILE001 to BU1SXFILE001 this can last few hours 16:52 [ DenBolle ] ok, and all would take about say 2-3 hrs from now? (I've got to inform security here that it's going to be a little later than 1800 hrs) 16:52 [ Polish-1 ] no idea. It's 500GB to copy. 16:53 [ DenBolle ] thats (500 diveded by 16M memory sticks) say quite some later than 1800 hrs. I'll keep it at 2000 hrs ok? 16:54 [ Italian ] [Pg-1] could you be there unil the end of the copy ? 16:54 [ Polish-1 ] ok. If it's not finished until that time you can go home and I'll let you know via sms if all data is copied. I'll be here until the end of the copy. 16:55 [ DenBolle ] No, I can stay as long as needed, but I would like to inform the security guy before he's disapointed. 16:55 [ Italian ] because last step is to stop BU2SXFILE001 16:55 [ DenBolle ] /scared because [Ig-1]'s ghost is here.. 16:55 [ Italian ] eheheheh I'm waiting for my wife 16:56 [ Polish-1 ] I know but I have no info from K. if he checked if we have access to every file. If I need correct something manually it may last longer. 16:56 [ Italian ] she has to drive me home 16:57 [ DenBolle ] ok, for that lets see what happens when the copy part is starting. And have a chat about that then. First lets see that we have the DB's checked and Domino team informed. 16:57 [ Polish-1 ] one moment. I have an urgent reboot in Canada. 16:58 [ Polish-2 ] Den, I am wondering whether you could raise a ticket to DBA team when all VMs are up? 16:58 [ DenBolle ] sure, np, give me the RA and I'll submit one 16:58 [ Polish-1 ] in a minute or two 16:59 [ Italian ] TECH-CENTRE-DATABASE-SUPPORT 16:59 [ Polish-1 ] right 16:59 [ Polish-2 ] if this is a minute or two, I think that I will manage to do that by myself. 17:00 [ Polish-1 ] ok all VMs are up now 17:02 [ DenBolle ] allright, who's going to submit the ticket? to prefent a double entry. 17:02 [ Polish-2 ] I am going to submit the ticket to DBA team. 17:02 [ DenBolle ] ok 17:05 [ Italian ] good luck and hope you finish asap 17:05 [ DenBolle ] Thx, and snjeoy your evening! snjeoy = enjoy 17:06 [ Italian ] :-) 17:06 [ Polish-2 ] Ticket to DBA: UKIM20002972635 17:07 [ DenBolle ] ok, let see if they pcik it up quickly 17:08 [ Polish-1 ] I talked to [Pg-3]. He'll check backups about * PM today and he'll send us an email with update. 17:08 [ DenBolle ] ok, so that's covered. What about Domino team? 17:08 [ Polish-1 ] I'm creating a ticket now 17:09 [ DenBolle ] Great, this is going well guys, thx a lot for your effort on this one. 17:11 [ Polish-1 ] UKIM20002972653 17:11 [ DenBolle ] ok, thx 17:32 [ Polish-1 ] OK I started copying data from BU2SXFILE001 to BU1SXFILE001 1 min 200 MB 2 min 410 MB 3.73 MB/s 17:34 [ DenBolle ] mhm.. if I calculate right 200 mb / min = 1Gb/ 5 min --> 12Gb / hrs.. 17:35 [ Polish-1 ] looks like this 17:35 [ DenBolle ] ie.. whole copy will take approx 40 hrs.. and not ready monday morning.. 17:36 [ Polish-1 ] 5 min and 1.16GB now performance is about 4 MB/s 17:38 [ DenBolle ] A. (messaging team) tells that domino is working fine.. 17:38 [ Polish-1 ] ok 17:39 [ Polish-2 ] No one from DBA haven't picked up the ticket yet. I'll try to contact them now. 17:40 [ DenBolle ] ok. [Pg-1] : lets wait a litte to see if it speeds up, but 4mb/sec is to slow to get ready 17:41 [ Polish-1 ] ok now I have 3.61MB /s and 2.17GB copied we can leave BU2SXFILE001 working and correct DNS entry. The migration can be done later if the server is accessable for users after migration. What you think? 17:43 [ DenBolle ] like having a shadowcopy in place and then just a smal migration? 17:44 [ Polish-1 ] migrate files during the week and during the weekend copy only files which changed. 17:44 [ DenBolle ] yeah, that's the best approach I think, sync the files, and switch in the next weekend. how does that goes with the change we requested? can we fit it in? 17:45 [ Polish-1 ] We can do an exception. it's always possible. 3.34MB/s now 17:46 [ DenBolle ] ok, so the chance that it will speed up significant is low, let's go for the 'hidden sync' and migrate next weekend then. in this speed we're doomed 17:47 [ Polish-1 ] ok so I need to rise a ticket for a DNS entry change. Then we will check if it works. I'll keep copying task running during the weekend to have the main part already copied. A new change request will be required but I'll prepare it. 17:48 [ DenBolle ] ok, and the DNS is only for the BU2SXFILE001 right? 17:48 [ Polish-1 ] right this will point now to the new address 126.96.36.199 right/ 17:49 [ DenBolle ] let me fetch that one, it... 17:56 [ DenBolle ] just a sec again.. I've got .164 .. let me fect the documentation to [Ig-1] 17:56 [ Polish-1 ] ok 17:57 [ DenBolle ] The last IP address 188.8.131.52 has been required as a temporary address in case of any issue on migrated VMs. (Temporary emergency Vms component) 17:58 [ Polish-1 ] From changerequest: 3. 15 min. Modify the IP address and default gateway of BU2SDWN001 as per configuration items changes. Change the IP of BU2SXFILE001 to 184.108.40.206 and apply BU1 network settings. To provide screen shot of new IP config. 17:58 [ DenBolle ] yeah, it's ok. use it 17:59 [ Polish-1 ] ok so I'm rising a ticket for DNS change. 17:59 [ DenBolle ] wait.. 17:59 [ Polish-1 ] ok 18:03 [ DenBolle ] mhm.. this one isn't clear. the doc from [Ig-1] (23/12/2010) states .164 as FS ip. and the 168 as emergency... Go ahead: use .168: this is an emergency. 18:03 [ Polish-1 ] ok 18:04 [ DenBolle ] I'll update [Ig-1] on this per email. 18:08 [ Polish-1 ] 7002306 18:10 [ DenBolle ] ? 18:10 [ Polish-1 ] request number to network for DNS change 18:10 [ DenBolle ] ah ok [Pg-2], any luck on the DBA team? 18:13 [ Polish-2 ] not yet, I wrote a SMS to person on-duty, without any luck yet I'll wait additional 5-10 mins and I will call them 18:13 [ DenBolle ] ok, in that case I'll hand out the coffee 18:13 [ Polish-2 ] ok 18:14 [ DenBolle ] for [Pg-1], and this one for you [Pg-2]osz 18:14 [ Polish-2 ] thx a lot 18:14 [ DenBolle ] mail from [Pg-3] : Hello All, Just FYI. The backup service is OK after the migration. I have just started all quarter jobs except BU1SXFILE001 that is during data copy. So far so good for the backup side. Regards, [Pg-3] 18:14 [ Polish-2 ] good 18:26 [ Polish-2 ] weird.. the guy who is supposed to be on-duty from DBA team is not answering neither the SMS and the phone calls.. 18:26 [ DenBolle ] and a 'second' one isn't available? 18:27 [ Polish-2 ] I will try to call the 'second' one 18:31 [ Polish-1 ] 21,72Gb in 1 hour 18:31 [ DenBolle ] hey.. 18:31 [ Polish-2 ] so it seems that speeded up? 18:32 [ DenBolle ] thats twice what we expected.. 18:32 [ Polish-2 ] 25h for 500GB 18:32 [ DenBolle ] ie.. tomorrow al is done? 18:32 [ Polish-1 ] now 6.17 but we don't know how many small files are befor us 18:32 [ DenBolle ] if no files have issues that is. 18:32 [ Polish-1 ] no it's about 26 hours it's still risky 18:33 [ DenBolle ] lets keep the BU2SXFILE001 alive.. to be save. 18:33 [ Polish-1 ] yes 18:34 [ DenBolle ] and stick to the plan. A dns change is easy. If all is save we can do it perhaps earlier then next weekend. 18:36 [ Polish-1 ] sure 18:49 [ Polish-2 ] I've contacted [naam] from DBA at last.. He has muted his phone and left it in thie jacket's pocket.. and didn't hear that it was ringing. He will be on-line in a minute and check DBs. 18:50 [ DenBolle ] pfoe.. I assume we can recognize [naam] at his black eyes monday? 19:00 [ Polish-1 ] C:\>ping BU2sxfile001 Pinging BU2sxfile001.lan01.com [220.127.116.11] with 32 bytes of data: Reply from 18.104.22.168: bytes=32 time=95ms TTL=119 Reply from 22.214.171.124: bytes=32 time=92ms TTL=119 Reply from 126.96.36.199: bytes=32 time=77ms TTL=119 Reply from 188.8.131.52: bytes=32 time=75ms TTL=120 Ping statistics for 184.108.40.206: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 75ms, Maximum = 95ms, Average = 84ms C:\> 19:01 [ DenBolle ] great thx. Solved this one. After DBA verification we're done for today ? 19:02 [ Polish-1 ] Please check if you can map shares without problems. If yes, we can say that's all for you today. 19:07 [ DenBolle ] BU2sxfile001 resolves to 220.127.116.11 --> BU1sxfile001 when was the dns set? 19:08 [ Polish-1 ] I talked to this guy. This will be set in a minute. let's wait for a while. They also have a tough day. 19:08 [ DenBolle ] sure, np.. 19:23 [ DenBolle ] on the BU1SXFILE001 the BU2SXFILe001 resolves to 18.104.22.168, but on the network it doesn't, I'll wait some more. 19:24 [ Polish-1 ] [Pg-2] is working on it now. 19:25 [ DenBolle ] sure, no hurries on this one 19:25 [ Polish-1 ] He had a call to Riga network outage till monday 19:25 [ DenBolle ] in Riga? 19:25 [ Polish-1 ] yes. Packets get lost from time to time. my remedy queue is blue 19:26 [ DenBolle ] because you're working all day allready ? 19:26 [ Polish-1 ] yes 19:27 [ DenBolle ] grin grin.. just keep up a little.. we're almost done for today 19:27 [ Polish-1 ] you are almost done I'm till midnight 19:28 [ DenBolle ] wow.. thats 18 hrs ? 19:28 [ Polish-1 ] 17 19:28 [ DenBolle ] what happend? 19:28 [ Polish-1 ] when we are on remedy during weekend we have to proceed all weekend's tasks that's all. my weekend shift is between 12PM - 24 because today we had BU2migration I had to start earlier. There was an option KL... 19:29 [ DenBolle ] in that case : my compliments ! I appriciate it that you supported us outside your regular shift times. 19:30 [ Polish-1 ] It's always better to have work in one heands that pass it on and on from person to person. 19:31 [ DenBolle ] I know what you mean.. losing that one detail and then no one knows what happened. 19:31 [ Polish-1 ] eg. 19:39 [ DenBolle ] allright : in the network the BU2sxfile001 in the bu1 subnet works at 22.214.171.124 and mapping is possible. I'm now going to the bu2 network to verify a login 19:40 [ Polish-2 ] A. just sent the info about DBs: all is fine. In that case, I think that my work for today is done. 19:42 [ Polish-1 ] ok. thank you for your assistance. Have a nice sleep. 19:44 [ Polish-2 ] Thank you for your support. I wish you [Pg-1] a calm on-duty time. I hope that you'll be able to relax tomorrow. 19:44 [ Polish-1 ] thanks, I hope so. 20:19 [ DenBolle ] ok, the followme system has issues. I've shut down all printers and will restart them one by one 20:19 [ Polish-1 ] ok and now? 20:20 [ DenBolle ] I wait a few mins to calm the server. It can't handle the amount of config requests, and then I'll boot up the printers again 20:20 [ Polish-1 ] ok 20:22 [ Polish-2 ] OK, I'm logging off now. @Den: Thanks for cooperation. I hope that all remaining issues will be resolved soon. Thank you guys, have a nice Sunday! bye 20:22 [ Polish-1 ] bye! Thank you! 20:43 [ DenBolle ] [Pg-1], I need a reboot on the BU1SPWN001 ... the priner server isn't acting wel 20:46 [ Polish-1 ] ok I'll do it right now. 20:46 [ DenBolle ] great 20:50 [ DenBolle ] and? 20:50 [ Polish-1 ] server is booting up now 20:50 [ DenBolle ] top, thx 20:53 [ DenBolle ] ah.. it's also the DHCP server... 20:53 [ Polish-1 ] yes 20:53 [ DenBolle ] lost my connection I'm on wifi, is it back on ? 20:55 [ Polish-1 ] yes the server is back on 20:56 [ DenBolle ] ok, let me see that I'm getting the printer back on 20:56 [ Polish-1 ] ok 21:48 [ DenBolle ] ok, I'm back.. All printers tested and are working. 21:48 [ Polish-1 ] ok good. what next? 21:48 [ DenBolle ] beer? 21:48 [ Polish-1 ] I'm at work 21:49 [ DenBolle ] I've got to drive the motor for 1,5 hrs... then beer We're done I think. Db's checked, mappings checked, printers checked, all works, only thing now is copying the data 21:50 [ Polish-1 ] 83Gb so far and 5.52 MB/s 21:50 [ DenBolle ] so we're done in the majority of task 21:50 [ Polish-1 ] I think yes. I'm happy it worked. 21:54 [ DenBolle ] me to. Thx a lot for all. I'll inform the users. Next weekend the last step ? 21:55 [ Polish-1 ] I think yes. I'll contact you in the week. Thanks you. Have a nice end of weekend. 21:55 [ DenBolle ] ok, excellent. Thanks for your help and patience. Hear you next weekend.