Announcement

Collapse
No announcement yet.

A little work story

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • A little work story

    Over the last few months we've been moving our branch locations onto a fiber optic based layer 2 network as our T1 contracts end. Along with the new connections we've been installing new 2811 routers to replace our aging 2621s.

    Because this was a phased migration we had to keep some older equipment in production until all the T1s were taken down. Our core router, a 3640, located at our main branch had all the T1's terminating into it. Once we started hooking up the fiber we installed a 2811 at our main branch and ran it in parallel with our 3640.

    Just last week we got the last branch moved onto the fiber. Depsite that move the 3640 was still doign some work routing traffic to a few third party routers we have at our site. It was also acting as the default gateway for all of our clients.

    With the new 2811 at our main site having all the static third party routes added to it my plan was to wait until our branches closed and then change the 3640s IP to something different and then have the 2811 take the old IP of the 3640. That way I would not have to worry about clients and other third party devices (routers) having to change their default route.

    The plan seemed to work out well. I tested several sites including our third party apps and they all worked. With that done I went to punch out. Our timeclock is a web based app that we access via the internet. For some reason it was not loading properly, but the internet was working just fine. I had done some browsing to make sure of that when I did my testing. I chalked it up to the webclock being down yet again and sent my boss an email to alter my time.

    That all happened last night. This morning, about 30 minutes before I usually get up my boss called me saying that some of the early morning staff were having problems accessing sites. It was all very intermittent and inconsistent. One person could access this secure site whereas the next person could not. Internet access seemed a bit sluggish as well. I tried to VPN in but for some reason I could not access anything on our internal network so I got ready early and headed straight into work. The fact that I made a network change the previous night pointed me in the direction that I had messed something up. But, we could access the internet. It was just flaking out on us randomly and secure sites in particular were having difficulties. We use Websense for internet filtering so out first step was to shut that server down to eliminate it as a possible cause. At the same time I logged into the 2811 at our main site and began looking through the config fully expecting not to find anything.

    This is what I find.

    ip route 0.0.0.0 0.0.0.0 172.16.1.1
    ip route 0.0.0.0 0.0.0.0 172.16.1.2

    1.1 is the internal address of our ASA
    1.2 is the old address of our 3640 which is now the address of the 2811.

    With the 2811 running in parallel with the 3640 I gave the 2811 a default route of the 3640. When I decommisioned the 3640 I added the ASA as my default route while failing to remove the old default route.

    Usually when I've messed something up on a router one or more networks become completely inaccessible which immediately tells me that I've done something wrong. With this mistake the problems were very intermittent and seemed to point away from the router as being the problem. Once I removed the extra default route the problems immediately went away.

    Happy Friday!!!
    Last edited by bill_sffcu; 7th August 2009, 16:00.
    CCA: XenApp 5.0

  • #2
    Re: A little work story

    When troubleshooting, always refer to the OSI model. In many cases, the problem is layer 8

    We had a problem where a user had come in and not been able to login. So he told everybody in his office that the system was down and not to turn their computers on.

    My boss gets in and the user speaks to him. The boss has already been able to login by this point... so resets the user's password and tells everybody else they can login.

    I was on holiday that day. When I got in the next day, the user came to me and said they couldn't login. So I reset his password. What I also did though, which my boss didn't do, was go and watch the user login.

    Overnight, it seemed he'd decided his username was no longer firstname.lastname, but now [email protected]

    Unfortunately it's frowned upon to place users in sacks and beat them like baby seals.
    Gareth Howells

    BSc (Hons), MBCS, MCP, MCDST, ICCE

    Any advice is given in good faith and without warranty.

    Please give reputation points if somebody has helped you.

    "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

    "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

    Comment


    • #3
      Re: A little work story

      That would be fun gforce. But what if we are the ones messing things up? Do the users get to put us in sacks and beat us like baby seals?

      CCA: XenApp 5.0

      Comment


      • #4
        Re: A little work story

        We don't mess up. We just discover udocumented bugs.

        Besides, we are the keepers of the sacks
        Gareth Howells

        BSc (Hons), MBCS, MCP, MCDST, ICCE

        Any advice is given in good faith and without warranty.

        Please give reputation points if somebody has helped you.

        "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

        "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

        Comment


        • #5
          Re: A little work story

          Originally posted by bill_sffcu View Post
          That would be fun gforce. But what if we are the ones messing things up? Do the users get to put us in sacks and beat us like baby seals?

          When we mess up it's because we're "beta testing" new products or procedures. When users mess up it's because they still haven't grasped what we've told them four times.

          How many times have we all thought that we've pretty much seen it all, and that no fault caused by a user could ever take us by surprise, only to swiftly be proven wrong.

          To cheer you all up this link is a clip of Irish Comedian Dara O'Briain on the subject of IT, with worrying accuracy.
          BSc, MCSA: Server 2008, MCSE, MCSA: Messaging, MCTS
          sigpic
          Cruachan's Blog

          Comment


          • #6
            Re: A little work story

            Damn you. Doctor told me I'm not allowed to watch Live at the Apollo without a pint in me hand

            Ahh Dara... one of the jewels in the crown of Britsh comedy. Ok so he's from the Irish part of Ireland, but it's close enough

            I do think he showed a general lack of IT experience though - surely the problem->solution process should have ended..

            "Have they gone?"
            *switch it on*
            Gareth Howells

            BSc (Hons), MBCS, MCP, MCDST, ICCE

            Any advice is given in good faith and without warranty.

            Please give reputation points if somebody has helped you.

            "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

            "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

            Comment


            • #7
              Re: A little work story

              Thanks for sharing that with us Bill.

              Isn't it funny, well not really, how it is the tiny little things that trip us up? Had a proxy script not work on a Proxy 2 Server (yes, I am that old ). Anyone remember the bloody syntax?
              http://servername:80/array.dll?Get.Routing.Script (we had maximum length server names at the time which didn't help). I copied the the text, dropped it into a Word document and ENLARGED it until I found a pesky little comma instead of the required fullstop. The optomitrist appreciated the business the incident resulted in. Was also able to claim the glasses on tax since they became an essential work tool.
              1 1 was a racehorse.
              2 2 was 1 2.
              1 1 1 1 race 1 day,
              2 2 1 1 2

              Comment


              • #8
                Re: A little work story

                Glasses etc may be covered under our company health plan, depending on job description - obviously that costs the company money. I managed to convince the accounts department that it'd be cheaper if they let me just buy everyone a 23" 1080p monitor (and then in the case of most users, just not run them at 1080p )

                For those in the UK with a Dabs4Work account... 95. How they get them that cheap is beyond me. And despite the 'value' label they're pretty damn good - certainly match the 150 Samsungs we were buying before in terms of quality.
                Gareth Howells

                BSc (Hons), MBCS, MCP, MCDST, ICCE

                Any advice is given in good faith and without warranty.

                Please give reputation points if somebody has helped you.

                "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

                "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

                Comment


                • #9
                  Re: A little work story

                  Showing my age but biggles's story reminded me of doing something almost identical when typing in a game program from a book. I had a comma instead of a fullstop or vice versa (It was a LONG time ago ) and it took me ages to figure it out. I was so happy that I had done, and then gutted swiftly after as the game was rubbish.
                  BSc, MCSA: Server 2008, MCSE, MCSA: Messaging, MCTS
                  sigpic
                  Cruachan's Blog

                  Comment


                  • #10
                    Re: A little work story

                    Good times I grew up with a Dragon64. Remember when you used to get a powercut right before you'd finished saving a program that took you 2 days to write to cassette?
                    Gareth Howells

                    BSc (Hons), MBCS, MCP, MCDST, ICCE

                    Any advice is given in good faith and without warranty.

                    Please give reputation points if somebody has helped you.

                    "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

                    "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

                    Comment


                    • #11
                      Re: A little work story

                      BBC B and a Speccy 48K for me. Every now and then I'll fire up BeebEm on my laptop and play Manic Miner and go to sleep that night with "In the Hall of the Mountain King" running through my head all night.

                      I remember being jealous of people who had tape decks with 7pin DIN connectors that could be sent power signals by the BBC and didn't have to be stopped and started manually.
                      BSc, MCSA: Server 2008, MCSE, MCSA: Messaging, MCTS
                      sigpic
                      Cruachan's Blog

                      Comment


                      • #12
                        Re: A little work story

                        I got a second hand BBC Master and an Acorn Electron from my primary school too... it just wasn't the same though.

                        It's funny how these modern languages without GOTOs make you appreciate DragonBasic
                        Gareth Howells

                        BSc (Hons), MBCS, MCP, MCDST, ICCE

                        Any advice is given in good faith and without warranty.

                        Please give reputation points if somebody has helped you.

                        "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

                        "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

                        Comment

                        Working...
                        X