Announcement

Collapse
No announcement yet.

Exchange & IIS Services crash with 7031/7034 Errors

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Exchange & IIS Services crash with 7031/7034 Errors

    Hi there
    I have a couple of servers that have started to display some odd service crashes.

    One server is Windows 2003 Small Business Server, the other is Windows 2003 Server with Exchange 2003 Standard.

    Both servers are fully up to date with Microsoft patches as of last "Patch Tuesday" and I have checked them both today to ensure there is nothing new for them.

    Both servers are running Symantec AntiVirus Corporate Edition 10 and Symantec Mail Security for Exchange with the Premium AntiSpam plugin.

    The symptoms on both servers are similar, but not exactly the same, I am going to concentrate on the non-SBS server in this post to make it easier for me to describe.

    The initial symptom that led to investigation was the website/webmail being down and mail sitting in the Outbox in Outlook.

    Investigation of the System event log shows the following four errors:

    ---------------------------
    Source: Service Control Manager
    Type: Error
    Event ID: 7031
    Time: 11:51:01 PM

    The IIS Admin Service service terminated unexpectedly. It has done this 58 time(s). The following corrective action will be taken in 1 milliseconds: Run the configured recovery program.
    ---------------------------
    Source: Service Control Manager
    Type: Error
    Event ID: 7034
    Time: 11:51:01 PM

    The Microsoft Exchange Routing Engine service terminated unexpectedly. It has done this 57 time(s).
    ---------------------------
    Source: Service Control Manager
    Type: Error
    Event ID: 7034
    Time: 11:51:01 PM

    The Simple Mail Transfer Protocol (SMTP) service terminated unexpectedly. It has done this 58 time(s).
    ---------------------------

    I also sometimes, but not always, get these errors:

    ---------------------------
    Source: W3SVC
    Type: Error
    Event ID: 1011
    Time:

    A process serving application pool 'ExchangeApplicationPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '7812'. The data field contains the error number.
    ---------------------------
    Source: W3SVC
    Type: Error
    Event ID: 1013
    Time:

    A process serving application pool 'ExchangeApplicationPool' exceeded time limits during shut down. The process id was '5120'.
    ---------------------------
    Source: W3SVC
    Type: Error
    Event ID: 1010
    Time:

    Inetinfo terminated unexpectedly and the system was not configured to restart IIS Admin. The World Wide Web Publishing Service has shut down.
    ---------------------------

    It seems a lot like Microsoft KB Article 827214 however in that article the link sends me here and then onto here where it links me to the product update

    The update was published in 2004 and it appears that it would be contained in one or another service pack, so it doesn't seem like this is the fix for me.

    There are a few other people turning up here and there on the internet, but not great reams yet.

    I have a gut feeling that it is a new vulnerability, but it could just as easily be something Symantec screwed up in an update, or just some bug.

    I have turned on logging for incoming SMTP connections and activated the SMTP logging on both servers, and am now sitting with both servers up on my screen in the middle of the night trying to find more information.

    I will keep it coming as I have it.

    I am also going to cross post this to Experts Exchange and the Microsoft.Public.Exchange.Admin group - hopefully if it is a serious problem that will help people find answers.

  • #2
    Re: Exchange & IIS Services crash with 7031/7034 Errors

    Did you try disabling the Symatec AV for (just for test)?
    CNE 5, CCA, MCSE NT4.0-2003, MCSE 2003 messaging, Exchange Server MVP.
    Tzahi Kolber - IT Supervisor
    Polycom Israel.

    Comment


    • #3
      Re: Exchange & IIS Services crash with 7031/7034 Errors

      That is my next step - at the moment even if it is the AV stuff causing it, I want to leave it up so I can get as much information as I can.

      Once that is done, I will go to bed, get up early and kill the AV stuff. If that fixes it, it can stay that way during the day so the users can use the server, and I will take it from there.

      Thanks for the suggestion.

      Comment


      • #4
        Re: Exchange & IIS Services crash with 7031/7034 Errors

        Further Details

        On my server that is SBS, I just got a crash at 00:36:28

        The firewall and 2K3 server have their clocks synchronised, so it is easier to pull together the logs on that one (I am having a separate issue getting access to my firewall to adjust the time for a 3 minute drift and daylight savings on the other site)

        Here is how it went down:

        Event log - 7031(IIS)/7034(NNTP)/7034(MS Exchange Routing Engine)/7034(SMTP) all fall over at 00:36:28

        Firewall - SMTP incoming traffic:

        00:36:23 FROM 200.119.210.170 (Unresolvable)
        00:36:24 FROM 160.83.65.200 (Unresolvable)
        00:36:29 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
        00:36:31 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
        00:36:37 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
        00:36:38 FROM 203.94.218.69 (dialup-mum-203.94.218.69.mtnl.net.in)
        00:37:58 FROM 64.237.216.98 (adsl-64-237-216-98.prtc.net)

        SMTP Logs (we are +11hr on GMT here)

        2006-12-04 13:36:01 129.41.76.38 mail2038.rm02.net SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 BDAT - +<[email protected] > 250 0 129 86408 20750 SMTP - - - -
        2006-12-04 13:36:01 129.41.76.38 mail2038.rm02.net SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 QUIT - mail2038.rm02.net 240 23734 69 4 0 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 220+ESMTP+on+WinWebMail+[3.7.3.1]+ready.++http://www.winwebmail.com 0 0 67 0 188 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 EHLO - domainchanged.com.au 0 0 4 0 203 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250-SIZE 0 0 8 0 391 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250+AUTH+LOGIN 0 0 14 0 703 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 MAIL - FROM:<> 0 0 4 0 703 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250+OK 0 0 6 0 906 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 RCPT - TO:<[email protected]> 0 0 4 0 906 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250+OK,+recipient+accepted 0 0 26 0 1094 SMTP - - - -
        2006-12-04 13:36:05 61.129.51.171 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 DATA - - 0 0 4 0 1094 SMTP - - - -
        2006-12-04 13:36:06 61.129.51.171 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 354+Send+checkpointed+message,+ending+in+CRLF.CRLF 0 0 50 0 1297 SMTP - - - -
        2006-12-04 13:36:26 200.119.210.170 plwrag SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 HELO - +plwrag 250 0 50 11 0 SMTP - - - -
        2006-12-04 13:36:26 160.83.65.200 imr8.us.db.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 EHLO - +imr8.us.db.com 250 0 313 19 0 SMTP - - - -
        2006-12-04 13:36:26 160.83.65.200 imr8.us.db.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 MAIL - +From:<[email protected]> 250 0 45 52 0 SMTP - - - -
        2006-12-04 13:36:26 160.83.65.200 imr8.us.db.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 RCPT - +To:<[email protected]> 250 0 39 36 0 SMTP - - - -
        2006-12-04 13:36:27 200.119.210.170 plwrag SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 MAIL - +FROM:+<[email protected]> 250 0 43 31 0 SMTP - - - -
        #Software: Microsoft Internet Information Services 6.0
        #Version: 1.0
        #Date: 2006-12-04 13:36:34

        It feels to me like it is the 160.83.65.200 that is the source of the problem, the other crashes have very similar charateristics - an EHLO/MAIL/RCPT/Crash as if it starts to send the DATA and that is where everything goes wrong.

        Stay tuned for further details at 11.00

        Comment


        • #5
          Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

          Originally posted by tkolber View Post
          Did you try disabling the Symatec AV for (just for test)?
          Actually, even better, I have two sites with the same problem, I will kill the AV at one tonight and leave it open on the other, see what happens.

          C

          Comment


          • #6
            Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

            I had a similar issue with AV, when a beta signiture crashed the server every time.
            CNE 5, CCA, MCSE NT4.0-2003, MCSE 2003 messaging, Exchange Server MVP.
            Tzahi Kolber - IT Supervisor
            Polycom Israel.

            Comment


            • #7
              Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

              The plot thickens

              An hour on, and who do we see in the log again?

              Services died at 01:39:35

              2006-12-04 14:39:16 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 EHLO - +vmx1.rbc.com 250 0 314 17 0 SMTP - - - -
              2006-12-04 14:39:16 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 MAIL - +FROM:<[email protected]> 250 0 45 42 0 SMTP - - - -
              2006-12-04 14:39:16 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 RCPT - +TO:<[email protected]> 250 0 36 33 0 SMTP - - - -
              2006-12-04 14:39:24 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 DATA - +<[email protected] pine.fg.rbc.com> 250 0 149 1299 7296 SMTP - - - -
              2006-12-04 14:39:24 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 MAIL - +FROM:<[email protected]> 250 0 45 42 0 SMTP - - - -
              2006-12-04 14:39:24 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 RCPT - +TO:<[email protected]> 250 0 39 36 0 SMTP - - - -
              2006-12-04 14:39:25 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 DATA - +<[email protected] pine.fg.rbc.com> 250 0 149 1305 421 SMTP - - - -
              2006-12-04 14:39:29 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 220+mx1.istrbc.com+ESMTP 0 0 24 0 4094 SMTP - - - -
              2006-12-04 14:39:29 142.245.33.100 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 EHLO - domainchanged.com.au 0 0 4 0 4094 SMTP - - - -
              2006-12-04 14:39:29 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250-mx1.istrbc.com 0 0 18 0 4422 SMTP - - - -
              2006-12-04 14:39:29 142.245.33.100 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 MAIL - FROM:<>+SIZE=2812 0 0 4 0 4422 SMTP - - - -
              2006-12-04 14:39:29 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250+sender+<>+ok 0 0 16 0 4953 SMTP - - - -
              2006-12-04 14:39:29 142.245.33.100 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 RCPT - TO:<[email protected]> 0 0 4 0 4953 SMTP - - - -
              2006-12-04 14:39:30 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250+recipient+<[email protected]>+ok 0 0 39 0 5375 SMTP - - - -
              2006-12-04 14:39:30 142.245.33.100 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 DATA - - 0 0 4 0 5375 SMTP - - - -
              2006-12-04 14:39:30 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 354+go+ahead 0 0 12 0 6078 SMTP - - - -
              2006-12-04 14:39:30 142.245.29.136 vmx1.rbc.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 QUIT - vmx1.rbc.com 240 15640 69 4 0 SMTP - - - -
              2006-12-04 14:39:31 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 250+ok:++Message+215759852+accepted 0 0 35 0 6547 SMTP - - - -
              2006-12-04 14:39:31 142.245.33.100 OutboundConnectionCommand SMTPSVC1 SERVERNAMECHANGED - 25 QUIT - - 0 0 4 0 6547 SMTP - - - -
              2006-12-04 14:39:31 142.245.33.100 OutboundConnectionResponse SMTPSVC1 SERVERNAMECHANGED - 25 - - 221+mx1.istrbc.com 0 0 18 0 6875 SMTP - - - -
              2006-12-04 14:39:32 160.83.65.200 imr8.us.db.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 EHLO - +imr8.us.db.com 250 0 313 19 0 SMTP - - - -
              2006-12-04 14:39:32 160.83.65.200 imr8.us.db.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 MAIL - +From:<[email protected]> 250 0 45 52 0 SMTP - - - -
              2006-12-04 14:39:32 160.83.65.200 imr8.us.db.com SMTPSVC1 SERVERNAMECHANGED 192.168.X.X 0 RCPT - +To:<[email protected]> 250 0 39 36 0 SMTP - - - -
              #Software: Microsoft Internet Information Services 6.0
              #Version: 1.0
              #Date: 2006-12-04 14:39:40

              Comment


              • #8
                Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                And thickens further.

                The server delivering the message that appears to be causing the above machine to crash is imr8.us.db.com.

                I have just discovered from the logs that the server causing the other machine to crash is loninmrp6.uk.db.com.

                I won't jump to any conclusions yet - could be spoofed IPs, or any of a million other problems.

                Comment


                • #9
                  Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                  Well guys, I turned off AV on one of the servers and waited the hour for the message to retry, here is the email that made it so angry

                  --------------------------
                  Return Receipt

                  Your document: FW: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

                  was received by: [email protected]

                  at: 01/12/2006 15:37:23

                  --------------------------

                  The read receipt was generated by a notes user at DB in response to a legitimate email. Who knows why that made Symantec angry, but I will be calling them tomorrow to ask - it has made me a little angry too.

                  Oh well, off to bed now, 4 hours of solid rest till I am back at it.

                  A

                  Comment


                  • #10
                    Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                    So, case closed?
                    Cheers,

                    Daniel Petri
                    Microsoft Most Valuable Professional - Active Directory Directory Services
                    MCSA/E, MCTS, MCITP, MCT

                    Comment


                    • #11
                      Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                      It is now - it was a Symantec Spam filtering problem - details at:

                      http://tinyurl.com/yhszuq

                      Thanks for listening!

                      Comment


                      • #12
                        Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                        I told you guys the Symantec products suck. Big time.
                        Cheers,

                        Daniel Petri
                        Microsoft Most Valuable Professional - Active Directory Directory Services
                        MCSA/E, MCTS, MCITP, MCT

                        Comment


                        • #13
                          Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                          Daniel,

                          I agree with your assessment of Symantec. However, what do you recommend in it's place for either SPAM or simply AV scanning on an Exchange 2003 server?

                          Chad

                          Comment


                          • #14
                            Re: Exchange &amp; IIS Services crash with 7031/7034 Errors

                            Definitely TrendMicro - wall to wall coverage (email, file scaning, spam, personal FW, spyware...):
                            http://www.trendmicro.com/en/home/us/personal.htm
                            CNE 5, CCA, MCSE NT4.0-2003, MCSE 2003 messaging, Exchange Server MVP.
                            Tzahi Kolber - IT Supervisor
                            Polycom Israel.

                            Comment

                            Working...
                            X