Reverese NDR attack – change IP/MX records

Home Forums Messaging Software Exchange 2000 / 2003 Reverese NDR attack – change IP/MX records

This topic contains 22 replies, has 7 voices, and was last updated by Avatar Klever 7 years, 9 months ago.

Viewing 23 posts - 1 through 23 (of 23 total)
  • Author
    Posts
  • Avatar
    NDR
    Member
    #159204

    It’s been quite a while since I monkeyed around with Exchange so keep that in mind with what follows.

    I recently inherited a split-site network for the next month or two, until they sort out a permanent in-house IT solution.

    There is a 100MB Ethernet link (BT Short Haul Data Services) joining the two sites.

    8 Windows 2003 Standard servers and one Exchange 2003 Enterprise box serve about 100 clients.

    Breakout to the internet is currently a leased line which is soon to be discontinued.

    ADSL at the main branch will take over – most traffic is LAN based file sharing and printing; internet use it very low, so ADSL should suffice at a fraction of the cost of the leased line.

    Last week the Exchange box was subject to what I think was a reverse NDR attack.

    I sorted the flooded queues in Exchange, triple checked that the server was not an open relay, ran various virus and spyware scanners and all the other usual nonsense we have to do when this happens.

    I also had to arrange de-listing from some blacklists on Wednesday last week.

    In this enlightened age though, Yahoo, Gmail, Microsoft et al run their own blocks and getting off their lists is a major event.

    Given that our current public (leased line) IP address is not blacklisted there ought to be no problems but mails are being rejected by Yahoo, Gmail, Microsoft, so something needs to be done.

    Migrating Exchange to the ADSL public IP seems a promising solution to deal with this – I wouldn’t normally consider it but since we were going to shift from leased line to ADSL anyway…

    So, long story short, I need to refresh myself and make sure the plan I have is sound.

    Reverse DNS has been requested from the ADSL provider and this will be in place by tomorrow.

    In no order of preference:

    • Change current A record to new public IP address
    • Add new public IP address(s) as MX records
    • Wait and think lofty thoughts while global DNS sorts itself out
    • Meanwhile, back at the ranch, stop the current Default SMTP Virtual Server after making a new one (could just tweak the existing one but I tend to do it this way)
    • Go though internal DNS and overcheck any records referencing the legacy public IP addresses
    • Configure new firewall on the DSL line with all required ports forwarded for Exchange, OWA etc

    I know the above is a gross oversimplification and not necessarily listed in order of priority, but can anyone see any screaming howlers that I might have overlooked ?

    If I pull the trigger on this late on Friday evening I should have all the kinks worked out by start of play on Monday morning so that staff can come in and get on as usual.

    Thoughts ?

    Avatar
    Virtual
    Member
    #334351

    Re: Reverese NDR attack – change IP/MX records

    Do you have any mail scanning in place externally or internally? settings such as recipient lookup can help with an NDR attack and activating Tar pitting.

    You can see this via the SMTP virtual server natively within Exchange as well, so worth considering.

    Avatar
    Klever
    Member
    #386225

    Re: Reverese NDR attack – change IP/MX records

    Thanks for your reply – the only mail scanning done at present by a corporate Kaspersky product (Total Space Security I think, from memory) and there was also an Untangle gateway in front of Exchange that was dealing with SPAM.

    Untangle died (hardware failure) and I intend shoving a pfsense box or something similar in for combined mail scanning, SPAM filtering and possibly proxying to replace it.

    I will check later today to see if there are any other elements I have missed but I think that’s about it. It appears to be a simple, even rudimentary setup.

    I’m in two minds whether or not to shove a inbound/outbound SMTP Mail Relay in place using one of the other servers as an additional layer between Exchange and the outside world. There is a member server and also a DC that both sit almost completely idle for most of each day, so there is scope without investment to do this.

    Looks like my weekend is going to be fun…

    EDIT:

    Just found that the Exchange server is configured for port 26 so that port 25 can be used by Mailwasher Enterprise…

    Avatar
    Lior_S
    Member
    #282966

    Re: Reverese NDR attack – change IP/MX records

    Well before you begin, (not sure if you did already or this is a plan) drop the TTL on all public DNS records that you will be updating. This will reduce the change time, when you actually need them to change quick, then raise them back up to a 1 day or so, once the drama subsides.

    Also if you SPF in place you might not have had to deal with the banning in the first place, since there is a mix of domain AND ip to get on the big mailers blacklists. Yahoo is the worst……

    EDIT: Woops never mind SPF wont help, got confused

    Avatar
    Sembee
    Member
    #260824

    Re: Reverese NDR attack – change IP/MX records

    If Exchange isn’t answering the inbound email from the internet, then you need to look at what is. Mailwasher was the cause of the problems, because by the time that Exchange would reject the email, it was too late.

    As a minimum I expect all antispam solutions to do recipient validation. If they do not, and they expect to answer SMTP traffic rather than Exchange (or whatever is hosting email) then they don’t even get considered. I have sites that drop 80% of email, where it is addressed to invalid recipients.

    Tarpit and Recipient Validation on Exchange can and does work, but Exchange needs to be answering the email.

    SPF records, as already corrected, would have zero effect on this. SPF records are for others receiving email from you, they do next to no good for your own incoming email.

    Simon.

    Avatar
    Klever
    Member
    #386226

    Re: Reverese NDR attack – change IP/MX records

    Mailwasher was installed post NDR attack, or so I am told.

    Prior to that the Untangle box was filtering SPAM with Exchange still on port 25.

    Untangle died (blown mainboard) the day after the reverse NDR attack.

    ‘Facts’ are a little thin on the ground since they have no onsite IT and everyone is pitching in with their version of what they think happened.

    To be frank, I am bouncing the idea around of putting a new Exchange box in by re-purposing an existing server, set it up from scratch properly and work with a known quantity, move the mailboxes and so on.

    I hate following along after a mess because you never really know what damage has been done, or where.

    I will be happier with Exchange back on port 25, Mailwasher uninstalled, an AD aware gateway device of some sort for SPAM filtering before the emails hit Exchange and the shiny new A record and MX record IPs resolving to the FQDN

    One plus about the coming weekend – there’s a great 7-day-a-week coffee shop two minutes away from where I’ll be tussling with this…

    ;)

    Avatar
    Sembee
    Member
    #260825

    Re: Reverese NDR attack – change IP/MX records

    I haven’t really found anything open source that can do recipient validation that I am happy with. If you really want to put something in front of Exchange, then IIS with a copy of Vamsoft ORF works well, although I will often run Vamsoft ORF directly on the Exchange 2003 server. I find it protecting most Exchange 2003 servers that I support and it works well – plus it is cheap.

    Simon.

    Avatar
    Lior_S
    Member
    #282968

    Re: Reverese NDR attack – change IP/MX records

    You could just do spamhaus zen alone in 2003, that should help. No cost at all…..

    Avatar
    Sembee
    Member
    #260826

    Re: Reverese NDR attack – change IP/MX records

    Lior_S;261165 wrote:
    You could just do spamhaus zen alone in 2003, that should help. No cost at all…..

    If you don’t mind someone unaccountable deciding what email you can receive…

    Simon.

    Avatar
    Lior_S
    Member
    #282969

    Re: Reverese NDR attack – change IP/MX records

    Sembee;261198 wrote:
    If you don’t mind someone unaccountable deciding what email you can receive…

    Simon,
    I was hoping you would say that :twisted:,

    Quote:
    From Vamsoft website:
    Employing the extremely effective combination of automatic tests like DNS blacklists and SURBLs (including Spamhaus ZEN and uribl.com), the reverse DNS test, the HELO/EHLO domain test, the SPF test and Greylisting……

    Vamsoft seems to use spamhaus as well. If you had an issue, how fast would they refer you to spamhaus to remove the blocked IP…? They seem to have a winning combination of spam filtering, I like that, just saying that RBL are probably the most effective first level spam control, and you can get that alone at no cost.

    I know you don’t like it, but you seem to be using it. :shock:

    Avatar
    Klever
    Member
    #386227

    Re: Reverese NDR attack – change IP/MX records

    Sembee;261164 wrote:
    … If you really want to put something in front of Exchange, then IIS with a copy of Vamsoft ORF works well, although I will often run Vamsoft ORF directly on the Exchange 2003 server. I find it protecting most Exchange 2003 servers that I support and it works well – plus it is cheap.

    Simon.

    Testing the product for an couple of hours or so today on my own server (the things we do to our own environments) and I’m very impressed on all fronts.

    Very lightweight, no noticeable impact at all on system resources, zero false positives so far and that price can’t be argued with.

    Thanks for the suggestion – this is definitely going into the box of tricks and I will probably look to install it this weekend on the previously mentioned Exchange server.

    A large part of the problem appears to be that Google, Microsoft, Yahoo, Sky and similar are ignoring blacklists in favour of their own ‘block’ lists.

    Having spent hours jumping through their respective hoops trying to get them to unblock this particular server the only reply I got came from Yahoo and it basically said no, go away and look at our bulk sender policy.

    All of this from a reverse NDR attack…

    The really annoying thing is that small companies who get tripped up like this may sometimes have almost nowhere to go for rapid resolution.

    As it happens this company has an ADSL line that only went live a week ago, so there is somewhere to shift things across to.

    BT have set up reverse DNS PTR for the two of the new public IP addresses on the ADSL line so this weekend should see things back to where they should be, more or less.

    I noticed some squirrelly internal DNS though – looks like some missing records are causing nslookup to fail on the Exchange box, so some additional jiggery pokery to do there.

    Thanks once again for the product recommendation – very much appreciated.

    Avatar
    Sembee
    Member
    #260829

    Re: Reverese NDR attack – change IP/MX records

    Lior_S;261202 wrote:
    Simon,
    I was hoping you would say that :twisted:,

    Vamsoft seems to use spamhaus as well. If you had an issue, how fast would they refer you to spamhaus to remove the blocked IP…? They seem to have a winning combination of spam filtering, I like that, just saying that RBL are probably the most effective first level spam control, and you can get that alone at no cost.

    I know you don’t like it, but you seem to be using it. :shock:

    Vamsoft CAN use blacklists, but you don’t have to use them, and I don’t. Almost every antispam product I have seen can use them.

    I do use the Blacklists that Vamsoft can produce for you, based on the email delivery attempts that are seen by your server, but I have 100% control over those.

    I have never been a fan of blacklists like Spamhaus and the like, simply because I don’t have control and if you get on one of their blacklists and they say “tough”, there is nothing you can do about it. For a mission critical app like email, that simply isn’t good enough.

    Simon.

    Avatar
    Sembee
    Member
    #260830

    Re: Reverese NDR attack – change IP/MX records

    NDR;261203 wrote:
    Testing the product for an couple of hours or so today on my own server (the things we do to our own environments) and I’m very impressed on all fronts.

    Very lightweight, no noticeable impact at all on system resources, zero false positives so far and that price can’t be argued with.

    Thanks for the suggestion – this is definitely going into the box of tricks and I will probably look to install it this weekend on the previously mentioned Exchange server.

    A large part of the problem appears to be that Google, Microsoft, Yahoo, Sky and similar are ignoring blacklists in favour of their own ‘block’ lists.

    Having spent hours jumping through their respective hoops trying to get them to unblock this particular server the only reply I got came from Yahoo and it basically said no, go away and look at our bulk sender policy.

    All of this from a reverse NDR attack…

    The really annoying thing is that small companies who get tripped up like this may sometimes have almost nowhere to go for rapid resolution.

    As it happens this company has an ADSL line that only went live a week ago, so there is somewhere to shift things across to.

    BT have set up reverse DNS PTR for the two of the new public IP addresses on the ADSL line so this weekend should see things back to where they should be, more or less.

    I noticed some squirrelly internal DNS though – looks like some missing records are causing nslookup to fail on the Exchange box, so some additional jiggery pokery to do there.

    Thanks once again for the product recommendation – very much appreciated.

    Run the product for a few days without actually blocking anything, but white listing outbound email. In my experience most users email the same core set of recipients all of the time. Therefore the auto white list will ensure that most of that email flows without interruption.
    I rave about this product quite a bit – check my blog for some of the results I have seen with it. There is a new version soon, which if you buy now you will get, a few more checks but the same core functionality.
    For most clients, Recipient Validation, DHA and then setting up honeypots based on the most common non-valid users has pretty much knocked out the spam.

    Simon.

    Avatar
    Lior_S
    Member
    #282970

    Re: Reverese NDR attack – change IP/MX records

    touché.

    But spamhaus is not like the rest. Its the only one I would ever use, 30 min TTL….

    The others can be a nightmare. Most notably sorbs, dont ever use them. They suck so much I want them to fail and go away. Unfortunately they are making loads of money by holding people hostage, they are not a reputable RBL and spoil it for the rest of them………but many use them :-x

    Anyway, NDR sorry for hijacking your thread a bit.

    Avatar
    Klever
    Member
    #386228

    Re: Reverese NDR attack – change IP/MX records

    Update:

    Everything went more or less as planned over the weekend and normal service was resumed by Sunday afternoon, ready for start of play this morning.

    Staff were dismayed, or so I am told, that they could get on with things and not use email server problems as an excuse or shield…

    One anomaly remains: Google is still bouncing messages but, curiously, it is bouncing them with the correct FWDN of the Exchange server, but the wrong (old) public IP.

    This is affecting GMail – several companies this client deals with host their services with Google.

    Sending to [email protected] is fine, but GMail or businesses who use Google’s hosted services are adversely affected.

    I spent hours last week jumping through the hoops involved in requesting de-list from Yahoo, Google and Microsoft’s ‘block’ lists. Yahoo said no (paraphrased), Microsoft said no, get your act together (paraphrased) and Google failed to respond at all.

    So, new public IP that was triggered on Friday evening and went globally stable yesterday and Google still bounces emails from the correct FQDN bound to the old public IP address.

    This probably just goes to show something…

    :?

    I am astonished that Google is (mis)handling emails like this, and that there is no method of addressing the problem with them.

    Avatar
    Ossian
    Moderator
    #186919

    Re: Reverese NDR attack – change IP/MX records

    As a rule of thumb, DNS changes take up to 48 hours to propagate around the internet, so maybe allow a little more time for googles dns servers to catch (or is it cache) up?

    Avatar
    Klever
    Member
    #386229

    Re: Reverese NDR attack – change IP/MX records

    The DNS update was triggered on Friday, so there should have been plenty of time between then and Monday morning for global propagation.

    Here we are at approaching close of play on Tuesday and they are still blocking our emails and the bounce message still details the correct FQDN with the old public IP address.

    Something else is definitely at work here – they appear to be caching information long past the point where the rest of the world has updated itself, and their systems don’t even both to check the public DNS in real time.

    Happy days.

    Other than that, everything is hanging together very nicely (famous last words).

    Avatar
    Klever
    Member
    #386230

    Re: Reverese NDR attack – change IP/MX records

    Emails were going through OK to [email protected] until a few minutes ago.

    Now the client is getting:

    Your message did not reach some or all of the intended recipients.

    Subject: test
    Sent: 10/07/2012 18:07

    The following recipient(s) cannot be reached:

    [email protected] on 10/07/2012 18:07
    There was a SMTP communication problem with the recipient’s email server. Please contact your system administrator.

    =======================================================

    That IP address (212.140.241.130) is the old one, which was changed on Friday. As I say, emails to *@googlemail.com were going through fine until a minute ago and now they too are being blocked, referenced by our old public IP.

    No kit is connected to the line that IP was on, although the BT leased line box our firewall was connected to is still powered on.

    This is becoming irritating – I can’t figure why old records would be used unless organisations are storing their own long-term cache that intercepts a FQDN regardless of whether it is on a different public IP.

    Our new IP is in a 213.x.x.x range.

    All tests I have run say we are not blacklisted, PTR records are OK for reverse lookup, not an open relay, blah, blah, blah.

    Stupmed !

    Avatar
    Sembee
    Member
    #260836

    Re: Reverese NDR attack – change IP/MX records

    ISPs cache a lot of things.
    Have the PTR records all been updated so that everything resolves correctly?

    Simon.

    Avatar
    Klever
    Member
    #386231

    Re: Reverese NDR attack – change IP/MX records

    Yes, the PTR records were set up by BT on the new IP address in advance of the domain DNS record changes. I had confirmation from them on Thursday last week.

    Domain A and MX records were changed/triggered on Friday evening.

    All tests against the domain from late Saturday through into Sunday morning onwards showed everything is as it should be, on its new public IP.

    Certain ISPs appear to be looking up to their own cached lists so I guess that means they are reacting to the FQDN of the server since the public IP point of origin has totally changed.

    Irritating because when you fill in the forms requesting Microsoft, Yahoo and Google to unblock your domain they all have required fields asking how many bulk mailers you send where your opt out page lives, where your opt-in privacy policy lives and so on.

    The client doesn’t bulk mail, they have no opt-in, opt-out or privacy policy because they only deal direct with medical tender account.

    There seems to be no way out of this other than to wait, and wait, and…

    Trouble is, many of their clients in the far and middle east use Google and Microsoft mail accounts so coms are seriously affected byt his at present.

    I suggested registering a second domain for temporary mail use. Send emails out from domain-2 with replies configured to come back into domain-1. Messy, but it might get them over this hump until the dust settles.

    Avatar
    Sembee
    Member
    #260838

    Re: Reverese NDR attack – change IP/MX records

    Have you checked that BT have done the work, rather than just accept their word for it? I wouldn’t trust BT more than I can see their van round the next corner.

    Send the email via a smart host – I think almost every client I have has an issue with one domain or another and we have to route via a smart host.

    Simon.

    Avatar
    Klever
    Member
    #386232

    Re: Reverese NDR attack – change IP/MX records

    Yes, BT have (surprisingly) done the work and the PTR record exists and correctly resolves.

    I was trying to avoid using a smart host, but it looks like this could be the only way forward.

    These things are sent to try us…

    Avatar
    cruachan
    Participant
    #330422

    Re: Reverese NDR attack – change IP/MX records

    If you do go the smart host route, you will need to contact BT again as they are one of the very few ISPs who require you to inform them before they will let you relay through their smart host. Usually takes 24 hours to setup.

Viewing 23 posts - 1 through 23 (of 23 total)

You must be logged in to reply to this topic.