Announcement

Collapse
No announcement yet.

Spamhaus, UCEProtect, CBL - are they worth it?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spamhaus, UCEProtect, CBL - are they worth it?

    I am a little (well, very much actually) annoyed about the way these reputation services work.

    We use MxToolbox to keep tabs on our website and email IP addresses. It monitors the services shown above plus many more and notifies us if any of our IP addresses have been blacklisted. We have three website IP's and one email IP. Our email IP has never been blacklisted.

    We have had two of our website IP's blacklisted a couple of times in the last 18 months. We employ a web-master to manage our websites and he updates them and troubleshoots any issues for us. When we receive a notification I contact him and ask him to check the affected site.

    We have been blacklisted 4 or 5 times in total. Once one of our websites had been hacked and was being used for phishing but it was the web host who identified it and shut the site down immediately. We cleared it up and had the site back online within a few hours.

    Apart from that the notifications we have received have said the blacklisted IP addresses were being used to pump out spam.

    Each time a 'spam' notification has been issued our web-master has not found anything wrong. He is very thorough and checks the number of files/folders is correct, dates and times etc. Additionally, the website hosts have very strict usage guidelines. Any hint of abuse and the site is immediately disconnected by the hosting provider (as happened with the hack). But, the web hosts have not detected anything regarding spam. It is quite certain that the blacklist is due to a 'false-positive'. I delist the address after the check has been completed.

    I have since discovered that when an IP is blacklisted it may be because one or more IP's in a range of addresses assigned to a provider have been pumping out spam and so the monitoring organisation takes it upon themselves to blacklist the entire IP range and thus blacklist IP's that are clean.

    I can't believe that anyone would use such a prehistoric method to deal with blacklisting an IP.

    I have also discovered that usually it is just the IP address that is put on the blacklist, and not the domain name which is reassuring. But, two years ago we were suddenly unable to send mail to a local government authority. All other mail was being received without a problem. I discovered that the authority was using a reputation list provided by McAfee and that this list included a single identification of one instance of spam being sent out from one of our website IP's four years previously. The identification was another 'false-positive' but McAfee or the original provider had linked our domain name to the IP and blocked mail from our domain name. It took three days of painful bureaucracy to fix this.

    Again, I find this incredible.

    I feel I am having to subscribe to a service that notifies me about errors caused by the blacklist providers, not identified by them.

    So, does anyone else have this problem? More importantly, are there professional reputation services that use intelligent methods to blacklist sites rather than knee-jerk responses. I don't know how I can effectively combat this. It is a waste of my time and the time of the people who have to deal with the fall-out from these incorrect blacklisting events.

    I do understand that overall these services provide a valuable service. I just think that they could implement their services far more intelligently.

    A recent poll suggests that 6 out of 7 dwarfs are not happy

  • #2
    Re: Spamhaus, UCEProtect, CBL - are they worth it?

    After about 18 months of being blacklisted I finally moved my email off my home Exchange Server and used a hosted web based service. It gave me the shits that they would blacklist my Domain Name when all they had to do was cross check the Domain Name with my Static IP and BINGO, the spam did NOT originate from the name.

    At the end I was Sending email via Google and Receiving it on my Exchange Server. It just got too frustrating with it only being for personal email. It was however really good for keeping some (a teenie weenie bit) skill level in Exchange. It was one of my Electives (Exchange 5.5) for my MCSE in 1999.
    1 1 was a racehorse.
    2 2 was 1 2.
    1 1 1 1 race 1 day,
    2 2 1 1 2

    Comment


    • #3
      Re: Spamhaus, UCEProtect, CBL - are they worth it?

      Yes - this is my point. IP addresses are unique. Our DNS Mail and @ records point to the single IP address we use to send and receive mail and it is not linked in any way to our website IP's - it's not even in the same range. But, when I look our MX reputation when a false-positive ID is made, it drops.

      I suspect the people who maintain these lists are simply lazy. If they are unable to implement intelligent interpretations to the ID's - which, as you point out is very simple - they should leave the monitoring to others who will.
      A recent poll suggests that 6 out of 7 dwarfs are not happy

      Comment


      • #4
        Re: Spamhaus, UCEProtect, CBL - are they worth it?

        They certainly are lazy (or overprotective). I once had a static IP from a block that used to be dynamic (dial up) and was blacklisted based on past history of the address - it was never resolved so I had to use a smart-host for that client
        Tom Jones
        MCT, MCSE (2000:Security & 2003), MCSA:Security & Messaging, MCDBA, MCDST, MCITP(EA, EMA, SA, EDA, ES, CS), MCTS, MCP, Sec+
        PhD, MSc, FIAP, MIITT
        IT Trainer / Consultant
        Ossian Ltd
        Scotland

        ** Remember to give credit where credit is due and leave reputation points where appropriate **

        Comment


        • #5
          Re: Spamhaus, UCEProtect, CBL - are they worth it?

          Allow me to provide a contrary opinion on this matter.

          First, let me say that there has been and still are some less-than-fully-professional RBL services out there. For instance, one service that shall not be named insists on blacklisting mail servers using IP addresses without PTR records. Not only are such records not required by any standard, but the presence or absence of such records says absolutely nothing about whether a mail server is likely to send spam or not. That type of filtering is obviously counterproductive, as it will result in a lot of false positives.

          Neither SpamHaus or SpamCop have ever done anything like the above. Their blacklists are well-documented and based on good data and sound reasoning. I've used one or both of these RBL providers on every mail server I've ever installed, and in 15+ years I have yet to experience any real issues with the quality of their services.

          Yes, I've certainly had to deal with blacklisted mail servers on numerous occasions, and of course the customer was up in arms about their mails being rejected by other servers using these blacklists. However, in absolutely none of these cases were the RBL providers to blame for the blacklisting.

          Originally posted by Blood View Post
          We have had two of our website IP's blacklisted a couple of times in the last 18 months. We employ a web-master to manage our websites and he updates them and troubleshoots any issues for us. When we receive a notification I contact him and ask him to check the affected site.

          We have been blacklisted 4 or 5 times in total. Once one of our websites had been hacked and was being used for phishing but it was the web host who identified it and shut the site down immediately. We cleared it up and had the site back online within a few hours.
          That's an excellent example of blacklists working exactly as intended. Your web server must have had serious security issues, and you should be asking your webmaster and/or server administrator some hard questions. Things like that don't just happen.
          Originally posted by Blood View Post
          Apart from that the notifications we have received have said the blacklisted IP addresses were being used to pump out spam.
          OK...
          Originally posted by Blood View Post
          I have since discovered that when an IP is blacklisted it may be because one or more IP's in a range of addresses assigned to a provider have been pumping out spam and so the monitoring organisation takes it upon themselves to blacklist the entire IP range and thus blacklist IP's that are clean.

          I can't believe that anyone would use such a prehistoric method to deal with blacklisting an IP.
          I certainly can.

          It seems your organization is using an ISP or VPS provider that takes a laissez-faire approach to spammers. A number of such providers exist, particularly in the low-cost end of the spectrum. They make money by allowing spammers to buy services, and when the complaints come in, they close down the spammers accounts after a while, but do nothing to prevent the same spammers to re-register and purchase new services with new IP addresses.

          These providers are the scourge of the Internet, and the only way to deal with them is to block their entire IP range. If your provider gets this treatment, there's a reason for it. If you don't want to become collateral damage in the war against spam, switch to a provider with a more reasonable anti-spam policy.
          Originally posted by Blood View Post
          I have also discovered that usually it is just the IP address that is put on the blacklist, and not the domain name which is reassuring.
          Domain names are only blacklisted when they're used to host services related to illegal activities, such as (fake) websites pushing malware.
          Originally posted by Blood View Post
          But, two years ago we were suddenly unable to send mail to a local government authority. All other mail was being received without a problem. I discovered that the authority was using a reputation list provided by McAfee and that this list included a single identification of one instance of spam being sent out from one of our website IP's four years previously. The identification was another 'false-positive' but McAfee or the original provider had linked our domain name to the IP and blocked mail from our domain name. It took three days of painful bureaucracy to fix this.
          McAfee hasn't been considered a professional provider of, well, anything for at least a decade. Even John McAfee has done his best to distance himself from their products, and that's a guy who openly admits to using drugs and bribing 3rd world officials.

          For some reason, Intel recently bought McAfee. Let's hope they clean up their products.
          Originally posted by Blood View Post
          So, does anyone else have this problem?
          A lot of people claim that RBLs have given them numerous problems. However, every time I've looked into such reports, the problem has turned out to be with the ISP, the VPS provider or the customer complaining about the RBLs.

          For instance, if your IP address is on a list of dynamic IP addresses when the ISP has actually reassigned that range to customers with static addresses, it's perfectly possible that the ISP has neglected to update the database.

          (I actually ran into that exact problem once, and all it took to fix it was a phone call to the ISP and a mail to SpamHaus. The RBL providers have to rely on reports from spamtraps and data in WHOIS; they aren't mind-readers.)
          Originally posted by Blood View Post
          More importantly, are there professional reputation services that use intelligent methods to blacklist sites rather than knee-jerk responses.
          SpamHaus and SpamCop certainly use intelligent methods. If you're affected by their blacklists, chances are there's a very good reason for it.

          Comment


          • #6
            Re: Spamhaus, UCEProtect, CBL - are they worth it?

            Thanks a lot for taking the time to reply.

            First - No 'spam' infections or evidence of spamming activity was found on any of these occasions when checked after one of our IP's had been blacklisted. Secondly, the phishing incident was identified by our hosting provider and not by any of the blacklisting organisations (the affected IP address was not blacklisted).

            We use Heart Internet to host our websites. This is the same provider that immediately shut down the website. I have to admit that I have no idea how reputable they are but based on our own experience of how seriously they take security, and how helpful they have been with all aspects of their service I have no reason to suspect they are slack.

            I understand the thinking behind blacklisting an entire IP range, but I still regard it as prehistoric. It adversely impacts the innocent. My view is that it is not for the blacklisting organisations to 'police' these services, instead it should be up to the services one level above the providers, or those at the top of the chain who assign the IP addresses in the first place. It is their responsibility to ensure the organisations who manage these services are up to the task.
            A recent poll suggests that 6 out of 7 dwarfs are not happy

            Comment


            • #7
              Re: Spamhaus, UCEProtect, CBL - are they worth it?

              One thing I did find was that a site would Blacklist me and others would blacklist based on the first site's results. (NOT including SpamHaus and SpamCop in this though).

              One site wanted 105 Euros to remove me from their listing. Barracuda were the first to blacklist me but fortunately they do have a tool on their site where you can get your Domain Name removed. The ones that want money are just as bad a the spammers themselves IMO.
              1 1 was a racehorse.
              2 2 was 1 2.
              1 1 1 1 race 1 day,
              2 2 1 1 2

              Comment


              • #8
                Re: Spamhaus, UCEProtect, CBL - are they worth it?

                Yes, while I was researching this I came across a site that allowed you to add your IP to a whitelist - and they charged a fee. If an entire IP range was blacklisted and your IP fell within that range, it would be automatically delisted. If your IP was proven to be a spam factory then your IP would be removed and would never be allowed on the whitelist again. But what about FP's? How would they determine when a FP had occurred? Would they believe me if I said, hang on - there is no evidence of any tampering/infection at all?

                As I said, I think there is value in these services, but I also think their modus operandi could benefit from a major rethink.
                A recent poll suggests that 6 out of 7 dwarfs are not happy

                Comment


                • #9
                  Re: Spamhaus, UCEProtect, CBL - are they worth it?

                  Originally posted by Blood View Post
                  First - No 'spam' infections or evidence of spamming activity was found on any of these occasions when checked after one of our IP's had been blacklisted. Secondly, the phishing incident was identified by our hosting provider and not by any of the blacklisting organisations (the affected IP address was not blacklisted).

                  We use Heart Internet to host our websites. This is the same provider that immediately shut down the website. I have to admit that I have no idea how reputable they are but based on our own experience of how seriously they take security, and how helpful they have been with all aspects of their service I have no reason to suspect they are slack.
                  As I said, there are bad actors trying to profit from spam by providing DNSBL services of poor quality or, as you mentioned in another post, by charging a fee for putting IP addresses on a 'whitelist'. If you were affected by any of those lists, the fault lies with the operators of the receiving mail servers for using services from shady companies that are no better than the spammers themselves.

                  Having said that, most network and server admins using DNS blocklists do indeed take the time to evaluate the quality of a blocklist before including it in their spam filtering strategy. If you find that you're unable to send e-mails to a significant portion of Internet users due to blacklisting, you're probably listed by at least one of the three major DNSBL providers, which are SpamHaus, SpamCop and SORBS.

                  Now, consider how an IP address might end up on one of those lists:
                  • If multiple spam e-mails are sent from your IP address to one the DNSBL provider's spam traps, the address will be placed on a list of known spammers for at least 48 hours
                  • If multiple automatic spam reports are received from multiple spam filtering gateways at different organizations (usually more than 10), the address also gets placed on the 'known spammers' list for a specific period of time (again, 48 hours if the commonly used interval)
                  • If a scan reveals the IP address to be hosting an open SMTP relay, it ends up on the list over open relays until a subsequent scan indicates the opposite
                  • If a server is found to be hosting an open web or SOCKS proxy service, it ends up on the 'open proxy' list
                  • If an IP address hosts a web site advertising or selling products that are marketed through spamming, the address (and occasionally also the domain name) is put on a blocklist
                  • If a web server is hosting malware, it ends up on a blocklist
                  • If traffic generated by malware if found to be originating from an IP address, the address is added to a list of infected systems
                  • If WHOIS information indicates that an address range is used for dynamic IP allocation (and as such should never be hosting legitimate SMTP services), the range is added to a 'network policy' block list
                  • If the DNS records for an address or a domain are found to be invalid (MX records pointing to IP addresses instead of hostnames; PTR records pointing to non-existent hostnames; an invalid e-mail address in the SOA record; A records pointing to invalid IP addresses), a domain or IP address may be added to a blocklist
                  • And finally, if a service provider is found to be actively and persistently assisting spammers in trying to get around any of the measures above, IP addresses allocated to that service provider may be manually added to a 'known spammers' list

                  Since SMTP (and HTTP/HTTPS) traffic is TCP based and thus cannot be used from a spoofed IP address, how would the IP address of a legitimate, non-spamming, non-malware-spreading organization end up on one of those lists? Unless they got an address or address range that was already blacklisted due to the actions of a previous owner, I don't really see how that is possible.

                  I'm not saying that incorrect blacklisting by a reputable DNSBL provider is absolutely impossible and cannot ever occur, because obviously no system is perfect and 100% bug free, but as I said: In 15+ years as a consultant and network admin I've yet to see a case of that actually happening.
                  Originally posted by Blood View Post
                  I understand the thinking behind blacklisting an entire IP range, but I still regard it as prehistoric. It adversely impacts the innocent. My view is that it is not for the blacklisting organisations to 'police' these services, instead it should be up to the services one level above the providers, or those at the top of the chain who assign the IP addresses in the first place. It is their responsibility to ensure the organisations who manage these services are up to the task.
                  I and others use DNSBL providers because we want them to identify likely sources of spam. It is definitely their job to 'police' these services, just as it is the job of anti-virus vendors to 'police' the spreading of malware.

                  Senders of spam or malware are often 'innocent' in the sense that they are completely unaware that their PCs or servers are being used by spammers, or that their service provider is allowing spammers to play musical chairs with IP addresses in their range, but if you're operating a server on the Internet, you can't really afford to be ignorant of these issues.

                  As for Tier 1 providers, they are already actively assisting in the fight against spam and malware by filtering malicious network traffic. However, they do not concern themselves with the content of the packets, only with traffic which is clearly disrupting legitimate IP services, such as (D)DoS attacks.

                  In other words, they stick to 'policing' Layer 3 traffic, which I believe is a very sensible approach for a provider of high-bandwidth Layer 3 connectivity. They will only block low-volume traffic if they receive reports of malicious activity from peering partners or downstream ISPs (their customers), or if government agents turn up on their doorstep with a court order or one of those infamous National Security Letters (in the U.S. that is).

                  Comment


                  • #10
                    Re: Spamhaus, UCEProtect, CBL - are they worth it?

                    Thanks a lot for that list. This is the type of info I was trying to discover when researching this.

                    My frustration lies with the fact that all blacklist notifications have effectively tarred us, and presumably other innocent parties, with the same brush.

                    I do appreciate your input - I have learnt a lot more and understand the conditions for blacklisting much better than I did.
                    A recent poll suggests that 6 out of 7 dwarfs are not happy

                    Comment


                    • #11
                      Re: Spamhaus, UCEProtect, CBL - are they worth it?

                      I understand your position, I really do. But the fact is that it should be almost trivially easy for anyone to avoid ending up on a realtime blocking list:
                      • Make sure your public DNS records are valid.
                      • And speaking of DNS, make sure your domain has SPF records listing the IP addresses of your SMTP servers. That will prevent other spammers from using your e-mail addresses in spoofed "From" fields, something which in turn could cause some rubbish filtering systems to incorrectly block e-mails from your domain.
                      • Don't just scan inbound mail. You should run outbound e-mails through a spam filter as well, to catch mails from infected client systems.
                      • If you're NATing clients behind the IP address of a router or firewall, block outbound SMTP on port 25 from any internal address other than that of your mail server. This is particularly important if you use the same public IP address for both NAT overloading and outbound SMTP.
                      • Consider routing outbound e-mail through your ISP's Smart Host. They will typically have their own scanning/filtering system, which you then get to use at no extra cost.
                      • Log outbound SMTP traffic (ports 25 and 587); don't rely on the server logs alone. NetFlow is great for this, but if your router doesn't support NetFlow, a mirror/monitor port on a switch and something like Snort or NTOP will do just fine.

                      Sure, even if you do all this, you may still encounter scenarios where a client or partner organization has made a particularly poor choice in selecting a mail filtering service or product and as a result, your mails end up being blocked for absolutely no good reason. That can usually be fixed pretty easily by politely requesting that the server operator on the receiving end add your server to a whitelist. In any case, such corner cases will only affect mails to one specific organization or user.

                      Comment


                      • #12
                        Re: Spamhaus, UCEProtect, CBL - are they worth it?

                        Thanks.

                        Our email is managed in-house. We use a third-party spam/virus/etc filtering service for all incoming/outgoing mail so our MX records point to them and only to them. We have spf records and, where required, txt records setup. Our Mail and @ records point to our office IP so that our office IP is identified as a legitimate source of email for the domains we manage.

                        Our mail server denies relaying except from known IP addresses (we relay mail sent via Google Mail business accounts through our mail server so we have a copy), and other external SMTP connections must be authenticated.

                        We keep full logs of all SMTP, POP3 and IMAP sessions so I can trace any problems.

                        So, it's just our own office IP address that sends/receives mail for our domains. Mail has not been sent via any of our web/domain host's IP's for about 7 years now.
                        A recent poll suggests that 6 out of 7 dwarfs are not happy

                        Comment


                        • #13
                          Re: Spamhaus, UCEProtect, CBL - are they worth it?

                          Slightly belt and braces, but in the past using firewalls like ISA Server/TMG, it's part of the default setup of a mail serverf to only allow SMTP traffic from the IP address of the mail server outbound. Most firewalls (IME) allow all traffic outbound by default.

                          Blocking 25 outbound prevents any BOTs that may be on infected machines from sending out, and it's also simple to trace them from the firewall logs.

                          Spam coming from external IP addresses that then get blocked is, again IME, very rarely being sent through the mail server but almost always from infected PCs so locking down the outbound firewall rules is one of my default steps these days.
                          BSc, MCSA: Server 2008, MCSE, MCSA: Messaging, MCTS
                          sigpic
                          Cruachan's Blog

                          Comment


                          • #14
                            Re: Spamhaus, UCEProtect, CBL - are they worth it?

                            Thanks.

                            That's more valuable advice.

                            Our firewall is set to block by default so nothing gets out unless I configure it.
                            A recent poll suggests that 6 out of 7 dwarfs are not happy

                            Comment


                            • #15
                              Re: Spamhaus, UCEProtect, CBL - are they worth it?

                              Was the blacklisted IP address your own, or one used by the filtering service?

                              Would an outbound SMTP connection from an internal (and possibly infected) PC be blocked by your router/firewall? Would the attempt, successful or not, be logged?

                              If the security of your mail server was somehow compromised and someone were able to install a bulk e-mail client with its own SMTP engine, would you be able to tell what was going on?

                              A little over a year ago, a client experienced a security incident where an unsecured account was used by an outsider to log in to a local server. (Specifically, the "ftp" account had a valid shell and a simplistic password, when it should have had neither.) As a result, an outsider was able to install bulk e-mailing software, which he or she then used manually by logging in at irregular intervals to send out a few million spam e-mails.

                              Each time the deluge of outbound spam had stopped by the time the local admin became aware there was a problem due to the server suddenly being blacklisted. He didn't see anything in the SMTP logs (since the local SMTP service wasn't actually involved) and also didn't spot the software in the ftp home folder, so for weeks this client insisted they were the victims of repeated, baseless blacklistings by incompetent/malicious blacklist providers on the Internet. As you can see, that turned out not to be the case.
                              Last edited by Ser Olmy; 24th April 2015, 16:50.

                              Comment

                              Working...
                              X