Common ISP Mistakes

This page lists some common mistakes made by ISPs (or other network administrators) which cause other people grief. Some of these apply to backbone providers, some apply to non-backbone providers, and some apply to enterprise/organization network administrators. You may want to use this as a guide towards "best practice".

Some of the sections discuss things which you can do wrong and some suggest things which you can do right. The two are often not clearly separated; I trust people will be able to distinguish the two. A Little more consistency would be nice.

Advertising routes to 0.0.0.0/0 via BGP.

Blocking ICMP

Breaking ping is a bad thing, only to be done in emergencies.

Breaking PMTU discovery has the effect that connections between any two hosts that have an MTU greater than that of the smallest MTU hop on the path will not be able to communicate. Packets will be dropped, consistently enough to prevent communications even though some small part of each flow will frequently get through (generally the same part). This is a pattern (packet size) sensitive packet loss not a random one so TCP retransmission does not recover. For SMTP, the HELO, MAIL, RCPT dialog will happen and then the connection will hang on the message DATA (unless the message is very short) tying up the servers at both ends until they timeout. Normally all attempts to resend the message will also fail.

For HTTP, the browser will probably be able to do a "HEAD" (short response) but a "GET" will fail. The symptom to the user is that they are consistently unable to get through to certain web sites with the connection stalling at the same place each time. On very rare ocassions, I have seen a connection actually succeed to one of these unreachable servers when it was sufficiently loaded that it transmitted the data in smaller chunks.

If this happens to you, a workaround is to set the Max MTU to 576 on each of your clients (to fix outbound connections) and servers (to fix inbound ones). Setting the Mtu on your router does not seem to work (it might help in one direction (of data flow), by sending ICMP too bigs to your inside hosts at a threshold lower than the lost ICMP too bigs, but not the other direction in which both sets of "too bigs" get dropped. This does put a higher packet load on the backbone (more smaller packets have to be routed). The cause is usually because a router somewhere drops packets which are too large but have the DF (don't fragment) bit set without generating the required ICMP too big message (there is some defective hardware out there) or, more likely, that some cluelesss network operator filtered all ICMP traffic, usually as a naive attempt to protect against real or anticipated DoS attacks.

This is made much worse by filtering software which does not allow you to filter specific ICMP types and (unfragmented) packet sizes. A much better way to handle most of the DoS attacks (except smurf) is to force fragment reassasembly on a router which is not sucseptible before forwarding the packets; this puts a significant load on the router so it is best done on a router/firewall close to the systems being protected (which is also desireable because it gives more protection).

Note that PMTU can be broken at least three different ways, if you are suffering from it: Failure to generate ICMP "too big", filtering ICMP "too big", and configuring equipment which handles traffic which may come from the public internet with private addresses on the routers.

Marc Slemco has a page on Broken PMTU Discovery.

Using defective hardware that drops oversized packets without sending ICMP "too big" messages

Some "repeaters" which bridge incompatible (from an MTU perspective) media types (like fddi to/from ethernet) are guilty of this offense. A repeater is generally a brain dead device which simply cannot generate ICMP messages. These devices should be avoided. They might be usable in certain carefully controlled situations (where every single device on the larger MTU side can be, and is, programmed with the smaller MTU.

Filtering Broadcast addresses incorrectly

Failure to filter traffic with forged IP source addresses

On some cisco's, you may find the "ip verify unicast reverse-path" command to be helpful. On almost any box which has IP filtering capability, you can filter out packets from any interface which do not match the IP address range assigned to that interface.

Blocking traceroute

Don't allow your downstream networks to be used as smurf amplifiers

Configuring router interfaces with Private IP Addresses

Reassembling packet fragments

RFC-1812 specifically forbids a router from doing this because the fragments may take different routes and therefore not be all be availible for reassembly, thus causing packet loss which may be pattern sensitive and not random and therefore not fixed by retransmission. However, RFC-1812 was clearly written taking into consideration only the dubious performance reasons for doing this. The security reasons are far more compelling and I think RFC-1812 will need to be rewritten in the future to permit reassembly under certain limited situations, namely: A) where the router performing reassembly is on the only possible route into or out of the network being protected, or B) where steps have been taken to force all fragments from any given packet to converge at a common point for reassembly (be carefull that you don't inadvertantly cause fragmented packets to appear to have originated from inside your protected network in the processs).

Failure to read RFC-1812

Failure to document any deviations from RFCs

Failure to properly delegate reverse address mappings

Some ISPs do not know how to properly delegate reverse address mappings to a downstream customers domain name server for network blocks smaller than /24. The O'Reilly and Associates DNS book devotes several pages to how to do this. This failure interferes with the downstream networks administration of their address space.

Failure to prevent SPAM originating within or relaying through your net

Enforce your acceptable use policy.

Failure to have, and enforce, an Acceptable Use Policy

Require proper identification for all prospective network customers (both dedicated circuit and dailup). Require all prospective customers to sign a statement that they will abide by the AUP and that they have never had access terminated before for any type of network abuse. At a minimum, run all credit cards through with name/address verification before granting access. Check all credit cards, names, addresses, and phone numbers against known abusers. Verify that the prospective customer can actually be reached at what they claim is their phone number. Publish the name, address, phone number, and the MD5 hash of the credit card number used by any spammer to the net-abuse newsgroup.

Although AOL does enforce its AUP after the fact (at least by terminating accounts, if not taking perpetrators to court), it is their promiscuity in signing up customers that leads to so much spam from their network.

If you do not want to bother with the hassle of suing perpetrators to recover damages, at least assign your rights to collect damages to a law firm which will sue on their own behalf. Write your AUP to make lawsuits lucrative by establishing minimum damages.

Your AUP should also allow you to monitor suspected net-abuse. If someone sends 1000 copies of an email message, you should be able to read it to see if it is SPAM (if they are sending 1000 copies, it probably isn't that private anyway).

Your AUP should differenciate from unauthorized security attacks and authorized security attacks by reputable security consultants or the target networks owner themselves.

Transparently proxying traffic without consent

• Server as a web accelerator for downstream customers who have explicitly requested in.
• Improve performance for downstream customers provided you have notified them of the transparent caching and provided a means (such as a web form) by which they can configure this on or off at will for their accounts.
• You are providing an informative error message for connections to which you would otherwise deny service silently.
• You can force caching of your downstream customers outgoing web requests without option to disable only if you are explictly selling substandard service.

Users of transparent caching should also beware that they can only legitimately do so if all tcp packets for a connection must travel through one point; otherwise, you have the same kinds of problems as with fragment reassembly.

Making changes without notifying affected parties

• Filter traffic to/from their network in any way.
• Deviate from any RFC on any of your routers, particularly RFC-1812
• Change the point of attachment for their connection.
• Change any of the above

Failure to adequately monitor for net abuse

• SPAM
  Any host or account on your net which sends excesive mail (total number of recipients for all messages) during a given period of time should trigger an alarm. In response to that alarm, you can investigate if it is spam or a legitimate mailing list.
• Ping of Death and other similar DoS
• Smurf
  Monitor for inbound attacks, outbound attacks, and attacks which are using your network as an amplifier.
• Any forged source address packets

Failure to document any deviations from RFC's

Failure to consider who is affected by your countermeasures

If you have to block some traffic, it is also a good idea to consider redirecting http (and possibly other common protocols like SMTP and FTP) to protocol specific servers which return suitable error messages referring people to "http://accesssdenied.your.net/". This minimizes the time wasted by people who may not be at fault, cuts down on calls to your tech support lines, and directs hostility towards those who are at least partially responsible. Allow all http traffic to the minimalist webserver "accessdenied.your.net", which does not necessarily need to be inside your own network; the unix program "/bin/cat /path/accessdenied" (which contains http headers+html text) might even be sufficient if it is invoked from a inetd substitutes with process fork limits. Similar programs may suffice for ftp and smtp access servers. Linux boxes (and probably *BSD boxes as well) have the capability to redirect blocked traffic to a specified port on the firewall, where you can have a protocal specific error server. Here is a sample error message file, asuming your 554 is a suitable error number for the protocol and it uses smtp/ftp style messages (http and imap need a little change).

554- Access has been blocked at your.net. to/from certain networks 
554- because of netabuse or security violations originating at
554- or aided by the blocked networks.
554- For further information, including where to direct your complaints,
554  visit http://accessdenied.your.net

Failure to adequately control your upstream links

Alternatively, you can install your own router on your upstream providers end of your upstream links. In some cases, this will require that you have a direct router to router connection so your upstream can protect against any shenagans (promiscuous mode, masquerading, etc) from the router under your control.

These measures allow you to (possibly automatically) stop hostile traffic (such as smurf echo replies) before it clogs your upstream link.

Another possibility is to get your upstream provider to implement a secure http based server which allows you, and only you, to program certain characteristics, and only those characteristics, of your link. Filter rules (except those which prevent you from sending/forwarding traffic with spoofed source addresses).

Check suspected abuse against known good guys

Block spammers web sites

Try to verify that the owners of the web site were the ones who really sent the spam (i.e. it wasn't a frame). Posing as an interested customer is usually helpful in this case.

Subscribe to vixie's validated spam blocking BGP feed

Don't Shoot the messenger

Security isn't important here

• Your site, once compromised, may be used to attack other sites. You may even be liable if your site is used to harm others because of your negligence. Your connectivity to all or parts of the internet may be terminated if you cannot control access and stop abuse.
• How would your organization be affected if all the computers were unavailible or malfunctioned simultaneously? Particularly at a crucial time? How would you feel if you couldn't pay your rent because someone crashed all the machines in payroll? If you couldn't meet your own deadlines because someone crashed your machines?
• How many people use secure services from within your net? How many people order books online with a credit card, for example, from your campus computers?
• You may have much more confidential information than you realize. How would your customers react if their credit card number were stolen off of your online store? How would your students react when a list of the self help books they checked out from the campus library was made public?
• Do visitors to your campus log in to their home systems from your network?

Failure to engineer networks with a sniffer resistant topology

No host in your web farm, for example, should be able to evesdrop on traffic to/from any other host (particularly those owned, operated, or controlled by a different organization).

Consider setting up a trace host

Or, consider uploading a report from your abuse monitors to your web server every 5 minutes. This report should include clear abuses only to protect privacy.

24x7 NOC or staff on call

A 24x7 answering service is not a 24x7 NOC.

Rapid response is important for tracking down and stopping many types of net abuse.

Educate your network customers

Make sure your dialup customers understand they must not do business with spammers. Make sure they understand that spammers often try to make untargetted unsolicited email look like targetted mail; just because you actually are interested in the products being offered doesn't mean they didn't send the advertisement to 65 million other people who weren't.

Cooperation

NOCs need to be responsive to queries from other NOCs.

The preferred solutions to many problems require a significant development effort. Network providers can cooperate by sharing internally developed tools with others or by organizing with others to fund development of tools which meet a common need. Tapping the network infrastructure fund might be appropriate for some of these tools.

Net abuse is one of the areas where the old maxim "If you aren't part of the solution then you are part of the problem" applies.

Failure to support required or recommended email addresses

Three very common reasons someone might abandon their local ISP Ok. This particular section is theroetically a bit self-serving (because you might outsource some work to me). But failure to outsource where appropriate is one of the primary reasons that most small networks are likely to go the way of the dodo bird, driven out of business by large vertical monopolies with better economies of scale. This is something I don't want to see happen.

You do of course need to use secure communication channels when you allow contract personel to remotely administer some of your systems.

Alternate message submission port

Mail servers used for outgoing mail (with authentication) should also accept connections on port 587 as specified n RFC-2476. This allows people who are connecting from ISPs that block port 25 to connect to the mail server. Esentially, message submission is not considered separate from message transport. However, the mail server accepting conections must authenticate using some method method (POP3, SASL, to prevent the SPAM relaying and spam zombies that blocking port 25 was used to prevent in the first place.

Inbound SPAM and virus filtering

Many techniques are availible for filtering SPAM. ISPs should implemnt some of these solutions but should allow users to configure which filters will be applied to their mail.

• White Listing
• Black Listing
  Various black lists are availible using DNS queries. These allow filtering of messages originating from known open relays, dialup pools, ISPs who do not conform to RFCs, etc.
• Grey Listing
  By simply temporarily rejecting mail the first time it is received from a new sender, significant amounts of spam can be rejected (since most spammers don't automatically retry) at the expense of delaying incoming messages. One situation where these delays are anoying is when signing up for new accounts with email verification.
• Statistical filters (such as bayesian filters)
  These can be quite effective but require training.
• SPF(Sender Policy Framework) allows the envelope sender address to be checked against a list of hosts allowed to send for that domainname. The lookup is done via DNS TXT records published by the domain.
• virus checkers

Proposal for user based IP filtering

When a user logs in to a leaf node router (or is autoconnected), the radius authentication server should send back a list of account specific filtering rules for outbound traffic from that host.

Certain actions such as sending to SMTP email (port 25), would be restricted to impede spammers and spammer zombies but would be enabled (without fee) on request for those who need them.

All BLOCKs must be clearly spelled out on the ISPs website.

Many ISPs outsource the task of actually providing POPs to POP providers (who serve other ISPs as well). This is one of the reasons that ISPs don't enforce rules properly. Another is the fact that some users need fewer restrictions. However, as outlined here, the authentication server can configure filtering and monitoring on a per port basis customized to the account using that port at any given time.

These rules are in a ficticious easy to read syntax

    [OUTBOUND]
    // Unless user has SEND_RESERVED priviledge enabled:
    // RFC1918 + loopback network
    // security professionals would need this priviledge
    rule action=block sourceaddr==192.168.0.0/16
    rule action=block sourceaddr==10.0.0.0/9      
    rule action=block sourceaddr==172.31.0.0/16      
    rule action=block sourceaddr==127.0.0.0/8

    // Unless user has SEND_OTHER_IP priviledge enabled
    // security professionals would need this priviledge
    // multihomed networks with forwarding would need this priviledge
    rule action=block sourceaddr!=THISPORT_SUBNET
    rule action=allow

    // allow sending email through isp mail servers
    rule action=allow destport==25 mail.isp.net

    // Optional, if user has configured other SMTP hosts
    rule action=allow destport==25 mail.myemployer.com

    // Unless user has SEND_PORT_25 priviledge enabled
    rule action=block destport==25 
    // If user has SEND_PORT_25 enabled
    rule action=allow snoop=snoop_server destport==25

    [INBOUND]
    rule action=block sourceaddr==192.168.0.0/16
    rule action=block sourceaddr==10.0.0.0/9      
    rule action=block sourceaddr==172.31.0.0/16      
    rule action=block sourceaddr==127.0.0.0/8
    rule action=block destaddr==THISPORT_BROADCAST

    // If user has DENY_INBOUND_TCP
    // insert exceptions here
    rule action=allow protocol==tcp && established
    rule action=block protocol==tcp

    // If user has DENY_INBOUND_UDP
    rule action=allow protocol==udp dns-reply
    rule action=block protocol==udb    

    // default 
    rule action=allow

Cisco calls this "RADIUS Centralized Filter Management". Other router brands may have different implementations or no implementation. If the router has no implementation, then you need a hacked authentication server that can log in to the administrative account on the router (through a proxy if the router is owned by another company). As an ISP, if your hardware or POP provider does not support these things, you have a responsibility to bring pressure to bear to having these things implemented.

Note that since each port has its own temporary ruleset, traffic is not slowed down significantly by adding rules.

Control Panel

ISPs should implement a web based control panel for downstream customers (not just WWW service customers). This control panel would allow the user to set their password, view/change filtering rules, configure DNS, etc.. The settings can be roughly categorized as follows:

• Setting the User is not even allowed to see
  This would just be a few settings such as wiretaps by the US Department of Homeland Opression.
• Settings the user can see but can't change at all.
  avoid this category.
• Settings the user doesn't have permission to change by themselves. But they can enter a new value and a justification and click [Request Adminstrative Override].
• Settings the user can change.

SMTP Monitoring

Email sent through the companies servers (or snooped through the company routers), should be processed as follows: Compute an MD5 checksum of the message (excluding headers and with all white space (which frequently gets mangled). Lines which begin with ">" should have all leading ">" and " " characters ignored for the MD5 checksum (since this is likely to be messed up when mail is forwarded to abuse@myisp.net. The MD5 of the message, the sender email address, the sender IP, and the date/time stamp should be logged.

Now, when someone sends a spam complaint, you can verify that the enclosed spam was really sent by the user through your network and not forged to cause trouble.

You may want to ammend your terms of service to stipulate that if a message is sent to more than a specified number of recipeints that it will not be considered private and may be inspected for signs of spamming. Also, you can require downstream customers handling mailing lists to register the mailing list with the ISP. Then, if you get the same signature on a large number of messages, you can save a copy for inspection. If it looks like spam and the user has not registered a mailing list, you can take action.

If the user sends more than some number of email messages (specified in terms of service) in a 24 hour period and they haven't registered a mailing list, you can consider contermeasures such as slowing down message delivery.

Enforcing rules on subnets and domains

For each downstream customer, an ISP should run some checks on their network. This primarly applies to subnets but some checks apply to individual dialup/dsl users as well. If the network fails to pass the tests, the port would be shut down (after a fair warning)

• Check that the customer's router protects against ICMP echo packets to the broadcast address.
• Check that every host that is reachable on the subnet has reverse DNS information and that the PTR record points to a domain that points back to the IP address. If it is reachable by ICMP, DNS, HTTP, SMTP, etc. then it needs to have a pointer record. If this fails, figure out whether ISP or the customer is responsible for DNS on that address (i.e. has it been delegated or has ISP made a mistake.
• Check for open SMTP relays.
• For every domain found within the downstream port (reverse DNS gives a good starting point), occasional checks need to be done occasionally to insure that the standard email addresses required by RFCs are implemented (such as postmaster). Warnings should also be issued if certain other addresses (such as abuse) are not found.

These checks should be done from a specific, well identified IP address (such as security.isp.net) since they may set of intrusion detection systems and you want the customer to be able to configure their IDS to treat tests from this system as false alarms.

Use appropriate technology

Proprietary routers make sense for the backbone where routers need to be simple and fast. The leaf nodes, however, where traffic needs to be filtered going to and coming from downstream customers, need to be smart. These should be based on computers with an open source operating system, if possible (unfortunately computers with compact PCI slots and multiport DSL/cable upstream end modems are not readily availible).