<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>don at nz.net</title>
	<atom:link href="http://www.don.nz.net/wordpress/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.don.nz.net/wordpress</link>
	<description>pushing packets for money</description>
	<lastBuildDate>Mon, 30 Jan 2012 21:17:17 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.2</generator>
		<item>
		<title>Lies, Damn Lies and ARP Replies</title>
		<link>http://www.don.nz.net/wordpress/lies-damn-lies-and-arp-replies/</link>
		<comments>http://www.don.nz.net/wordpress/lies-damn-lies-and-arp-replies/#comments</comments>
		<pubDate>Mon, 30 Jan 2012 21:17:17 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Pushing packets]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=122</guid>
		<description><![CDATA[At NZNOG 2012, I presented our work on applying point-to-point semantics to Ethernet-like interfaces, described in my earlier post, Broadcast Interface Addressing Consiudered Harmful. The slides are available here. We&#8217;ve done a bit more work on this since the original article. One thing that occurred to us was that if are prepared to keep maiking [...]]]></description>
			<content:encoded><![CDATA[<p>At <a href="http://2012.nznog.org/cms_display.php">NZNOG 2012</a>, I presented our work on applying point-to-point semantics to Ethernet-like interfaces, described in my earlier post, <a title="Broadcast Interface Addressing Considered Harmful" href="http://www.don.nz.net/wordpress/?p=62">Broadcast Interface Addressing Consiudered Harmful</a>.</p>
<p>The <a href="http://www.don.nz.net/research/arp.pdf">slides are available here</a>.</p>
<p>We&#8217;ve done a bit more work on this since the original article. One thing that occurred to us was that if are prepared to keep maiking ARP requests for a client, you know whether the link is alive or not. In fact, you can do ARP to a host even if you&#8217;re not really talking to it.</p>
<p>Consider: We have an IP host, say 10.99.1.11/24. We tell it that its default gateway is 10.99.1.1. We answer for all ARP requests the host makes, except for itself (see the earlier paper).</p>
<p>But now instead of one upstream router, we have two. And furthermore, we have the two routers, using 10.99.11.2 and 10.99.11.3 respectively as their local IP addresses, i.e. the addresses they will put in their ARP packets (and in any ICMP packets generated from the interfaces). We still tell the client its default gateway is 10.99.11.1.</p>
<p>The two routers both ARP for the host. Both routers know if they can reach the host. Between them, via a &#8220;back channel&#8221; (i.e. a protocol running over the backbone), they agree which host should be the &#8220;active&#8221; router for that host.</p>
<p>The active router simply behaves as the upstream router as previously described. The inactive router does nothing more than make ARP requests for the host, and report its availability. This way, if the active router stops participating in the information protocol (i.e. dies), or the active router loses contact with the host, and the inactive router can still contact the host, the inactive router can take over the active role.</p>
<p>As it does this, it can generate an unsolicited unicast ARP reply to the host, to inform it that the &#8220;default&#8221; IP address (10.99.1.1 in our example) has changed. Other addresses will sort themselves out depending on the host&#8217;s ARP caching strategy. Ideally, the client host will have a fairly rapid ARP time-out and will retry its broadcast ARP for any such addresses.</p>
<p>This approach has advantages over protocols like VRRP. VRRP works by changing the interface MAC address to a &#8220;shared&#8221; address, so that IP clients don&#8217;t know that there has been a change when the active router swaps over. While that makes for a potentially more rapid fail-over, it comes with a number of disadvantages:</p>
<ul>
<li>The shared MAC address changes requires a change to the MAC table on layer 2 switches;</li>
<li>There is some risk of MAC address collisions, especially in Q-in-Q (stacked VLAN) configurations;</li>
<li>the VRRP protocol is visible (multicasted) on the client VLAN;</li>
</ul>
<p>But the major advantage of this approach is that since there is a handshake with the end client. VRRP and similar protocols have no such handshake; they&#8217;re fine for detecting and replacing a failed router, but where the failed component is intervening layer 2 infrastructure, VRRP has no way of knowing that the host is not reachable from the active host, but is reachable from the inactive one. For example:</p>
<ul>
<li>Switch X connects to Y, and Y to Z</li>
<li>Client C connects to switch Y</li>
<li>Client D connects to switch X</li>
<li>Router A connects to switch X, and is active for clients C &amp; D</li>
<li>Router B connects to switch Z, and is inactive for client C &amp; D</li>
</ul>
<p>If the link between switches X and Y fails, Router A loses connectivity to Client C. With ARP handshaking, this loss of connectivity is detected and handled by failing over advertisement of Client C&#8217;s address to Router B. Furthermore, Client D remains reachable from Router A (and indeed connectivity is lost from Router B), but since each client IP address is processed independently, the active router for that host does not change.</p>
<p>We believe this is applicable to a number of situations, especially Internet access networks, be they in a data centre or layer-2 metropolitan access networks.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/lies-damn-lies-and-arp-replies/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>IPocalypse Now</title>
		<link>http://www.don.nz.net/wordpress/ipocalypse-now/</link>
		<comments>http://www.don.nz.net/wordpress/ipocalypse-now/#comments</comments>
		<pubDate>Mon, 11 Apr 2011 08:40:44 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Pushing packets]]></category>
		<category><![CDATA[ego]]></category>
		<category><![CDATA[ipv4]]></category>
		<category><![CDATA[packets]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=91</guid>
		<description><![CDATA[Juha Saarinen dropped me a note a week or two back, asking for an update to my last post, in the wake of the IANA IP address pool finally running out and the recently announced successful bid for Nortel Networks&#8217; IP address space by Microsoft for inclusion in NZCS Newsline. ﻿The published article can be [...]]]></description>
			<content:encoded><![CDATA[<p>Juha Saarinen dropped me a note a week or two back, asking for an update to my last post, in the wake of the <a href="Juha Saarinen dropped me a note a week or two back, asking for an update to my last post, in the wake of the IANA IP address pool finally running out and the recently announced successful bid for Nortel Networks' IP address space by Micrtosoft  ﻿" target="_blank">IANA IP address pool finally running out</a> and the recently announced <a href="http://blog.internetgovernance.org/blog/_archives/2011/3/23/4778509.html" target="_blank">successful bid for Nortel Networks&#8217; IP address space by Microsoft</a> for inclusion in <a href="http://www.nzcs.org.nz/news/" target="_blank">NZCS Newsline</a>.</p>
<p>﻿The published article <a href="http://www.nzcs.org.nz/newsletter/article/102" target="_blank">can be found here</a>, and is different enough from the previous version to warrant re-posting.</p>
<p><span id="more-91"></span></p>
<h1>The sky has fallen</h1>
<div id="wrapper">
<div id="content">
<div id="post-82">
<div>
<p>The <a href="http://etherealmind.com/network-dicitonary-ipocalypse/">IPocalypse</a> is upon us. On the 3rd of February, 2011, IANA allocated the last five /8 IPv4 address blocks, one each, to the five Regional Internet Registries (RIRs).</p>
<p>As of that day, there are no more. Oh woe is us!</p>
<p>Or perhaps we could take a closer look at the situation. There are a bunch of ways that we can measure IP address space usage. They include:</p>
<ol>
<li>The number of addresses available. Formally, this is 2<sup>32</sup> minus the 588,514,560 addresses that are assigned for special uses (multicast, reserved, private addressing etc), leaving 3,706,452,736 addresses (or the equivalent of a little under 221 /8 blocks) available for present or future end-user assignment.</li>
<li>The amount of addresses assigned by IANA to RIRs for allocation. As of the 3rd of February, that would be all of it.</li>
<li>The amount of address space allocated by RIRs to RIR members. According to <a href="http://www.potaroo.net/">Geoff Huston, Chief Scientist at APNIC</a>, this is likely, at current rates of assignment, to run out in mid 2011 for <a href="http://www.apnic.net/">APNIC</a>, with the other two large RIRs (<a href="http://www.ripe.net/">RIPE NCC</a> and <a href="https://www.arin.net/">ARIN</a>) over the following 12 months. The two smaller RIRs (<a href="http://www.lacnic.net/">LANIC</a> and <a href="http://www.afrinic.net/">AfriNIC</a>) will take a few more years.</li>
<li>The amount of address space that is <em>actually advertised to the Internet</em>. Right now, a little under two thirds of the allocatable address space (that is, excluding private, multicast and reserved address space) is actually advertised to the global routing table. <em>That’s right, one third of the IP address space is unequivocally dark.</em></li>
<li>The amount of address space actually allocated to infrastructure and end users. Now things get murky. Is a /8 advertisement actually representing a /8 worth of allocation? Or is the holder of that /8 advertising it simply because they can?</li>
<li>The amount of address space actually in use within allocated address space. This too is largely unmeasurable. Many advertisements, especially smaller ones. are to achieve multi-homing, in which case a /24 may have very small numbers of hosts actually assigned to it. The nature of IP address assignment is that you always have to allocate a larger subnet than you plan to use, unless you can do single IP address per client allocations.</li>
</ol>
<p>Measurements 1 through 4 are easy. 5 and 6 are hard. All we can say for sure is that each measurement will give a smaller number of addresses in use than the one above. If an address appears on the global routing table, we can follow it to its associated autonomous system, but beyond that, we have to look at individual addresses, and even then an assigned and in-use address my be behind a firewall, making it effectively invisible but no less actively in play.</p>
<p>So, the question of when IP address space will run out remains difficult to answer. <a href="http://www.potaroo.net/tools/ipv4/index.html">Geoff Huston’s IPv4 Address Report </a>shows a curve in address advertisements (figure 11) which,although initially exponential, seems to have settled to a linear growth of about 176,000,000 addresses per year in actual advertisements since 2006. If that rate is maintained, the 1.3 billion or so unadvertised addresses should run out in about seven years.</p>
<p>But I suspect that as RIR space becomes unavailable, we’ll start to see address space that is currently advertised but not actually in use being re-allocated (read: sold). For starters, there are about 200 million addresses tied up in non-carrier addresses that are currently advertised as /8s. Admittedly, a large proportion of that space may actually be in use, but one suspects that much of it isn’t. There are a lot of equally historical /16 assignments and smaller blocks assigned under multi-homing policies that are similarly underutilised, and which could be reduced and the balance re-allocated as their holders discover that the space is worth more to them in someone else’s hands than in their own.</p>
<p>So I’m going to lick my finger and stick it in the wind. I think we have ten years or so before we really, genuinely run out of IPv4 addresses, and that&#8217;s ignoring the transition to IPv6 completely. In reality, as IPv4 addresses become scarce (read: expensive), we’ll see folks making do with less and looking harder at IPv6 transition, so I doubt we’ll ever actually run out. Sure, there’s a whole bunch of stuff you can’t do without lots of addresses, but those applications will simply have to developed using IPv6 and instead of IPv4.</p>
<p>Don’t get me wrong; I’m not suggesting for a moment that we don’t have to worry. The single thing that will prevent exhaustion is money. Scarce resources have value; the more scarcity, the more value. RIRs have some really hard choices ahead of them; they’re going to be in the firing line to manage the emerging market in IPv4 address space. Pretending that organisations don’t “own” their address space will stop being an option; the court cases haven’t started in earnest yet, but unless the RIRs urgently awake from the fantasy that IP address space is not a tradeable asset, they will. Similarly, claims that restrictive policies will prevent hoarding are demonstrably fallacious. O<em>ne third of the available address space </em><em>is already being hoarded</em>, leading directly to the conclusion that policies intended to prevent hoarding have already failed, and new, less restrictive policy is required to release that hoarded space for use.</p>
<p>Either the RIRs will rise to this challenge, or they’ll swept into irrelevance. I rather hope for the former rather than the  latter, because the latter means anarchy. The best we can hope for is that enough wiser heads prevail to ensure that the emerging IP address bourses have sufficient support to ensure that the fabric of the Internet isn’t torn apart by the conflict between those who long for the non-commercial Internet of old, and the immediate needs of a market where folks need to get stuff done.</p>
<h2>Post Scriptum</h2>
<p>I wrote most of this article in December, prior to the last /8 IANA blocks being allocated, but I have been predicting that IP address exhaustion would and must ultimately be addressed by the market for over a decade, often to the derision of fellow industry types. However, it was with a sense of vindication that I welcomed the news that Microsoft had successfully bid for a large block of address space held by the bankrupt carcass of Nortel Networks.</p>
<p>Personally, over the last decade, I have seen many examples of IP address space being treated as a business asset, often at the sharp end where the &#8220;ownership&#8221; of the &#8220;asset&#8221; was a subject of discussion or even dispute. IP address space regularly shows up on lists of assets to be transferred in corporate transactions. However, in none of these cases has anyone been so bold as to actually put a monetary value on address space in isolation.</p>
<p>Now, in the process of asset-stripping a once-proud telecommunications equipment vendor, we see public acknowledgement of that value. The price, approximately NZ$15 per address, is fairly reasonable &#8211; amortised over five years, it represents about 30c/month, a tiny proportion of the revenue that any service requiring an IP address can generate.</p>
<p>Less heartening was the reaction from John Curran, President and CEO of ARIN, hinting that ARIN could yet block the transfer if it is not in compliance with &#8220;the community-developed policies by which we maintain the ARIN Whois database&#8221;. What that means in practice remains to be seen, but if the transfer is blocked, we can be fairly sure that the address space won&#8217;t be made available to Internet users. At least, <em>not officially</em>.</p>
</div>
</div>
</div>
</div>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/ipocalypse-now/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>The sky is falling</title>
		<link>http://www.don.nz.net/wordpress/the-sky-is-falling/</link>
		<comments>http://www.don.nz.net/wordpress/the-sky-is-falling/#comments</comments>
		<pubDate>Wed, 29 Dec 2010 02:05:34 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Pushing packets]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=82</guid>
		<description><![CDATA[The IPocalypse is upon us. There are seven /8 IPv4 address blocks left! Soon there will be six. Then five. On that fateful day, when the sixth to last /8 block is assigned, the five Regional Internet Registries (RIRs) will receive one each of the remaining five /8s for final allocation. This will probably happen [...]]]></description>
			<content:encoded><![CDATA[<p>The <a href="http://etherealmind.com/network-dicitonary-ipocalypse/">IPocalypse</a> is upon us. There are seven /8 IPv4 address blocks left! Soon there will be six. Then five.</p>
<p>On that fateful day, when the sixth to last /8 block is assigned, the five Regional Internet Registries (RIRs) will receive one each of the remaining five /8s for final allocation. This will probably happen in the next month or two.</p>
<p>Then there will be no more! Oh woe is us!</p>
<p>Or not. There are a bunch of ways that we can measure IP address space usage. They include:</p>
<ol>
<li>The number of address available. Formally, this is 2^32 minus the 588,514,560 addresses (or just over 35 /8 blocks)that  are assigned for special uses (multicast, reserved, private addressing etc), leaving 3,706,452,736 addresses (or the equivalent of just over  220.9 /8 blocks) available for present or future end-user assignment.</li>
<li>The amount of addresses assigned by IANA to RIRs for allocation. Currently, this stands at pretty much all of the above space, less the aforementioned seven /8s (or 117,440,512 addresses).</li>
<li>The amount of address space allocated by RIRs. <a href="http://www.potaroo.net/">According to Geoff Huston</a>, this is likely, at current rates of assignment, to run out in mid-late 2011.</li>
<li>The amount of address space that is actually advertised. Right now, a little under 2/3rds of the allocatable address space (that is, excluding private, multicast and reserved address space) is actually advertised to the global routing table. That&#8217;s right, 1/3rd of the IP address space is unequivocally dark.</li>
<li>The amount of address space actually allocated to infrastructure. Now things get murky. Is a /8 advertisement actually representing a /8 worth of allocation? Or is the holder of that /8 advertising it simply because they can?</li>
<li>The amount of address space actually in use. This too is largely unmeasurable. Many advertisements, especially smaller ones. are to achieve multihoming, in which case a /24 may have very small numbers of hosts actually assigned to it. The nature of IP address assignment is that you always have to allocate a larger subnet than you plan to use, unless you can do single IP address per client allocations, e.g. using PPP &amp; friends, <a href="http://www.don.nz.net/wordpress/?p=62">my ARP hack</a> or layer-3 VLAN schemes.</li>
</ol>
<p>Measurements 1 through 4 are easy. 5 &amp; 6 are hard. All we can say for sure is that each measurement will give a smaller number of addresses in use than the one above. If an address appears on the global routing table, we can follow it to its associated autonomous system, but beyond that, we have to look ad individual addresses, and even then an assigned and in-use address my be behind a firewall or something and effectively invisible but none the less actively in play.</p>
<p>It did occur to me to look at reverse map entries, but experience suggests that these are unhelpful, being fairly universally badly managed.</p>
<p>So, the question of when IP address space will run out remains difficult to answer. <a href="http://www.potaroo.net/tools/ipv4/index.html">Geoff&#8217;s IPv4 Address Report </a>shows a curve in address advertisements (fig. 11c) which,although initially exponential, seems to have settled to a linear growth of about 176,000,000 addresses per year in actual advertisements since 2006. If that rate is maintained, the 1.3 billion or so unadvertised addresses should run out in about seven years.</p>
<p>But I suspect that as RIR space becomes unavailable, we&#8217;ll start to see address space that is currently advertised but not actually in use being re-allocated (read: sold). For starters, there are about 200 million addresses tied up in non-carrier addresses that are currently advertised as /8s. Admittedly, a goodly chunk of that space may actually be in use, but one suspects that a significant proportion isn&#8217;t. There are a lot of equally historical /16 assignments and smaller blocks assigned under multihoming policies that are similarly underutilised, and could shed a large proportions of their advertised allocation as their holders discover it&#8217;s worth more to them in someone else&#8217;s hands than in their own.</p>
<p>So I&#8217;m going to lick my finger and stick it in the wind. I think we have ten years or so before we really, genuinely run out of IPv4 addresses, and that ignores the transition to IPv6 completely. In reality, as IPv4 addresses become scarce (read: expensive), we&#8217;ll see folks making do with less and looking harder at IPv6 transition, so I doubt we&#8217;ll ever actually run out. Sure, there&#8217;s a whole bunch of stuff you can&#8217;t do without lots of addresses, but those applications will simply have to go to IPv6.</p>
<p>Don&#8217;t get me wrong; I&#8217;m not suggesting for a moment that we don&#8217;t have to worry. The single thing that will prevent exhaustion is money. Scarce resources have value; the more scarcity, the more value. RIRs have some really hard choices ahead of them; they&#8217;re going to be in the firing line to manage the emerging market in IPv4 address space. Pretending that organisations don&#8217;t &#8220;own&#8221; their address space will stop being an option; the court cases haven&#8217;t started in earnest yet, but unless the RIRs urgently awake from the fantasy that IP address space is not a tradable asset, they will.</p>
<p>Either they will rise to the challenge, or they&#8217;ll swept into irrelevancy. I rather hope the latter doesn&#8217;t happen, because the alternative is anarchy. The best we can hope for is that enough wiser heads prevail to ensure that the emerging IP address bourses have sufficient support to ensure that the fabric of the Internet isn&#8217;t torn apart by the conflict between those who long for a non-commercial Internet where everyone plays nice, and the immediate needs of a market where folks need to get stuff done.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/the-sky-is-falling/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Oh, there&#8217;s the any key!</title>
		<link>http://www.don.nz.net/wordpress/oh-theres-the-any-key/</link>
		<comments>http://www.don.nz.net/wordpress/oh-theres-the-any-key/#comments</comments>
		<pubDate>Fri, 11 Jun 2010 04:12:31 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Perversity]]></category>
		<category><![CDATA[hardware]]></category>
		<category><![CDATA[murphy's law]]></category>
		<category><![CDATA[tech support]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=76</guid>
		<description><![CDATA[This is a picture of my keyboard: Yes, it&#8217;s grubby. And yes, this keyboard really is old enough to not have Windows keys. Actually, it&#8217;s about twice that old. Twenty years ago I needed a new keyboard, so I bought a cheap one. (Back then, $200 or more was cheap for a keyboard.) I&#8217;m not [...]]]></description>
			<content:encoded><![CDATA[<p>This is a picture of my keyboard:</p>
<p style="text-align: center;"><a href="http://www.don.nz.net/wordpress/wp-content/uploads/2010/06/IMG_0631.jpg"><img class="size-medium wp-image-77 aligncenter" title="Any key" src="http://www.don.nz.net/wordpress/wp-content/uploads/2010/06/IMG_0631-e1276227389534-300x255.jpg" alt="" width="300" height="255" /></a></p>
<p style="text-align: left;">Yes, it&#8217;s grubby. And yes, this keyboard really is old enough to not have Windows keys. Actually, it&#8217;s about twice that old. Twenty years ago I needed a new keyboard, so I bought a cheap one. (Back then, $200 or more was cheap for a keyboard.) I&#8217;m not really sure what I&#8217;m going to do when it expires because I&#8217;ve never used a keyboard since that I liked. They don&#8217;t make keyboards like this any more, with discrete key switches and a distinct tactile click when the key goes down. (Well, they do, but they&#8217;re big heavy IBM keyboards that are so noisy they can be heard three blocks away.)</p>
<p style="text-align: left;">And yes, that key between the Ctrl and Alt keys is labelled &#8220;Any&#8221;.</p>
<p>The true irony of this is that this key doesn&#8217;t actually do anything. No key-code is generated when you press it, so pressing the &#8220;Any&#8221; key in response to &#8220;Press any key to continue&#8221; will result in a distinct lack of continuation.</p>
<p>We all have our favourite tech support stories, the &#8220;my cup holder is broken&#8221; cases, the &#8220;it works better if you plug it in&#8221; cases. So I wonder how many of us have actually had someone ask where the Any key was?</p>
<p>Once I got called out to look at a printer that apparently wasn&#8217;t working. The data plug was upside down. It was a D-shell plug, and they only go in one way, but there it was.</p>
<p>I turned it over and it worked fine.</p>
<p>I know you don&#8217;t believe me. I wouldn&#8217;t believe me. But it did happen &#8211; the male plug was a wee bit bigger than it should have been, and only had a few pins installed which in turn were a bit loose, and the combination of these faults meant it actually went together and seated tightly.</p>
<p>Many, many moons ago, back in the days of serial terminals and multiplexors, the boss came by, saying, &#8220;I just had a call from the Auckland office. They say all their terminals are down.&#8221; I muttered something unprintable, and wandered into the comms room.</p>
<p>Looking at the multiplexor, I noted the &#8220;RA&#8221; light flashing. Remote Alarm, meaning the mux couldn&#8217;t see the mux at the other end. Probably a comms fault, hardly the first time. Moving up the rack, the NTU on the data circuit to Auckland indicated that it couldn&#8217;t see its partner at the other end.</p>
<p>That could just about explain it.</p>
<p>So I ambled off in the direction of the technicians&#8217; office. Back then the telco stuff was handled by the people who looked after the phones, and that meant the techs. So I told Evans, the head tech of my findings, and he picked up the phone to put through a fault call.</p>
<p>Later that day, I ran into Evans in the corridor. &#8220;What&#8217;s up with that Auckland circuit?&#8221; I asked.</p>
<p>&#8220;Oh the fault man went out there. There&#8217;s no power.&#8221;</p>
<p>&#8220;What, to the NTU?&#8221;</p>
<p>&#8220;Nah, to the building.&#8221;</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/oh-theres-the-any-key/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Broadcast Interface Addressing Considered Harmful</title>
		<link>http://www.don.nz.net/wordpress/broadcast-interface-addressing-considered-harmful/</link>
		<comments>http://www.don.nz.net/wordpress/broadcast-interface-addressing-considered-harmful/#comments</comments>
		<pubDate>Thu, 28 Jan 2010 20:50:12 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Pushing packets]]></category>
		<category><![CDATA[arp]]></category>
		<category><![CDATA[broadband]]></category>
		<category><![CDATA[ethernet]]></category>
		<category><![CDATA[ipv4]]></category>
		<category><![CDATA[packets]]></category>
		<category><![CDATA[point-to-point ethernet]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=62</guid>
		<description><![CDATA[I hate IPv4 link broadcast interface (e.g. Ethernet) addressing semantics.  To recap, if I have two boxes on each end of a point-to-point link (say between a gateway and an end host), we address as follows (for example): 10.1.1.0: Network address (reserved) 10.1.1.1: Host 1 (gateway) 10.1.1.2: Host 2 (end host) 10.1.1.3: Broadcast address. That&#8217;s [...]]]></description>
			<content:encoded><![CDATA[<p>I hate IPv4 link broadcast interface (e.g. Ethernet) addressing semantics.  To recap, if I have two boxes on each end of a point-to-point link (say between a gateway and an end host), we address as follows (for example):</p>
<ul>
<li>10.1.1.0: Network address (reserved)</li>
<li>10.1.1.1: Host 1 (gateway)</li>
<li>10.1.1.2: Host 2 (end host)</li>
<li>10.1.1.3: Broadcast address.</li>
</ul>
<p>That&#8217;s four IP addresses, for a link to a single host.  Hello?  Haven&#8217;t you heard the news?  IP addresses are running out!</p>
<p>Some folks manage to get away with using /31 masks, e.g.</p>
<ul>
<li>10.1.1.4: Host 1 (gateway)</li>
<li>10.1.1.5: Host 2 (end host)</li>
</ul>
<p>which is just wrong.  Better in terms of address usage (two addresses instead of four), but still just plain wrong. An you&#8217;re still wasting addresses.</p>
<p>The PPP folks a long time ago figured that a session, particularly in client to concentrator type configurations, only needs one IP address. A &#8220;point to point&#8221; interface has a local address, and a remote address, of which only the remote address needs to be stuffed in the routing table.  The local address can be the address of the concentrator, and doesn&#8217;t even need to be in the same subnet.</p>
<p>So why can&#8217;t my Ethernet interfaces work the same way?</p>
<p>A point to point link really doesn&#8217;t have broadcast semantics.  Apart from stuff like DHCP, you never really need to broadcast &#8212; after all, our PPP friends don&#8217;t see a need for a &#8220;broadcast&#8221; address.</p>
<p>Well, we decided we had to do something about this.  The weapon of choice is NetGraph on FreeBSD.  NetGraph basically provides a bunch of kernel modules that can be linked together.  It&#8217;s been described as &#8220;network Lego&#8221;.  I like it because it&#8217;s easy to slip new kernel modules into the network stack in a surprising number of places. This isn&#8217;t a NetGraph post, so I won&#8217;t spend more verbiage on it,but it&#8217;s way cool. <a href="http://www.google.com/search?q=freebsd+netgraph">Google it</a>.</p>
<p>In a real point-to-point interface, both ends of the link know the semantics of the link.  For Ethernet point-to-point addressing, we can still do this (and my code happily supports this configuration), but obviously both ends have to agree to do so. &#8220;Normal&#8221; clients won&#8217;t know what we&#8217;re up to, so we have to do this in such a way that we don&#8217;t upset their assumptions.</p>
<p>So we cheat. And we lie. And worst of all,we do proxy ARP!</p>
<p>What we do is tell our clients that they are on a /24 network. Their IP address is, for example, 10.1.2.5/24, and the gateway is 10.1.2.1. Any time we get a packet for 10.1.2.5, we&#8217;ll send it out that interface, doing ARP as normal to resolve the remote host&#8217;s MAC address.</p>
<p>Going the other way, we answer ARP requests for any IP address in 10.1.2.0/24, except 10.1.2.5, with our own MAC address.  That means that if they ARP for 10.1.2.6, we&#8217;ll answer the ARP request, which directs that packet to us, where we can use our interior routes to route it correctly.  In our world, two &#8220;adjacent&#8221; IP addresses could be on opposite sides of the network, or it could be on a different VLAN on the same interface.</p>
<p>The result is one IP address per customer.  We &#8220;waste&#8221; three addresses per 256, the network (.0), gateway (.1) and broadcast (.255), and we have to be a bit careful about what we do with the .1 address &#8212; it could appear on every router that is playing with that /24.  But we can give a user a single IP address, and put it anywhere in the network.</p>
<p>We can actually have multiple IP addresses on the same interface; we do this by having the NetGraph module have a single Ethernet interface but multiple virtual point-to-point interfaces.  So if we want to give someone two IP addresses, we can do that as two, not necessarily adjacent, /32 addresses.  We don&#8217;t answer ARPs for any of the assigned addresses, but do answer everything else. The module maintains a mapping of point-to-point interface to associated MAC address.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/broadcast-interface-addressing-considered-harmful/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Don&#8217;t shout at your disks</title>
		<link>http://www.don.nz.net/wordpress/dont-shout-your-disks/</link>
		<comments>http://www.don.nz.net/wordpress/dont-shout-your-disks/#comments</comments>
		<pubDate>Sat, 09 Jan 2010 23:59:29 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Servers]]></category>
		<category><![CDATA[hardware]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=54</guid>
		<description><![CDATA[Seriously.  They don&#8217;t like it.  They sulk. Brendan Gregg of the Sun Microsystems Fishworks engineering team, has written up this effect, with video, at http://blogs.sun.com/brendan/entry/unusual_disk_latency Moreover, don&#8217;t vibrate your drives.  Why an I saying this? Because, three months ago we took delivery of three 1U pizza boxes. They&#8217;re small Supermicro boxes, with room for a [...]]]></description>
			<content:encoded><![CDATA[<p>Seriously.  They don&#8217;t like it.  They sulk.</p>
<p>Brendan Gregg of the Sun Microsystems Fishworks engineering team, has written up this effect, with video, at<a href="http://blogs.sun.com/brendan/entry/unusual_disk_latency"> http://blogs.sun.com/brendan/entry/unusual_disk_latency</a></p>
<p>Moreover, don&#8217;t vibrate your drives.  Why an I saying this?</p>
<p>Because, three months ago we took delivery of three 1U pizza boxes. They&#8217;re small Supermicro boxes, with room for a normal ATX motherboard and a hard drive.  We equipped these with terabyte drives, fairly normal Supermicro motherboards, 3 GHz Core2 Duo CPUs and 8GB memory each.</p>
<p>They just didn&#8217;t run right.  Occasionally, one wouldn&#8217;t even make it through an OS install, and the ones that did wouldn&#8217;t put through as much work as a much lower spec machine.</p>
<p>We suspected the drives; we suspected the power supply.  Actually, we really thought it was the power supply, but even though the PSUs on these chassis were small, and the 12V rails seems to be running slightly low, at 11.85V, no amount of bashing the numbers suggested that the systems were actually underpowered.</p>
<p>The first breakthrough was running &#8220;hdparm -t &#8211;direct /dev/sda&#8221; on the drive, which showed wildly fluctuating numbers, consistent with the behaviour we were seeing.  So it was something to do with the disk subsystem.</p>
<p>The next breakthrough was when we discovered that if we unplugged the chassis fan (an ugly centrigufal thing) from the motherboard, the problem went away.  The hdparm numbers stabilised at 100MB/s or more.</p>
<p>We saw small changes in power supply volts when we did this, so we were still suspecting the power supply.  I put an ammeter on the fan power line, to see how much power the fan was pulling.  1.2A at full speed.</p>
<p>We played with the fan speed in the BIOS; at its lowest speed, it would pull 0.25A, and the drive would perform well; at the &#8220;server&#8221; setting, with the server otherwise unloaded, it would pull about 0.6A.  At that rate, it was starting to have an effect on performance.</p>
<p>This was a PSU that was supposed to be able to deliver 18A on the 12V rail, and 260W total.  I really couldn&#8217;t see how the 12V would be at the edge when the PSU was pulling less than 100W (measured at the AC feed) and was running three fans and a hard drive and a few minor bits and pieces like the serial port and network interface, all of which should have summed to maybe 5A.  The numbers didn&#8217;t add up.</p>
<p>Finally, I had a brainwave.  I removed the fan from the chassis, still running.  The problem went away.  I touched the fan to the drive.  The drive throughput dropped through the floor.</p>
<p>After a few more experiments, the conclusion is that with the fan mounted close to the drive, the vibrations were enough to upset the performance of the drive, consistently.  Two different terabyte drives (one Seagate, one Western Digital) exhibited the same problem.</p>
<p>I duplicated this by applying abnormal vibration to the case of my desktop PC (half terabyte Seagate), and even the grottly little thing I have at home (a Seagate 160GB PATA drive).</p>
<p>Conclusion: all modern drives are subject to potentially serious performance issues when faced with abnormal vibration.  The Supermicro chassis exacerbated the problem  because of the placement of the fan with respect to the drive, and the fact the drive is mounted directly to the chassis.  Also, the placement of cables up against the fan meant that vibrations were being transferred directly through the connectors from the fan; somthing that could be partially alleviated by re-routing the power cable under the fan.</p>
<p>The fact that right angle SATA power connectors are so darned hard to get made this more of an issue than it should have been.</p>
<p>I think a bit of judicious use of closed-cell foam packing, turning the fan speed down, and re-routing cables away from the fan will finally solve the problem.</p>
<p>Hopefully.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/dont-shout-your-disks/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Poor Man&#8217;s Anycast</title>
		<link>http://www.don.nz.net/wordpress/poor-mans-anycast/</link>
		<comments>http://www.don.nz.net/wordpress/poor-mans-anycast/#comments</comments>
		<pubDate>Sat, 28 Nov 2009 03:39:13 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Pushing packets]]></category>
		<category><![CDATA[dns]]></category>
		<category><![CDATA[packets]]></category>
		<category><![CDATA[web]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=34</guid>
		<description><![CDATA[The following is a technique I&#8217;ve used over the last decade or so for distributing web traffic (or potentially any services) across multiple services, using just DNS.  Being an old DNS hack, I&#8217;ve called this technique Poor Man&#8217;s Anycast, although it doesn&#8217;t really use anycasting. But before we get into the technique, we need to [...]]]></description>
			<content:encoded><![CDATA[<p>The following is a technique I&#8217;ve used over the last decade or so for distributing web traffic (or potentially any services) across multiple services, using just DNS.  Being an old DNS hack, I&#8217;ve called this technique Poor Man&#8217;s Anycast, although it doesn&#8217;t really use anycasting.</p>
<p>But before we get into the technique, we need to make a brief diversion into the little-known but rather neat feature of the DNS, or more accurately, DNS forwarders, which makes this a cool way to do stuff. The feature is name server selection.</p>
<p>Most DNS clients, and by this I include your home PC, make use of a <em>DNS forwarder</em>.  The <em>forwarder </em>is the thing that handles (and caches) DNS requests from end clients, while a DNS <em>server </em>carries authoritative information about a limited set of domains and only answers queries for them. These two functions have historically been conflated rather severely, mainly due to the use of BIND for both, and why this is a bad thing is the subject for a whole other post.</p>
<p>Moving right along. A DNS forwarder gets to handle lots of queries for any domain that its clients ask for. When you ask for <strong>foo.example.net</strong>, it asks one of the root servers (<strong>a.root-servers.net</strong>, <strong>b.root-servers.net</strong> et al) for that full domain name (let&#8217;s assume it&#8217;s just come up and doesn&#8217;t have anything cached).  It gets back a <em>delegation </em>from the root servers, saying basically, &#8220;I don&#8217;t know, but the GTLD (<strong>.com</strong>, <strong>.net</strong>) servers will&#8221;, and tell you where to find the GTLD servers (<strong>a.gtld-servers.net</strong> et al).  You ask the one of the GTLD servers, and get back an answer that says that they don&#8217;t know either, but <strong>ns1.example.net</strong> and <strong>ns2.example.net</strong> do.</p>
<p>You then ask (say) <strong>ns1.example.net, </strong>and hopefully you&#8217;ll get the answer you want (e.g. the IP address).</p>
<p>Now, along the way, the forwarder has been caching everything it got. Every time it asks a name server for data, <em>it stores the time it took to reply</em>. That means that when looking up names in <strong>example.net</strong>, the forwarder has been collecting timing and reliability data which it uses to choose which name server to ask next time, as well as the answers it received.  So if <strong>ns1.example.net</strong> answers in 20 ms, but <strong>ns2.example.net</strong> answers in 10 ms, roughly two thirds of the queries for <strong><em>something</em>.example.net</strong> will be sent to <strong>ns2.example.net</strong>. If the timing difference is much greater, the split of queries will be even more marked. Similarly, if a name server fails to respond at all, that fact will be reflected in the accumulated preference assigned to that server, and it will get very few queries in future; just enough so that we know we can start sending it queries again when it comes back.</p>
<p>This is a powerful effect, and is of particular use when distributing servers over a wide geographical area. DNS specialists know about it, because poor DNS performance affects everything, and DNS people don&#8217;t like adversely affecting everything. (They&#8217;re really quite paranoid about it. Trust me, I&#8217;m one.) But it can also be used to pick the closest server for other things as well.</p>
<p>After all, closeness (in terms of round-trip time) is very important in network performance (see my post on <a title="Bandwidth Delay Product" href="http://www.don.nz.net/wordpress/?p=19" target="_self">bandwidth delay products</a>).</p>
<p>The technique is as follows.  Let&#8217;s say we have three web servers, carrying static content. Call them, say, <strong>auckland.example.net</strong>, <strong>chicago.example.net</strong> and <strong>london.example.net</strong>. Let&#8217;s say that they&#8217;re widely disparate. All three servers carry content for <strong>http://www.example.com/</strong>.</p>
<p>So, we start by configuring, on the example.com name servers:</p>
<blockquote>
<pre>$ORIGIN example.com.
$TTL 86400
www     IN      NS      auckland.example.net.
        IN      NS      chicago.example.net.
        IN      NS      london.example.net.</pre>
</blockquote>
<p>We then run a DNS server on all three web servers.  We configure the servers with a zone for <strong>www.example.com</strong> along the lines of:</p>
<blockquote>
<pre>$ORIGIN www.example.com
$TTL 86400                       ; Long (24 hour) TTL on NS records etc
@       IN      SOA     auckland.example.net. webmaster.example.com (
                                2009112900 3600 900 3600000 300 )
        IN      NS      auckland.example.net.
        IN      NS      chicago.example.net.
        IN      NS      london.example.net.
$TTL 300                         ; Short (five minute) TTL on A record
@       IN      A       10.0.0.1 ; Set this to host IP address</pre>
</blockquote>
<p>Now the key is that each web server serves up <em>its own IP address</em>. When a DNS forwarder makes a query for a <strong>www.example.com</strong>, it will be directed to one of <strong>auckland.example.net</strong>, <strong>chicago.example.net</strong> or <strong>london.example.net</strong>. But as more and more queries get made, one of those three will start handling the bulk of the queries, at least if that one is significantly closer than the other two. And if <strong>auckland.example.net</strong> gets the query, it answers with its <strong> </strong>own IP address, meaning that it also gets the subsequent HTTP request or other services directed to it. The short DNS TTL (5 minutes in the example) mean that the address gets queried moderately often, allowing the name server selection to &#8220;get up to speed&#8221;. Much longer TTLs on the name servers mean the data doesn&#8217;t get forgotten too quickly.</p>
<p>The result is that in many cases, the best server gets the request.</p>
<p>The technique works best if there are lots of domains being handled by the same set of servers, and there are lots of requests coming through. That way the preferences get set quickly in the major ISPs&#8217; DNS forwarders. The down side of the technique is that far away servers will still get <em>some</em> queries. This non-determinism may be a reason for not deploying this technique.  If you want determinism, you&#8217;ll need to look at more industrial grade techniques.</p>
<p>Now, this isn&#8217;t what players like Akamai do, and it isn&#8217;t what anycasting is about. Akamai and (some) other content distribution networks work by maintaining a map of the Internet, and returning DNS answers based on the requester&#8217;s IP address. But this is a fairly heavyweight answer to the problem. It&#8217;s not something you can implement with just BIND alone.</p>
<p>Anycasting on the other hand relies on advertising the same IP address in multiple places, and letting BGP sort out the nearest path. This has three disadvantages:</p>
<ol>
<li>It potentially breaks TCP. If there is an equal cost path to a given anycast node, it&#8217;s possible one packet from a stream might go one way, while the next packet might be sent to a completely different host (at the same IP address). In practice, this has proven to be less of a problem than might be expected, but there is still scope for surprises.</li>
<li>Each of your nodes has to be separately BGP peered with its upstream network(s). That&#8217;s a lot more administration than many ISPs will do for free.</li>
<li>Most importantly, being close in BGP terms is not the same as being close physically or in terms of round-trip time. Many providers have huge reach within a single AS, so a short AS-path (the main metric for BGP) may actually be a geographically long distance, with a correspondingly long round-trip.</li>
</ol>
<p>The other nice thing about poor man&#8217;s anycast is that it&#8217;s dynamic; if a node falls off the world, as long as its DNS goes away too, it&#8217;ll just disappear from the cloud as soon as the TTLs time out. If a path to it gets congested, name server selection will notice the increased TTL and de-prefer that server.</p>
<p>And of course you don&#8217;t need to be a DNS or BGP guru, or buy/build expensive, complex software systems to set it up.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/poor-mans-anycast/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Bandwidth Delay Product</title>
		<link>http://www.don.nz.net/wordpress/bandwidth-delay-product/</link>
		<comments>http://www.don.nz.net/wordpress/bandwidth-delay-product/#comments</comments>
		<pubDate>Thu, 12 Nov 2009 11:44:18 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Pushing packets]]></category>
		<category><![CDATA[bandwidth]]></category>
		<category><![CDATA[broadband]]></category>
		<category><![CDATA[packets]]></category>
		<category><![CDATA[regional fibre]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=19</guid>
		<description><![CDATA[I found myself explaining this one at Curry tonight, in the context of discussing fast broadband. Basically, if you have a reliable stream protocol like, to take a random example, TCP, and you&#8217;re not doing anything imaginative with it, you run into the following problem: Every byte you send might need to be resent if [...]]]></description>
			<content:encoded><![CDATA[<p>I found myself explaining this one at Curry tonight, in the context of discussing fast broadband.</p>
<p>Basically, if you have a reliable stream protocol like, to take a random example, TCP, and you&#8217;re not doing anything imaginative with it, you run into the following problem:</p>
<p>Every byte you send might need to be resent if it gets lost along the way.  So, you buffer whatever you send up until you get an acknowledgement from the other end.  Let&#8217;s say, for argument&#8217;s sake you use a 64k buffer. We call this buffer the <em>window</em>, and the size is the <em>window size.</em></p>
<p>Now, let&#8217;s say you have a looooonnnngggg path between you and your remote endpoint. Let&#8217;s say it&#8217;s 200 milliseconds, or 1/5th of a second. This is pretty reasonable for an NZ-US connection &#8212; the speed of light is not our friend.</p>
<p>And finally, for simplicity sake, let&#8217;s say that the actual bandwidth over that path is Very High, so serialisation delays (the time taken to put one bit after the next) are negligible.</p>
<p>So,  if I send 64k bytes (or 512 k bits) worth of data, it takes 200 ms before I get an acknowledgement. It doesn&#8217;t matter how fast I send my 64k; I still have to stop when I&#8217;ve sent it.  200 ms later, I get a bunch of acknowledgements back, for the whole 64k (assuming nothing got dropped), and I can now send my next 64k.</p>
<p>So the actual throughput, through my SuperDuperBroadband connection, is 64k bytes per 200 ms, or 2.5 Mbps.</p>
<p>To turn this around, if I want 2.5 Mbps at 200 ms, I need a 64k byte window; if I want 5 Mbps on a 200 ms path, I&#8217;m going to need to up the window size to 5 Mbps times 200 ms = 128 k bytes.</p>
<p>That window size calculation is the <em>bandwidth delay product</em>.</p>
<p>There&#8217;s &#8216;s the theory. Pick a big window size and go fast. Except:</p>
<ol>
<li>You don&#8217;t get to pick. Even if you control your application, for downloads you can ask for a bigger window size, but you don&#8217;t necessarily get it. Probably, you&#8217;ll get the smaller of what the applications at either end asked for.</li>
<li>Standard, 1981 edition TCP has the window (buffer) size that can be communicated by the endpoints maxed out at 64k. This isn&#8217;t the end of the world; in 1992 Van Jacobson and friends rode to the rescue with RFC 1323, which allows the window size to be scaled, to pretty much anything you like. But most TCP stacks come with a default window size in the 64k-ish range, and may applications don&#8217;t change it.</li>
<li>Even if both ends of a TCP session ask for and get a large maximum window size, they don&#8217;t start with it. TCP congestion control requires that everyone start slowly (it;s called <em>slow start</em>), and this is done by starting with a small window size, and increasing it as the acknowledgements flow in and the sending end can get an idea about how much bandwidth there is.  So if your application uses lots of short TCP sessions rather than one long one, you&#8217;ll never reach your maximum window size and therefore never saturate your connection.</li>
</ol>
<p>What to do? It depends what you&#8217;re trying to achieve.  For file transfers, run lots of TCP sessions side by side &#8211; can anyone say BitTorrent? Local caching helps for web traffic; move the content closer, and the bandwidth delay product is less. Use a different protocol. I have to say I&#8217;ve seen quite a few UDP-based file transfer protocols come and go, because tweaking TCP parameters at both ends is usually a darn sight easier than getting a new protocol right. (see <a title="don's law" href="http://www.don.nz.net/wordpress/?p=5" target="_self">don&#8217;s law</a>).</p>
<p>What it comes down to, is that if all you&#8217;re going to use your UltraSuperDuperFast broadband connection for is downloading videos from US servers, you&#8217;re going to be disappointed. The real key to making this useful is local, or at least, locally hosted, content. Preferably located right by the fibre head-ends. It&#8217;s a parallel stream to the effort to get the fibre in the ground and get it lit, and it needs to be attended to PDQ.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/bandwidth-delay-product/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>don&#8217;s law</title>
		<link>http://www.don.nz.net/wordpress/dons-law/</link>
		<comments>http://www.don.nz.net/wordpress/dons-law/#comments</comments>
		<pubDate>Wed, 04 Nov 2009 04:45:39 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Perversity]]></category>
		<category><![CDATA[ego]]></category>
		<category><![CDATA[murphy's law]]></category>
		<category><![CDATA[sod's law]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=5</guid>
		<description><![CDATA[is: &#8220;If there&#8217;s an unexpected way to implement a widely used protocol or process, someone out there has done so.&#8221; Yes, it&#8217;s a lot like the original Murphy&#8217;s Law, &#8220;If there is any way to do it wrong, he will&#8221;, attributed to Edward Murphy, an engineer on the rocket sled tests carried out in the [...]]]></description>
			<content:encoded><![CDATA[<p>is:</p>
<blockquote><p>&#8220;If there&#8217;s an unexpected way to implement a widely used protocol or process, someone out there has done so.&#8221;</p></blockquote>
<p>Yes, it&#8217;s a lot like the <em>original</em> Murphy&#8217;s Law, &#8220;If there is any way to do it wrong, he will&#8221;, attributed to Edward Murphy, an engineer on the rocket sled tests carried out in the 1950s, in discovering that all of the accelerometers on the a test subject had been wired backwards, requiring a re-run of an expensive test.</p>
<p>don&#8217;s law isn&#8217;t just a re-statement of Murphy&#8217;s Law, or even Sod&#8217;s Law (&#8220;if it can go wrong, it will&#8221;).  It&#8217;s a recognition that there are a lot of implementations of stuff out there,  some by people <em>who don&#8217;t think the way you do. </em> (Or I do.  Or, in some cases, like any sane individual.) And the way those implementations work, especially under exception conditions, can vary enormously.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/dons-law/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>because I can.</title>
		<link>http://www.don.nz.net/wordpress/hello-world/</link>
		<comments>http://www.don.nz.net/wordpress/hello-world/#comments</comments>
		<pubDate>Wed, 04 Nov 2009 03:46:55 +0000</pubDate>
		<dc:creator>don</dc:creator>
				<category><![CDATA[Meta]]></category>
		<category><![CDATA[ego]]></category>

		<guid isPermaLink="false">http://www.don.nz.net/wordpress/?p=1</guid>
		<description><![CDATA[&#8230; is why I&#8217;ve started this blog. Yes, it&#8217;s an ego thing.  A platform to spout the random things that spring to mind, when they&#8217;ve sprung.]]></description>
			<content:encoded><![CDATA[<p>&#8230; is why I&#8217;ve started this blog.</p>
<p>Yes, it&#8217;s an ego thing.  A platform to spout the random things that spring to mind, when they&#8217;ve sprung.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.don.nz.net/wordpress/hello-world/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

