>>> 2022-02-19 PCM

I started writing a post about media container formats, and then I got severely sidetracked by explaining how MPEG elementary streams aren't in a container but still have most of the features of containers and had a hard time getting back to topic until I made the decision that I ought to start down the media rabbit hole with something more basic. So let's talk about an ostensibly basic audio format, PCM.

PCM stands for Pulse Code Modulation and, fundamentally, it is a basic technique for digitization of analog data. PCM is so obvious that explaining it is almost a bit silly, but here goes: given an analog signal, at regular intervals the amplitude of the signal is measured and quantized to the nearest representable number (in other words, rounded). The resulting "PCM signal" is this sequence of numbers. If you remember your Nyquist and Shannon from college data communications, you might realize that the most important consideration in this process is that the sampling frequency must be twice the highest frequency component in the signal to be digitized.

In the telephone network, for example, PCM encoding is performed at 8kHz. This might seem surprisingly low, but speech frequencies trail off above 3kHz and so the up-to-4kHz represented by 8kHz PCM is perfectly sufficient for intelligible speech. It is not particularly friendly to music, though, which is part of why hold music is the way it is. For this reason, in music and general digital audio a sampling rate of 44.1kHz is conventional due to having been selected for CDs. Audible frequencies are often defined as being "up to 20kHz" although few people can actually hear anything that high (my own hearing trails off at 14kHz, attributable to a combination of age and adolescent exposure to nu metal). This implies a sampling rate of 40kHz; the reason that CDs use 44.1kHz is essentially that they wanted to go higher for comfort and 44.1kHz was the highest they could easily go on the equipment they had at the time. In other words, there's no particular reason, but it's an enduring standard.

Another important consideration in PCM encoding is the number of discrete values that samples can possibly take. This is commonly expressed as the number of bits available to represent each sample and called "bit depth." For example, a bit depth of eight allows each sample to have one of 255 values that we might label -127 through 128. The bit depth is important because it limits the dynamic range of the signal. Dynamic range, put simply, is the greatest possible variation in amplitude, or the greatest possible variation between quiet and loud. Handling large dynamic ranges can be surprisingly difficult in both analog and digital systems, since both electronics and algorithms struggle to handle values that span multiple orders of magnitude.

In PCM encoding, bit depth has a huge impact on the resulting bitrate. 16-bit audio, as used on CDs, is capable of a significantly higher dynamic range than 8-bit audio at the cost of doubling the bitrate. Dynamic range is important in music, but is also surprisingly important in speech, and a bit depth of 8 is actually insufficient to reproduce speech that will be easy to understand.

And yet, due to technical constraints, 8kHz and 8-bit samples were selected for telephone calls. So how is speech acceptably carried over 8-bit PCM?

We need to talk a bit about the topics of compression and companding. There can be some confusion here because "compression" is commonly used in computing to refer to methods that reduce the bitrate of data. In audio engineering, though, compression refers to techniques that reduce the dynamic range of audio, by making quieter sounds louder and louder sounds quieter until they tend to converge at a fixed volume. Like some other writers, I will use "dynamic compression" when referring to the audio technique to avoid confusion. For both practical and aesthetic reasons (not to mention, arguably, stupid reasons), some degree of dynamic compression is applied to most types of audio that we listen to.

Companding, a portmanteau of compressing and expanding, is a method used to pack a wide dynamic range signal into a channel with a smaller dynamic range. As the name suggests, companding basically consists of compressing the signal, transmitting it, and then expanding it. How can the signal be expanded, though, given that dynamic range was lost when it was compressed? The trick is that both sides of a compander are non-linear, compressing loud sounds more than quiet sounds. This works well, because in practice many types of audio show a non-linear distribution of amplitudes. In the case of speech, for example, significantly more detail is found at low volume levels, and yet occasional peaks must be preserved for good intelligibility.

In practice, companding is so commonly used with PCM that the compander is often considered part of the PCM coding. When I have described PCM thus far, I have been describing linear PCM or LPCM. LPCM matches each sample against a set of evenly distributed discrete values. Many actual PCM systems use some form of non-linear PCM in which the possible sample values are distributed logarithmically. This makes companding part of PCM itself, as the encoder effectively compresses and decoder effectively expands. One way to illustrate this is to consider what would happen if you digitized audio using a non-linear PCM encoder and then played it back using a linear PCM decoder: It would sound compressed, with the quieter components moved into a higher-valued, or louder, range.

Companding does result in a loss of fidelity, but it's one that is not very noticeable for speech (or even for music in many cases) and it results in a significant savings in bit depth. Companding is ubiquitous in speech coding.

One of the weird things you'll run into with PCM is the difference between µ-law PCM and A-law PCM. In the world of telephony, a telephone call is usually encoded as uncompressed 8kHz, 8-bit PCM, resulting in the 64kbps bitrate that has become the basic unit of bandwidth in telecom systems. Given the simplicity of uncompressed PCM, it can be surprising that many telephony systems like VoIP software will expect you to choose from two different "versions" of PCM. The secret of telephony PCM is that companding is viewed as part of the PCM codec, and for largely historic reasons there are two common algorithms in use. The actual difference is the function or curve used for companding, or in other words, the exact nature of the non-linearity. In the US and Japan (owing to post-WWII history Japan's phone system is very similar to that of the US), the curve called µ-law is in common use. In Europe and most other parts of the world, a somewhat different curve is used, called A-law. In practice the difference between the two is not particularly significant, and it's difficult to call one better than the other since both just make slightly different trade offs of dynamic range for quantization error (A-law is the option with greater dynamic range and greater possible distortion).

Companding is rarely applied in music and general multimedia applications. One way to look at this is to understand the specializations of different audio codecs: µ-law PCM and A-law PCM are both simple examples of what are called speech codecs, Speex and Opus being more complex examples that use lossy compression techniques for further bitrate reduction (or better fidelity at 64kbps). Speech codecs are specialized for the purpose of speech and so make assumptions that are true of speech including a narrow frequency range and certain temporal characteristics. Music fed through speech codecs tends to become absolutely unlistenable, particularly for lossy speech codecs, which hold music on GSM cellphones painfully illustrates.

In multimedia audio systems, we instead have to use general-purpose audio codecs, most of which were designed around music. Companding is effectively a speech coding technique and is left out of these audio systems. PCM is still widely used, but in general audio PCM is assumed to imply linear PCM.

As previously mentioned, the most common convention for PCM audio is 44.1kHz at 16 bits. This was the format used by CDs, which effectively introduced digital audio to the consumer market. In the professional market, where digital audio has a longer history, 48kHz is also in common use... however, you might be able to tell just by mathematical smell that conversion from 48kHz to 44.1kHz is prone to distortion problems due to the inconveniently large common multiple of the two sample rates. An increasingly commonly used sample rate in consumer audio is 96kHz, and "high resolution audio" usually refers to 96kHz and 24 bit depth.

There is some debate over whether or not 96kHz sampling is actually a good idea. Remembering our Nyquist-Shannon, note that all of the extra fidelity we get from the switch from 44.1kHz to 96kHz sampling is outside of the range detectable by even the best human ears. In practice the bigger advantage of 96kHz is probably that it is an even multiple of the 48kHz often used by professional equipment and thus eliminates effects from sample rate conversion. On the other hand, there is some reason to believe that the practicalities of real audio reproduction systems (namely the physical characteristics of speakers, which are designed for reproduction of audible frequencies) causes the high frequency components preserved by 96kHz sampling to turn into distortion at lower, audible frequencies... with the counterintuitive result that 96kHz sampling may actually reduce subjective audio quality, when reproduced through real amplifiers and speakers. In any case, the change to 24-bit samples is certainly useful as it provides greater dynamic range. Unfortunately, much like "HDR" video (which is the same concept, a greater sample depth for greater dynamic range), most real audio is 16-bit and so playback through a 24-bit audio chain requires scaling that doesn't typically produce distortion but can reveal irritating bugs in software and equipment. Fortunately the issue of subjective gamma, which makes scaling of non-HDR video to HDR display devices surprisingly complex, is far less significant in the case of audio.

PCM audio, at whatever bit rate and bit depth, is not so often seen in the form of files because of its size. That said, the "WAV" file format is a simple linear PCM encoding stored in a somewhat more complicated container. PCM is far more often used as a transport between devices or logical components of a system. For example, if you use a USB audio device, the computer is sending a PCM stream to the device. Unfortunately Bluetooth does not afford sufficient bandwidth for multimedia-quality PCM, so our now ubiquitous Bluetooth audio devices must use some form of compression. A now less common but clearer example of PCM transport is found in the form of S/PDIF, a common consumer digital audio transport that can carry two 44.1 or 48kHz 16-bit PCM channels over a coaxial or fiber-optic cable.

You might wonder how this relates to the most common consumer digital audio transport today, HDMI. HDMI is one of a confusing flurry of new video standards that were developed as a replacement for the analog VGA, but HDMI originated more from the consumer A/V part of the market (the usual Japanese suspects, mostly) and so is more associated with televisions than the (computer industry backed) DisplayPort standard. A full treatment of HDMI's many features and misfeatures would be a post of its own, but it's worth mentioning the forward audio channel.

HDMI carries the forward (main, not return) audio channel by interleaving it with the digital video signal during the "vertical blanking interval," a concept that comes from the mechanical operation of CRT displays but has remained a useful way to take advantage of excess bandwidth in a video channel. The term vertical blanking is now somewhat archaic but the basic idea is that transmitting a frame takes less time than the frame is displayed for, and so the unoccupied time between transmitting each frame can be used to transmit other data. The HDMI spec allows for up to 8 channels of 24-bit PCM, at up to 192kHz sampling rate---although devices are only required to support 2 channels for stereo.

Despite the capability, 8-channel (usually actually "7.1" channel in the A/V parlance) audio is not commonly seen on HDMI connections. Films and television shows more often distribute multi-channel audio in the form of a compressed format designed for use on S/PDIF, most often Dolby Digital and DTS (Xperi). In practice the HDMI audio channel can move basically any format so long as the devices on the ends support it. This can lead to some complexity in practice, for example when playing a blu-ray disc with 7.1 channel DTS audio from a general-purpose operating system that usually outputs PCM stereo. High-end HDMI devices such as stereo receivers have to support automatic detection of a range of audio formats, while media devices have to be able to output various formats and often switch between them during operation.

On HDMI, the practicalities of inserting audio in the vertical blanking interval requires that the audio data be packetized, or split up into chunks so that it can be divided into the VBI and then reassembled into a continuous stream on the receiving device. This concept of packetized audio and/or video data is actually extremely common in the world of media formats, as packetization is an easy way to achieve flexible muxing of multiple independent streams. And that promise, that we are going to talk about packets, seems like a good place to leave off for now. Packets are my favorite things!

Later on computer.rip: MPEG. Not much about the compression, but a lot about the physical representations of MPEG media, such as elementary streams, transport streams, and containers. These are increasingly important topics as streaming media becomes a really common software application... plus it's all pretty interesting and helps to explain the real behavior of terrible Hulu TV apps.

A brief P.S.: If you were wondering, there is no good reason that PCM is called PCM. The explanation seems to just be that it was developed alongside PWM and PPM, so the name PCM provided a pleasing symmetry. It's hard to actually make the term make a lot of sense, though, beyond that "code" was often used in the telephone industry to refer to numeric digital channels.

>>> 2022-02-14 long lines in the Mojave

I have sometimes struggled to justify my love for barren deserts. Why is it that my favorite travel destinations consist of hundreds of miles of sandy expanse? Today, I'm going to show you one reason: rural deserts have a habit of accumulating history. What happens in the desert stays there---in corporeal form. Slow growth of vegetation, little erosion, and extraordinarily low property values turn vast, empty deserts into time capsules... if you spend enough time looking for the artifacts.

Also, we're going to talk about telephones! But first...

While the Mojave Desert was long open to homesteading claims, the extremely arid climate along with the distance to urban centers has always made the area challenging to put to use, and so has kept its population low. Places like Death Valley and Joshua Tree National Park are the best known Mojave destinations for their distinct geography and flora, but a different swath of the Mojave Desert began to attract a less formal sort of attention long ago. The development of the US Highway System, specifically highways 91 and 66, created a pocket of desert that was remote and yet still readily accessible from both Los Angeles and Las Vegas. Post-WWII, power sports (specifically dirt biking) lead to significant use of and impact on open land along highways. Combined, these created a bit of a contradiction: the empty desert was getting a little too crowded.

Through a series of political and administrative moves, the area of the Mojave desert roughly defined by US-91 to the north (now I-15) and US-66 to the south (now I-40 although the alignments vary) became first the East Mojave National Scenic Area of the Bureau of Land Management (the first such National Scenic Area established) and then, in 1994, the Mojave National Preserve of the National Park Service [1]. It is the third largest unit in the national park system, and due to its vast size, history, character, and perhaps most of all, miniscule budget, it remains surprisingly undeveloped and untamed.

Roughly in the center of the Preserve is a tiny town called Kelso. Kelso was established by the Los Angeles and Salt Lake Railroad (later part of Union Pacific) as a railroad depot and base for "helper" locomotives added to trains to help them make it up a steep grade eastwards towards the next tiny settlement, Cima. During its distinguished life as a railroad town, from the 1910s to the 1980s, it also supported a few surrounding mines. Elsewhere in what is now the National Preserve, many small mines, a few large ones, and a few major ranches made up the entire tiny population of the region. The nearest present-day town, Baker, has become little more than a row of gas stations and a surprisingly compelling Greek diner. In an earlier era, the multi-hour trip to Baker on horseback was the only connection to the outside world available to miners and ranchers out in the desert. Back then it was like a different world, and today it largely still is.

Situated roughly halfway between LA and Vegas, in San Bernardino County, California, the Mojave National Preserve's more than one and a half million acres display two wars worth of military collaboration by the Bell System, two generations of long-distance telephone infrastructure, and four distinct types of long-distance telephone lines. There is perhaps nowhere else that one can gain as powerful of an appreciation for the feat that is the long-distance telephone call. A call to Los Angeles requires of you only dialing and waiting. First, though, it required teams of workers digging thousands of huge post holes by hand. The Mojave has been described as "a nowhere between two somewheres" [1]. This is true not only on the ground but also in the wires, as a large portion of telephone calls in and out of one of America's largest cities had to pass through one hundred miles of blowing sand. They still do today.

It's hard to say when, but we can safely how the first telephones arrived in the Mojave: by rail. The railroads made extensive use of telegraphy and, later, telephony. By the 1920s the railroad depot at Kelso, and later some of the homes of railroad employees, were equipped with telephones on the Los Angeles and Salt Lake Railroad's (LA&SR) private system [2]. While railroad telephones operated on separate, wholly railroad-owned infrastructure that was not interconnected with the Bell system, railroad telephone departments enjoyed a close relationship with the Bell System and largely used the same techniques with equipment from the same manufacturers.

The LA&SR would have installed a series of multi-armed utility poles, likely as part of the original construction of the railroad. While these poles would have initially carried only telegraph circuits, they later gained telephone circuits, signal logic circuits, and even "code" circuits which used an early form of digital signaling to communicate with trackside equipment. Many of these circuits would have looked substantially similar to open-wire telephone leads, because they were: railroads employed the same open-wire design that AT&T used.

Railroad telephones went through generally the same technological progression as public telephones. The first equipment installed would have been magneto phones. To make a call, you would turn a crank on the phone which generated a high voltage "ringing" signal. Once an operator noticed and connected to the line, you asked the operator to connect your call. Individual phones were expensive to install. As a result, the Kelso depot started with only a single phone in the dispatcher's office, along with the telegraph. At some point, this telephone was placed in a specially built rotating cabinet that allowed the station agent in the dispatch office to spin it around, presenting it through the other side of the wall for someone in the lobby to take a call [2]. The clever pass-through phone was probably designed by a local worker as a practical solution to the problem that dispatchers often called the phone wanting to speak to visiting train crews, but railroad security policy forbade anyone other than a qualified agent in the dispatch office. The station agent must have quickly tired of relaying conversations sentence-by-sentence through the window.

Later, as the technology progressed and more resources became available, the railroad connected additional phones to other buildings. These extra extensions most commonly appeared in the homes of senior staff such as the station agent and track gang foreman; they would be the only way to reach someone at the depot (or, for that matter, in the entire town of Kelso) during an after-hours emergency. In this era an "extension" was a literal extension of the existing wiring; all of these phones would have rung together. Kelso Depot also featured another clever solution to the difficulties of reaching a remote employee before the widespread availability of radio: after the installation of electrical (CTC) signaling on the rail line, the dispatch office's semaphore display that once electromagnetically dropped flags to alert the agent that a train had passed a signal point approaching the station was rewired to drop a flag whenever the depot phone rang. This way, if the agent missed a call while attending to something outside of the office, they would at least know to call dispatch back when they returned [2].

Elsewhere, in 1915, the first transcontinental telephone line went into service. This line, generally from New York to San Francisco, passed through Northern California far from our area of interest. It ignited a fire that quickly spread, though, and the 1920s saw extensive construction of new long distance telephone lines in the West. In parts of Southern California, Pacific Telephone and Telegraph (PT&T) competed with the Home Telephone Company, until it acquired it. Much of PT&T's advantage over Home was its status as a member of the Bell System: Home had no long distance service. PT&T did.

The history of this early era is difficult to construct, as very few records remain and most of the artifacts have been removed. Around the Mojave National Preserve, though, two PT&T long-distance open-wire toll leads survive. One roughly paralleled US-91 (now I-15) from the area of Los Angeles (probably San Bernardino) towards, ultimately, Salt Lake City. This was one of the most important connections between the West Coast and the rest of the country.

Another, further south, roughly paralleled US-66 (now I-40). This was the fourth transcontinental line, constructed around 1938. Its Western terminus was likely Whitewater, which was less a town and more a particularly important telephone office in the early AT&T system. Whitewater is notable for existing in the territory of General Telephone & Electronics or GTE, a particularly long-lived non-Bell competitive telephone carrier. When GTE was later acquired by a Bell operating company, the two adopted the name Verizon. Back in the pre-WWII era, GTE's dominance in far southern California meant that AT&T had few actual customers but a great need to move long-distance traffic. Whitewater was thus a bit of an oddity: a major telephone office, in the middle of nowhere, with no customers but a surplus of traffic. At the other end, this southern open wire lead probably connected into Bullhead City or Laughlin, in the area of Needles.

Sections of both open wire routes remain and can be seen today. The northern one includes a particularly spectacular crossing of the freeway, employing a seldom seen today technique in which two steel "messenger" ropes suspended between stout poles hold up a series of wooden frames that substitute for poles and crossarms. These "floating poles" supported the actual telephone wires for the long canyon crossing. Today, a single multi-pair cable hangs sadly from the five-arm frames, apparently placed to provide telephone service to a nearby mine after the removal of the open-wire route. On the southern route, remaining poles are clearly visible from US-66 during its departure from the present-day I-40 alignment near Cadiz (itself an oddity, a town owned by a natural resources company with long-failed plans to pump and sell the groundwater to LA).

At the dawn of WWII, these two leads represented some of the only long toll leads on the West Coast. Following the attack on Pearl Harbor, Japanese invasion, or at least opportunistic sabotage, was a major concern for military planners. Concerningly, the handful of telephone connections between major western cities had poor redundancy and were mostly near the coast. They would be easy for a small team of Japanese soldiers, delivered by boat under cover of night, to find and destroy. This easy move would effectively decapitate military leadership on the West Coast, impairing the ability of the United States to mount a defense. Here, years before the nuclear bomb or mutually assured destruction, survivable C2 and the telecom infrastructure to support it became a national priority [3].

In early 1942, and in collaboration with the War Department, PT&T embarked on a project that would rival the first transcontinental: a new wartime long distance route called the Defense Backbone Route, or DBR. Over a span of under a year, with a monumental level of effort ranging from logistical innovations to the sheer manpower of hundreds of laborers working from roving camps, the DBR's open-wire spanned nearly 900 miles from the Mojave desert to Yakima. Most importantly, its entire route was well inland (including a large stretch in Nevada), making it difficult for any military force arriving via the coast to reach. The DBR presaged later developments like the L-3I by providing a survivable toll lead dedicated to military use, particularly for the purposes of defense and reprisal.

The southern terminus of the DBR is sometimes described as Los Angeles, but that seems to have been only by interconnection to the existing open-wire lead along US-66, south of the Preserve. The DBR itself ends at a location optimistically described as Danby, California, although Danby is not much of a town and the end of the DBR is not that near it. The "Danby" station, consisting of one small building which presumably originally contained carrier multiplexing equipment, is still there today, and seems to still be in use with an added microwave antenna. As we will see, telephone infrastructure in rural areas is often reused for cost efficiency. It appears that there are still active customers served by a modern multipair telephone cable installed using the open-wire route's right of way, and the Danby station remains with a microwave link to provide local loop connections to these customers.

From Danby, the DBR continued northwest and then north, almost right through the middle of the present-day Preserve. It wanders through valleys, passing within a few miles of Kelso, before reaching the northern open-wire lead at US-91, which it joins to head towards Las Vegas. The poles and wire of the DBR remain largely intact within the Preserve and can be seen from Kelbaker Road (shortened from Kelso Baker Road), in Kelso and on Kelso Cima Road, and later where it crosses the Mojave Road and numerous dirt mine and ranch roads. It can even be seen from I-15 if you look carefully, although the span near I-40 has been removed.

Not so long after the DBR was constructed, and shortly after it was converted to civilian use, the owners of the Cima mine (a small cinder mine in what is now the eastern portion of the Preserve) contacted PT&T or AT&T to request a telephone. Under a California law, PT&T was required to furnish telephones for use in very rural areas where no other phones existed. At the time, it was a multi-hour trip from the Cima mine to reach Kelso or Baker where, in an emergency, help could be summoned. To improve safety, but moreover to comply with California law, AT&T came to the desert with a "toll station." Toll stations, an artifact of the early phone system no longer seen today, can be thought of as telephones that are a long distance call from anywhere. Toll stations were used to connect very rural areas, and were often located on party lines and required operator assistance to call. The reason for these oddities is simple: toll stations were connected directly to long-distance wires, not via local exchanges, and so they represented an odd exception in the general architecture of the telephone system. On the upside, they were far easier to install in very rural areas than conventional local telephone service.

In 1948, the Cima mine's requested phone was unceremoniously placed dead center in the middle of the desert. Located at the intersection of the DBR and a dirt access road, the new phone was still some way from the Cima mine but much closer than any town. It was a magneto phone connected directly to the DBR (likely using the "voice frequency" or non-carrier "channel" of one of the sets of pairs). Lifting the handset and turning the crank prompted a long-distance operator in San Bernardino to pick up and ask where the user wanted to call. If one wanted to call the phone, they would have to ask their operator to be connected to the San Bernardino long distance operator, and then ask for the phone by name. The long distance operator would ring the phone, and someone would have to be waiting nearby. It is said that some local user left a chair by the phone, as a convenience to those waiting for incoming calls. Telephone users would sometimes adopt a regular schedule of visiting the phone during set hours in case someone wanted to reach them. Locals driving by the phone on the dirt road would roll down their window, just in case it rang, so that they could take a message that would almost certainly be for someone they knew. While far from convenient, it was the only phone service for both mines and ranchers in the area [2,4].

This lonely phone would prove to have remarkable staying power. The owners of the Cima mine seem to have continued to use it as their main telephone into the '90s. Over time the phone was modernized to a more conventional payphone and given a more conventional switching arrangement and phone number. In 1997, after a series of chance discoveries, it came to be widely known in counter-cultural circles as the Mojave Phone Booth. It was difficult to comprehend: an aluminum and glass telephone booth, in a way a symbol of modernity and urbanism, sitting in a lonely desert impossibly far from the civilization it connected to [2,4].

The phone booth's sudden fame, and significant increase in users, lead directly to its demise. Some combination of Park Service concern about environmental impact by visitors to the phone booth and upset by a local rancher who didn't appreciate the raucous visitors lead to its removal in 2000. Today nothing remains of the Mojave Phone Booth except for the DBR itself. Its segment in the northern Preserve, apparently maintained to keep the single connection to the phone booth, is still in good shape today (albeit with only one crossarm remaining) [2]. Unfortunately, while the Mojave Phone Booth is widely described in media ranging from Geocities-esque phone phreaking websites to an episode of 99% Invisible, few people know that the cross-desert phone line its wires once hung off of was itself an oddity, an artifact of WWII which had been hidden from the Japanese in the desert. The Mojave Phone Booth was a contradiction in an even deeper way than it might first seem: a phone placed for convenient access along a phone line placed specifically to avoid convenient access. That is how you get a phone booth in the middle of nowhere.

Elsewhere along the DBR, World War II had ended but the Cold war was just beginning. Early in the Cold War the greatest fear was the delivery of nuclear weapons by bombers. Air defense, before mature radar technology, was a difficult problem. In the style successfully employed by the UK during the battle for Britain, the Air Force established a Ground Observer Corps (actually the second, after one which operated during WWII). The Corps consisted of volunteers who, when activated, would search the sky by eye and ear for enemy aircraft and report them to a "filter center" where such reports were correlated and forwarded to the Air Defense Command.

Being the only thing for a very long ways, the Kelso railroad depot was an obvious choice for a Corps observation post. While not recorded, the volunteers were probably railroad employees who lived in railroad-provided housing in Kelso. There was one problem: the observers would need a telephone to report their sightings to the filter center, and the filter center was not on the railroad telephone network. As a result, the first public network telephone was installed in the lobby of the Kelso railroad depot in 1956 [1,2]. It is unclear today how exactly this phone was connected. I find it likely, although I cannot prove, that it was connected via railroad open-wire leased by AT&T and tied into an AT&T exchange in a larger town. It is also possible that it was a toll station attached to the DBR much like the Mojave Phone Booth, although inspection of the cabling which now exists from the DBR to Kelso suggests that it is a much newer connection than 1956.

In 1974, the Kelso depot phone was apparently still in service although it was likely connected differently (as we will later discuss). A railroad employee, responsible for the operation of the Depot, requested that PT&T move the phone outside under a covered walkway so that it would be accessible 24/7 after they introduced the practice of locking the depot doors at night (this on the advice of a UP police Special Agent who feared a midnight robbery of the depot's small restaurant, formerly 24/7 but by then closed nightly). There was, reportedly, also a phone at Kelso's small general store, across the street from the depot, which likely shared the line [2].

The DBR thus served two local telephones within the Preserve in addition to long-distance traffic. There is some reason to believe that the payphone, at least, was on a party-line circuit shared with phones installed in the homes of some local residents (probably ranch houses relatively near the DBR). The Kelso phone may have been as well, or may have been party-lined with other phones in railroad facilities if it was indeed on leased railroad open-wire. The general pattern of an open-wire toll lead with a single party line used to connect a few toll stations for local users was common in very rural areas at the time. Not just ranch houses and mines but also rural gas stations and businesses relied on this type of service, as toll leads often followed the same highways that rural businesses clustered near.

The Cold War would have a much bigger impact on the Mojave than a phone in the Kelso depot, although the introduction of telephone service to such remote sites as the Mojave Phone Booth and even the Kelso Depot (which did not even have an electrical connection, relying instead on a small on-site power plant operated by the railroad until 1960) was no small feat. The need for long-distance capacity between Los Angeles and the east had grown exponentially. More troubling, the genesis of nuclear weapons and the doctrine of mutually assured destruction created an exponentially greater need for fast, reliable, and survivable telecommunications. The "Flash Override" button on the President's telephone, intended foremost for use when ordering a nuclear strike, would be useless without a telephone network that could connect the President to their military branches... even after nuclear attack, and even through the remote Mojave.

Southern California, outside of its major cities, was rich with military installations (a result in part of the extensive use of the Mojave for WWII training) but poor in infrastructure. This created a particular challenge: in Southern California AT&T suddenly had a bevy of customers for AUTOVON (the Automatic Voice Network, a survivable military telephone system operated by AT&T on contract to the War Department), but very few customers for civilian service and thus very little capability to connect new phones. The Mojave needed strong telephone infrastructure, and it needed it very fast.

I feel fairly confident in stating that the greatest single achievement in the construction of AUTOVON, if not of the entire history of survivable military communications, was the L-3I transcontinental telephone line. This coaxial cable, analog, high capacity toll lead, installed mostly in 1964, could carry thousands of calls from coast to coast. Moreover, it was completely underground and hardened against nearby nuclear detonations. Manned L-3I support facilities, which were found every 100 miles, were underground bunkers staffed 24/7 and equipped with supplies for staff to survive two weeks in isolation. Because it was impractical to harden such facilities against direct nuclear attack, their survivability relied in part on remoteness. The L-3I was intentionally routed through rural areas, well clear of likely targets for nuclear attack. At the same time, it needed reliable connectivity to military installations that were almost certainly targets. This required a hybrid approach of both the hardened L-3I and multiple, redundant connections of other types.

The L-3I transcon adopted a southern route in the western United States and dipped into the western edge of Texas before crossing through the Southwest: New Mexico, Arizona, and then Southern California. There was an underground L-3I "main station" at Kingman, Arizona, after which the cable crossed under the Colorado River at Riviera and proceeded almost directly west into the Preserve. The L-3I cable passed about eight miles north of Kelso, roughly parallel to and not far south of the Mojave Road. After a northern jog it turned back west south of Baker until it met US-91, now I-15, and followed the highway to Beacon Station (which appears to be a former CAA intermediary airfield) where the next L-3I main station is found just north of the freeway.

L-3I cables were buried underground (actually pushed directly into the ground by means of a special plow), but their presence was clearly indicated on the surface by repeater stations and right of way (ROW) markers. Repeater stations were installed in buried concrete vaults every four miles, but a metal hut was installed on top of each vault to house test equipment and provide technicians a clean and dry workspace. ROW markers consisted of frequently placed tall wooden posts with orange metal bands wrapped around the tops, intended both to help repair crews locate the cable in the ground and to warn farmers and construction workers to dig with care.

In the Mojave, though, none of this can be seen today. The L-3I cable saw use through the Cold War, carrying both military and civilian traffic. In the '80s, it was upgraded to carry a digital protocol called P140. The 140Mbps capacity of P140 was limited, though, and the L-3I cable required significantly more maintenance and support than the fiber optic technology increasingly used by 1990. In 1997, AT&T disclosed its intentions to fully abandon the L-3I cable west of Socorro (although portions of the ROW would be reused for fiber). In response, the NPS and BLM performed an environmental analysis on abandonment of the cable in federal land. The analysis revealed several potential long-term environmental impacts from not only the cable and repeaters but also the ROW markers themselves. The marker posts provided an ideal perch for birds of prey, in a desert environment that offered very few other tall objects. The effect was increased predation of small animals, a particular problem for several endangered species in the region.

To mitigate the problem, the NPS required an effort that was rather unusual for L-carrier routes: complete removal. Repeater huts, vaults, ROW marker posts, and the cable itself were all demolished and hauled away throughout most of Arizona and California. Today, all that remains of the L-3I in the Preserve is the still visible scar of the trenching and excavation, marked on many maps as "Utility Road."

The western destination of the L-3I cable was the L-3I Main Station at Mojave, around one hundred miles west of the Preserve, which despite the retirement of the L-3I has grown into a large complex that remains an important long-distance switching center today. The AUTOVON switch at Mojave connected via microwave and buried cable to a long list of cities and military installations in Southern California.

Microwave radio systems can move large numbers of telephone calls over point-to-point radio links. Because they avoid the need to run cable through many miles of land, microwave can be a much more affordable option for long-distance telephone infrastructure. As a downside, the microwave technology of the time (called TD-2) could carry fewer calls and required a line-of-sight of generally under 50 miles between stations. To extend this range, simple stations called relays received a signal on one side and transmitted it again out the other. The microwave antennas used at the time were large, heavy, and required fairly regular maintenance, all of which made very tall towers impractical. Instead, to find a line of sight, microwave facilities were built on peaks and in high mountain passes.

Much of the microwave telephone infrastructure in the area of the Preserve was built at the same time as the L-3I cable, as both were part of the same larger AUTOVON project. The L-3I was a high-capacity, high-durability, but high-cost backbone, while microwave links were a more affordable technology that used redundancy rather than physical strength to ensure reliability. The lower cost and greater flexibility of microwave also made it ideal for shorter connections between the telephone network and defense facilities, encouraging more local microwave stations. This is why the mid-1960s AUTOVON effort lead to the creation of not one, but three independent east-west long-distance routes through the area of the Preserve: the L-3I cable, a northern microwave route, and a southern microwave route.

Although they were relatively inexpensive, microwave stations were not left undefended. AUTOVON microwave facilities were above ground but used hardened building techniques including thick concrete walls, blast shielded vents, and reinforced towers and antennas to survive nuclear strikes at a moderate distance. Most microwave stations were simple relays that operated unattended except for periodic visits by maintenance technicians, but larger stations with switching equipment were staffed 24/7 and supplied for two weeks of isolation, much like L-3I main stations.

The center of the microwave network in the Mojave, if not all of Southern California, was a remote mountaintop site called Turquoise. Located just north of the preserve, about five miles north of I-15 at Halloran Springs, Turquoise was a staffed facility with an AUTOVON switch. Its squat square tower bristled with horn antennas. Every day several shifts of exchange technicians made their way up Halloran Springs Road to the site and supervised the switching of military and civilian calls to destinations throughout southern California, as well as onto the L-3I and other cables for cross-country transit. Turquoise had direct or indirect connections to four different major long-distance microwave routes. As a secondary function, Turquoise included Ground Entry Point antennas for a system called ECHO FOX that provided radiotelephone service to Air Force One... directly to AUTOVON, for use in a nuclear emergency if necessary [6].

One pair of antennas on Turquoise's crowded tower pointed south towards a station called Kelso Peak. Kelso Peak, located in the Preserve a little under ten miles northwest of Kelso, served as a relay station on both a north-south route (north to Turquoise) and an east-west route (west to Hector, a relay not very close to anything but perhaps closest to Ludlow).

To the east, Kelso Peak connected to Cima, another AT&T relay in the Preserve. Cima station sits on a hill five miles due East of the town of Cima, and relays traffic northeast to a station in Nevada, almost to Las Vegas, charmingly called Beer Bottle.

To the south, Kelso Peak connects to the Granite Pass station. Granite Pass is directly next to Kelbaker Road 14 miles south of Kelso. Across Kelbaker Road from the Granite Pass microwave station is a much smaller tower installed by the National Park Service to relay internet and phone service to Kelso today. South from Granite Pass, the next station is Sheep Hole, south of the preserve and northeast of Twentynine Palms.

Putting these stations together, the northern east-west telephone route through the Mojave runs north of the preserve mostly parallel to I-15. The southern one (actually perhaps better called the "middle" route as there is yet another further south and nearer to Joshua Tree National Park) runs through the center of the Preserve, and would be visible on the horizon just north of Kelso if you could see microwaves. The north-south route runs directly through the western side of the Preserve.

Construction of these stations in 1964 was a formidable project. Crews from Southern California Edison installed many miles of new power lines, starting from Kelso and running outwards, to bring power to the Granite Pass and Kelso Peak stations (Kelso itself had only been connected to the grid a few years earlier). The microwave stations, like the L-3I cable, were built by PT&T crews for AT&T Long Lines. Crews from both SoCal Edison and PT&T occupied employee hotel rooms at the Kelso Depot while performing work, often for months long stays that irritated the station agent and stretched the depot's capacity to house and feed [2].

Each of the AT&T stations in the Preserve, like others in the Mojave, included an unusually large gravel apron around the facility. This leveled gravel lot served as a helipad; due to the remoteness of the facilities maintenance technicians were delivered by helicopter. The sandstorms which sometimes occur in the Preserve posed a maintenance and reliability challenge, and maintenance crews were kept busy.

Along with the L-3I and TD-2 came the end of open-wire, but in such a remote area it's hard to really tear out old infrastructure. Instead, when the decision was made to decommission much of the open wire by 1989, alternative arrangements were made for the few local customers once served by the DBR. South of Granite Pass and north of Kelso portions of the DBR were removed, but the Granite Pass microwave station was connected to the DBR open-wire as it passed by. East of Kelso where the DBR crossed the railroad, a section of new wire was used to connect pairs on the DBR to pairs on the railroad open-wire, which was removed except between the DBR and Kelso. The result was a direct local loop from the Kelso phones to the Granite Pass microwave station, a very unconventional setup to avoid the need for a new phone line to Kelso. A 1924 rail depot connected to a 1964 microwave station by reuse of a 1942 open-wire toll lead: the kind of thing you run into in the desert.

The phone booth seems to have found an alternate arrangement. The DBR was removed for a span north of Kelso, disconnecting the phone booth from the span to Granite Pass. Instead, the phone booth seems to have been reconnected along old DBR wire pairs to somewhere further north, likely Baker, where the phone booth had been assigned a regular local number.

Because of these two legacy arrangements, large spans of the DBR have remained intact in the Preserve to this day. On Kelso Cima Road just east of Kelso, the intersection of the DBR and the railroad can be seen, including the somewhat awkward interconnection of the DBR to the railroad open wire. Just north of this point the DBR abruptly ends, the remaining wires tied around the base of the last pole. The DBR is only absent for about two miles though, follow its route north and the poles will start again just as abruptly as they ended. 16 miles further, the ghost of the phone booth sits under the poles of the former DBR. Look carefully and you can see many details of this old infrastructure. I have posted a few photos I took at https://pixelfed.social/jbcrawford, although I intend to get better and more thorough ones on a future trip.

Today, the original TD-2 microwave equipment is long removed, and some of the large KS-15676 horn antennas have been removed as well (although they remain at some sites including the highly visible Granite pass). Even so, radio sites, once built, have a tendency to live on. Most of these microwave sites are still in use, either by a telco or under ownership of a leasing company such as American Tower. The remoteness of the Mojave means that radio remains an important technology and many of these microwave sites still carry telephone calls, using more modern equipment, and either as the primary route to some rural telephone exchanges or as a backup in case of damage to buried fiber optic lines. The late life of these facilities can sometimes be confusing. At Granite Pass, a much newer tower and small utility enclosure on the west side of Kelbaker road, next to the small NPS relay, are used by AT&T for telephone service. The original AT&T tower on the east side of the road is no longer used by AT&T but lives again as a relay site for the Verizon Wireless microwave backhaul network, which provides cell towers their connection to the rest of the phone system. Many microwave sites have also been reused as cellular towers.

Various newer telecommunications infrastructure can be found within the Preserve as well. At the town of Cima, a large radio tower erected by Union Pacific provides radio communications with trains and trackside equipment. Smaller towers found up and down the railroad link trackside equipment together as a replacement for the old open-wire lines. Just south of I-40 near the Essex exit, a modern small microwave relay site provides backhaul communications for several cellular carriers on solar power alone. At Goffs Butte, a conspicuous cinder cone south of Goffs, a busy radio site includes cellular and telephone microwave relays alongside broadcast radio stations. Cellular towers at Baker, Mountain Pass, Goffs, Ludlow, and others now provide coverage to some, but far from all, of the Preserve.

There is a very real sense, though, in which modern telecommunications technology has still failed to tame the desert. Satellite networks such as Globalstar and Iridium can be reached throughout the Preserve, but slowly and at significant cost. Cellphones are unreliable to unusable in many parts of the Preserve, and there are few landline phones to be found. Despite all of this infrastructure, the Mojave is still far from civilization. That's another great thing about the open desert, besides the memories it keeps: it's hard to get to, and even harder for anyone to bother you once you're there.

[1] "From Neglected Space to Protected Place: An Administrative History of the Mojave National Preserve," Eric Charles Nystrom for the National Park Service, 2003.

[2] "An Oasis for Railroaders in the Mojave: The History and Architecture of the Los Angeles and Salt Lake Railroad Depot, Restaurant, and Employees Hotel at Kelso, California, on the Union Pacific System," Gordon Chappel et al. for the National Park Service, 1998.

[3] "DBR: 'Damned Big Rush' or the Building of the Defense Backbone Route," The Electric Orphanage. https://the-electric-orphanage.com/the-damned-big-rush-or-building-of-the-defense-backbone-route/

[4] Correspondence, Telephone Collectors International mailing list.

[5] "Draft Environmental Impact Statement: Mojave National Preserve, P140 Coaxial Cable Removal Project," National Park Service, 1997.

[6] "Turquoise, California AT&T Long Lines Site," Path Preservation. http://www.drgibson.com/towers/turquoise.html

>>> 2022-01-24 the smart modem

I think I've mentioned occasionally that various devices, mostly cellular modems, just use the Hayes or AT command set. Recently I obtained a GPS tracking device (made by Queclink) that is, interestingly, fully configured via the Hayes command set. It's an example of a somewhat newer trend of converging the functionality of IoT devices into the modem baseband. But what is this Hayes command set anyway?

Some of you are no doubt familiar with the "acoustic coupler," a device that has two rubber cups intended to neatly mate with the speaker and microphone of a telephone handset. The acoustic coupler allowed a computer modem to be connected to the telephone system via audio instead of electrically, which was particularly important because, pre-Carterfone, nothing could be connected to the telephone system that was not leased from the telco. Acoustic couplers were also just convenient, as back in the days when all equipment was leased from the telco phones were fairly expensive, most houses did not yet have a panoply of telephone jacks, and so it was just generally handy to be able to easily use a normal phone with computer modem without having to swap around cabling.

Unfortunately, this scheme had a major limitation: the computer interacted with the telephone just like you would, via audio. The computer had no way, though, of taking the phone on or off hook or dialing. That was all up to the user. So, you'd pick up the phone, dial a number, and then set the phone down on the acoustic coupler. When you were done, you would take the phone off of the coupler and hang it back up. Besides being a bit of a hassle and sometimes prone to mistakes, this effectively ruled out any kind of automatic or scheduled modem usage.

Through the '70s, modems capable of automatic dialing and on/off hook were available but were expensive, large machines intended for commercial-scale use. For example, they were somewhat widely used by retail point of sale systems of the era to send regular reports back to corporate headquarters for accounting. For the home computer enthusiast, there were essentially no options, and among other implications this ruled out the BBS ecosystem that would emerge later since there was no way for a computer to automatically pick up the line.

Everything changed in 1981. Actually, the first fully computer-controlled modem came somewhat earlier, but because it was designed specifically for S-100 computers (like the Altair) and later Apple II, its popularity was limited to those platforms. Hayes, the same company that developed this early internal modem, released the Hayes Smartmodem in '81---which truly started the PC modem revolution. The basic change from their earlier internal modems was that the Smartmodem interfaced with the host computer via serial. RS-232-esque-ish serial ports were by this time ubiquitous on microcomputers, so the Smartmodem could be used with a huge variety of hardware.

It might be surprising that a modem that allowed programmatic control of the hook and dialing took so long to come around. It might be more obvious why if we think about the details of the modem interface to the host PC. The task of a modem is, of course, to send and receive data. In order to do so, modems have traditionally acted like transparent serial channels. In other words, modems have behaved as if they were simply very long serial cables between two computers. Whatever data was sent to the modem it transmitted, and whatever data it received it returned on the serial interface.

We could thus refer to the serial connection to the modem as being the data plane. How is the modem commanded, then? Well, originally, it wasn't... the user had to handle all aspects of call control manually. To bring about automatic call control, Hayes had to come up with a command set for the modem and a way to send those commands. Hayes solution is one that vi users will appreciate: they created two modes. A Hayes Smartmodem, in data mode, acted like a normal modem by simply sending and receiving data. A special escape sequence, though, which defaulted to "+++", caused the modem to change to command mode. Once in command mode, the computer could send various commands to the modem and the modem could reply with status information. The modem would switch back to data mode either after an explicit mode switch command or implicitly after certain connection setup commands.

All commands to a Hayes modem began with the letters "AT". There are a few reasons for this. Perhaps most obviously (certainly to any vim users), the use of two distinct modes creates a huge opportunity for "mode errors" in which the modem is somehow not in the mode that the software controlling it thinks it is. Prefixing all command strings with "AT" serves as an additional check that a line of text is intended to be a command is not actually data errantly sent during command mode, which might cause the modem to take all kinds of strange actions. Second, AT was used for automatic baud detection and clock recovery in the modem, since it was a known bit sequence that would be sent to the modem after the modem first powered on and before it was used to make a call.

It's because of this "AT" prefix, which in principle stands for "attention," that the Hayes command set is commonly referred to as the AT commands. If either Hayes or AT rings a bell, it will be because the influence of the Hayes Smartmodem on the computer industry has been incredibly long lasting: essentially all telephone network modems, whether landline or cellular, continue to use the exact same Hayes interface. In most cases, the operating system on your smartphone is, as we speak, using the Hayes command set to interact with the cellular baseband. If you buy an LTE module for something like an IoT application, you will need to send it Hayes commands for setup (under Linux the ModemManager daemon is responsible for this background work). If you use a USRobotics parallel telephone modem, well, you will once again be using the Hayes command set, but then that's less surprising.

Let's take a quick look at the Hayes commands. The format of them is somewhat unconventional and painful by modern standards, but keep in mind that it was intended for easy implementation in '80s hardware, serial interfaces were slow back then, and in general it is designed more for economy than user-friendliness. On top of that, it's been hugely extended over the years since, meaning that the awkward "extension" commands are now very common.

A Hayes command string starts with "AT" and ends with a carriage return (line feed may or may not be used depending on configuration). In between, a command string can contain an arbitrary number of commands concatenated together---there is no whitespace or other separation, rather the command format ensures that it is possible to determine where a command ends. You can imagine this has become a bit awkward with extension commands and so, in practice, it's common to only put one command per line except for rather routing multi-step actions.

The basic commands consist of a single capital letter which is optionally followed by a single digit. Most of the time, the letter indicates an action while the digit indicates some kind of parameter, but there are exceptions. Some commands take an arbitrary-length parameter following the command, and some commands accept a letter instead of the one trailing digit. Actually even the original Hayes command set is so inconsistent that it's hard to succinctly describe the actual syntax, and now it's been added on to so many times that exceptions outweigh the rules. It might be easier to just look at a few examples.

To do perhaps the most obvious thing, instruct the modem to go off hook and dial a telephone number, you send "ATDT" (D=Dial, T=Touch-Tone) followed by a string which specifies the phone number... and can also contain dialing instructions such as pausing and waiting for ringback. For example when dialing into a PABX that uses extensions, you might use "ATDT5055551234@206;". This tells the modem to dial (D), touch-tone (T), 505-555-1234, wait until ringback (@), dial 206, then stay in command mode (;). Without the semicolon, the D command usually causes an implicit switch to data mode.

Answering a call is simpler, since there are fewer parameters. The "A" command is answer. The command string would sort of technically be "ATA0" since A ostensibly conforms to the "one letter one digit" convention, but when the digit is 0 it can be omitted.

But wait... how would the computer know that the modem is "ringing" in order to answer? Well, for that you'll have to jump back to the post on RS-232, and study up on why the Hayes Smartmodem used a 25-pin connector. There's just a dedicated wire to indicate ringing, as well as a dedicated wire to indicate when the modem is ready to move data (i.e. when a data carrier is present). The serial interface in the computer was expected to expose the state of these pins to software as needed.

Some of you may remember that, in the days of dial-up, it was common to hear the modem dial and negotiate the data connection aloud. This too dates back to the Hayes Smartmodem, and it's somewhat related to the reason that fax machines usually provide a handset. If you misdial or there is a problem with the destination phone number or one of a number of other things, you may get an intercept message or someone answering or some other non-modem audio upon the call connecting. The Smartmodem featured a speaker to allow the user to hear any such problems, but of course few users wanted to listen to the whole data session. The Hayes "M" command allowed the host computer to set the behavior of the speaker, and "ATM1" was commonly sent which caused the modem to enable the built-in speaker until a data carrier was established, at which point it was muted.

The Hayes Smartmodem also included a number of registers in which configuration could be stored in order to affect the behavior of later commands. For example, the duration of a standard dialing pause could be adjusted by changing the value in the register. The "S" command allowed for selecting a register (e.g. ATS8 to select register 4), and the "?" and "=" commands could be used to query and set the value of a register. "=" of course took an argument, and so "ATS8=8" could be used to set the pause duration to 8 seconds. This might look like one long command but it's not, we could just as well send "ATS8" followed by "AT=8". The = is a command, not an operator.

As modems became faster and more capable and gained features, the Hayes command set gained many additions and variants. While the core commands remain very consistently supported, the prefixes "&", "%", "\", and "+" are all used to indicate various extended commands. Some of these are defined by open standards, while others will be proprietary for the modem manufacturer. For example, the GSM standard specifies extended Hayes commands useful for interacting with cellular modems. For example, "AT+CSQ" can be used to ask a cellular modem for the current signal strength (RSSI). The "+" prefix is, in general, used for ITU-standardized additional commands, and "+C" used for commands related to cellular modems. You'll see these prefixes very frequently today, as the Hayes command set is more and more seen in the context of cellular modems rather than telephone modems.

Of course, "+CSQ" being a command seems to violate the syntax I explained earlier for Hayes commands, and vendor proprietary commands frequently take this much further by introducing multi-parameter commands with parameter separators and all types of lengthier command names. For example, for a personal project I wrote software around a Telit LTE module that made use of the command string "AT#CSURV" (note non-standard prefix "#"). This command causes the modem to search for nearby cells and return a listing of cells with various parameters, which is useful for performing site surveys for cellular network reliability.

Many modern cellular modems have GPS receivers built-in, and it's possible to use the GPS receiver via Hayes commands. On the Telit module, a command string of "AT$GPSACP" causes the modem to return the current position, while the command string "AT$HTTPGETSTSEED=1,2199" (note two parameters) can be used to command the embedded GNSS module to load AGPS data from an HTTP source (the details of AGPS will perhaps be a future topic on this blog).

Brief tangent: some of you may be aware (perhaps I have mentioned it before?) that dialing emergency calls on GSM and LTE cellphones is, well, a little weird. Much of that is because the GSM specifications have built-in support for emergency calling, independent of phone numbers, that is intended to allow cellular phones to present a consistent emergency calling method regardless of the dialing conventions of a country/area the user might be roaming in. The exact commands are unstandardized, but on the Telit module "AT#EMRGD" initiates an emergency call (note that no phone number is specified) while "AT#EMRGD?" (it is a common convention in extended AT commands for a trailing "?" to change the command to a status check) causes the modem to report which phone numbers the GSM network has indicated should be used for different types of emergency calls---chiefly for display to the user. This is why dialing common international emergency numbers like 999 and 110 on a US cellular phone still results in a connection to 911---in actuality no dialing happens at all, when the dialer app determines that the number entered appears to be an emergency call it instead issues an AT command with no phone number at all. Part of the reason for this is due to enhanced GSM features for position reporting, which relate to what is called "e911" in the US and provide essentially a basic, slow data channel between a cellular phone and a PSAP that can be used by the phone operating system to report a GPS position to the PSAP. There are, of course, a half dozen AT commands on most cellular modems to facilitate this [1].

Now keep in mind that all of these commands happen over a channel that is also intended to send data. So, after dialing a call or by issuing the command string "ATO" (go online) further data sent over the serial connection will instead go "through" the modem to the other end. In practice, though, mode switching introduces a set of practical problems (not least of which is having to make sure the escape sequence "+++" does not appear in data) and so most modern modems actually don't do it any more. Instead, the Hayes protocol serial connection is usually used purely for modem commanding and a separate data channel is used for payload.

This is clearest if we look at the most common modern incantation of Hayes commands, a cellular modem connected to a host running Linux. Traditionally, ModemManager would issue a set of commands to the modem to set up the connection after which it would place the modem into data mode and then trigger pppd to establish a ppp connection with the modem serial device. In practice, most cellular modems today are "composite devices" in some sense (i.e. present multiple independent data channels, whether physically or as a virtual product of their driver) and appear as both a serial device and a network interface. The serial device is for Hayes commands, the network interface is, well, a plain old network interface, which makes network setup rather easier than having to use PPP. There are various ways that this happens mechanically; in the case of USB modems it is usually by presenting a composite USB device that includes some type of network interface profile like CDC Ethernet [2].

In fact, a lot of modems don't just present a serial interface and a network interface... it's not unusual for modems to present several. One will be for Hayes commands, but there's often a second to be used as a dedicated channel for PPP over serial (in case a different method of network connection isn't used) and then a third dedicated to GPS use. Since applications often want regular (unsolicited) updates from the GPS module and it's a bit silly to have to constantly poll via Hayes command or switch modes around, it's common for LTE modems to allow the host to issue a Hayes command that enables unsolicited GPS updates, after which they continuously generate GPS fix messages on a dedicated channel. These are usually in NMEA format, a widespread standard for GNSS information over simple serial channels that was originally developed to allow a single GNSS receiver on a boat to disseminate position information to multiple navigation devices. Yes, specifically a boat---NMEA is the National Marine Electronics Association, but they came up with a solid standard first and everyone else has copied it.

Despite the partial shift away, Hayes commands have a lot of staying power due to their simplicity. Some devices are going to the direction of using more Hayes commands, since it can actually eliminate the need for any "data channel proper" in some cases. Many LTE modems oriented towards IoT or industrial use provide extension Hayes commands that perform high level actions like "make a POST request". The modem implements HTTP internally so that developers of embedded devices don't have to. The Telit modules even support HTTPS, although setting up the TLS trust store is a bit of a pain.

The latest hotness in cellular modules is the ability to load new functionality at runtime. IOT LTE modems made by Digi, for example, include extra non-volatile and volatile storage and a MicroPython runtime so that business logic can run directly in the modem. You can bet that there are Hayes commands involved.

So 40 years later, a huge variety of modern electronics using cutting-edge cellular data networks are still, at least for initial setup, pretending to be a 300-baud Hayes Smartmodem. Maybe you can still find a case out there where coercing another computer to attempt to send "+++" followed by, after a sufficient pause, "ATH" will cause it to drop off the network.

A final tangent on Hayes commands, and what brought them to my mind: through a combination of good luck and force of will I have managed to get a dealership to take my money for a new car (this proved astoundingly difficult). Since Albuquerque is quite competitive in its effort to regain the recently lost title of Car Theft Capitol of the USA I have fit it with a tracking device. This device, made by a small Chinese outfit, runs the entirety of its logic within a modem with extended firmware. This is sometimes called a "microcontroller-less" design, although obviously the modem is essentially functioning as a microcontroller in this case. For configuration, the tracker exposes the modem's Hayes serial interface on an external connector, and the vendor provides a software tool that generates very long Hayes command strings to configure the tracker behavior (endpoint, report frequency, immobilize logic, etc). It's possible to use AT commands on this interface to send and receive SMS, for example, which makes the tracker a more flexible device than it advertises.

Actually, I lied, one more tangent: Wikipedia notes that the Smartmodem used a novel design of an extruded aluminum section that the PCB slid into, and a plastic cap on each end. This was an extremely common case design for '90s computer accessories. Cheaper plastic injection molding seems to have mostly killed it off, but it was super convenient to take these types of cases apart and I rather miss them now.

[1] In fact a new and somewhat upcoming GSM feature called "eCall" enables "data-only" emergency calls, mostly intended for use by in-vehicle assistive technologies that may connect an emergency call and then send a position and status report under the assumption that the occupants may be incapacitated and unable to speak.

[2] Note that newer modems and operating systems are starting to use MBIM more often, a newer USB profile that includes a newer command channel. If you have an LTE modem and do not see the expected Hayes serial device, MBIM may be the reason... but on most modems a Hayes command must be issued to switch the modem to MBIM mode, so the Hayes command set is still in use even if only briefly.

>>> 2022-01-16 peer to peer but mostly the main peer

I have admitted on HN to sometimes writing computer.rip posts which are extensions of my HN comments, and I will make that admission here as well. A discussion recently came up that relates to a topic I am extremely interested in: the fundamental loss of peer-to-peer capability on the internet and various efforts to implement peer-to-peer distributed systems on top of the internet.

Of course, as is usual, someone questioned my contention that there really is no such thing as a distributed system on the internet with the example of Bitcoin. Having once made the mistake of a graduate thesis on Bitcoin implementation details it is one of the things I feel most confident in complaining about, and I do so frequently. Rest assured that Bitcoin's technical implementation is 1) a heaping pile of trash which ought to immediately dispel stories about Nakamoto being some kind of engineering savant, and 2) Bitcoin is no exception to the fundamental truth that the internet does not permit distributed systems.

But before we get there we need to start with history... fortunate since history is the part I really care to write about.

Some time ago I mentioned that I had a half-written blog post that I would eventually finish. I still have it, although it's now more like 3/4 written. The topic of that post tangentially involves early computer networks such as ARPANET, BITNET, ALOHAnet, etc. that grew up in academic institutions and incubated the basic concepts around which computer networks are built today. One of my claims there is that ARPANET is overrated and had a smaller impact on the internet of today than everyone thinks (PLATO and BITNET were probably both more significant), but it is no exception to a general quality most of these early networks had that has been, well, part of the internet story: they were fundamentally peer to peer.

This differentiation isn't a minor one. Many early commercial computer networks were extensions of timeshare multiple access systems. That is, they had a strictly client-server (or we could actually call it terminal-computer) architecture in which service points and service clients were completely separated. Two clients were never expected to communicate with each other directly, and the architecture of the network often did not facilitate this kind of use.

Another major type of network which predated computer networks and had influence on them were telegraph networks. Telegraph systems had long employed some type of "routing," and you can make a strong argument that the very concept of packet switching originated in telegraph systems. We tend to think of telegraph systems as being manual, morse-code based networks where routing decisions and the actual message transfer were conducted by men wearing green visors. By the 1960s, though, telegraph networks were gaining full automation. Messages were sent in baudot (a 5-bit alphabetical encoding) with standardized headers and trailers that allowed electromechanical equipment and, later, computers to route them from link to link automatically. The resemblance to packet switched computer networks is very strong, and by most reasonable definitions you could say that the first wide-scale packet network in the US was that of Western Union [1].

Still, these telegraph networks continued to have a major structural difference from what we now consider computer networks. Architecturally they were "inverted" from how we think of hardware distribution on computer networks: they relied on simple, low-cost, low-capability hardware at the customer site, and large, complex routing equipment within the network. In other words, the "brains of the operation" were located in the routing points, not in the users equipment. This was useful for operators like Western Union in that it reduced the cost of onboarding users and let them perform most of the maintenance, configuration, upgrades etc. at their central locations. It was not so great for computer systems, where it was desirable for the computers to have a high degree of control over network behavior and to minimize the cost of interconnecting computers given that the computers were typically already there.

So there are two significant ways in which early computer networks proper were differentiated from time-sharing and telegraph networks, and I put both of them under the label of "network of equals," a business term that is loosely equivalent to "peer to peer" but more to our point. First, a computer network allows any node to communicate with any other node. There is no strict definition of a "client" or "server" and the operation of the network does not make any such assumptions as to the role of a given node. Second, a computer network places complexity at the edge. Each node is expected to have the ability to make its own decisions about routing, prioritization, etc. In exchange, the "interior" equipment of the network is relatively simple and does not restrict or dictate the behavior of nodes.

A major manifestation of this latter idea is distributed routing. Most earlier networks had their routes managed centrally. In the phone and telegraph networks, the maintenance of routing tables was considered part of "traffic engineering," an active process performed by humans in a network operations center. In order to increase flexibility, computer networks often found it more desirable to generate routing tables automatically based on exchange of information between peers. This helped to solidify the "network of equals" concept by eliminating the need for a central NOC with significant control of network operations, instead deferring routing issues to the prudence of the individual system operators.

Both of these qualities make a great deal of sense in the context of computer networking having been pioneered by the military during the Cold War. Throughout the early days of both AUTODIN (the military automated telegraph network) and then ARPANET which was in some ways directly based on it, there was a general atmosphere that survivability was an important characteristic of networks. This could be presented specifically as survivability in nuclear war (which we know was a key goal of basically all military communications projects at the time), but it has had enduring value outside of the Cold War context as we now view a distributed, self-healing architecture as being one of the great innovations of packet-switched computer networks. The fact that this is also precisely a military goal for survival of nuclear C2 may have more or less directly influenced ARPANET depending on who you ask, but I think it's clear that it was at least some of the background that informed many ARPANET design decisions.

It might help illustrate these ideas to briefly consider the technical architecture of ARPANET, which while a somewhat direct precursor to the modern internet is both very similar and very different. Computers did not connect "directly" to ARPANET because at the time it was unclear what such a "direct" connection would even look like. Instead, ARPANET participant computers were connected via serial line to a dedicated computer called an interface message processor, or IMP. IMPs are somewhat tricky to map directly to modern concepts, but you could say that they were network interface controllers, modems, and routers all in one. Each IMP performed the line coding to actually transmit messages over leased telephone lines, but also conducted a simple distributed routing algorithm to determine which leased telephone line a message should be sent over. The routing functionality is hard to describe because it evolved rapidly over the lifetime of ARPANET, but it was, by intention, both simple and somewhat hidden. Any computer could send a message to any other computer, using its numeric address, by transferring that message to an IMP. The IMPs performed some internal work to route the message but this was of little interest to the computers. The IMPs only performed enough work to route the message and ensure reliable delivery, they did not do anything further and certainly nothing related to application-level logic.

Later, ARPANET equipment contractor BBN, along with the greater ARPANET project, would begin to openly specify "internal" protocols such as for routing in order to allow the use of non-BBN IMPs. Consequentially, much of the functionality of the IMP would be moved into the host computers themselves, while the routing functionality would be moved into dedicated routing appliances. This remains more or less the general idea of the internet today.

Let's take these observations about ARPANET (and many, but not all, other early computer networks) and apply them to the internet today.

Peer-to-Peer Connectivity

The basic assumption that any computer could connect to any other computer at will proved problematic by, let's say, 1971, when an employee of BBN created something we might now call a computer worm as a proof of concept. The problem was far greater as minicomputers and reduced connectivity costs significantly increased the number of hosts on the internet, and by the end of the '90s network-based computer worms had become routine. Most software and the protocols it used were simply not designed to operate in an adversarial climate. But, with the internet now so readily accessible, that's what it became. Computers frequently exposed functionality to the network that was hazardous when made readily available to anonymous others, with Windows SMB being a frequent concern [2].

The scourge of internet-replicating computer worms was stopped mostly not by enhancements in host security but instead as an incidental effect of the architecture of home internet service. Residential ISPs had operated on the assumption that they provided connectivity to a single device, typically using PPP over some type of telephone connection. As it became more common for home networks to incorporate multiple devices (especially after the introduction of WiFi) residential ISPs did not keep pace. While it was easier to get a subnet allocation from a residential ISP back then than it is now, it was very rare, and so pretty much all home networks employed NAT as a method to make the entire home network look, to the internet, like a single device. This remains the norm today.

NAT, as a side effect of its operation, prohibits all inbound connections not associated with an existing outbound one.

NAT was not the first introduction of this concept. Already by the time residential internet was converging on the "WiFi router" concept, firewalls had become common in institutional networks. These early firewalls were generally stateless and so relatively limited, and a common configuration paradigm (to this day) was to block all traffic that did not match a restricted set of patterns for expected use.

Somewhere along this series of incremental steps, a major change emerged... not by intention so much as by the simple accretion of additional network controls.

The internet was no longer peer to peer.

Today, the assumption that a given internet host can connect to another internet host is one where exceptions are more common than not. The majority of "end user" hosts are behind NAT and thus cannot accept inbound connections. Even most servers are behind some form of network policy that prevents them accepting connections that do not match an externally defined list. All of this has benefits, there is an upside, but there is also a very real downside, which is that the internet has effectively degraded to a traditional client-server architecture.

One of the issues that clearly illustrated this to many of my generation was multiplayer video games. Most network multiplayer games of the '90s to the early '00s were built on the assumption that one player would "host" a game and other players would connect to them. Of course as WiFi routers and broadband internet became common, this stopped working without extra manual configuration of the NAT appliance. Similar problems were encountered by youths in peer-to-peer systems like BitTorrent and, perhaps this will date me more than anything else, eD2k.

But the issue was not limited to such frivolous applications. Many early internet protocols were designed with the peer-to-peer architecture of the internet baked into the design. FTP is a major example we encounter today, which was originally designed under the assumption that the server could open a connection to the client. RTMP and its entire family of protocols, including SIP which remains quite relevant today, suffer from the same basic problem.

For any of these use-cases to work today, we have had to clumsily re-invent the ability for arbitrary hosts to connect to each other. WebRTC, for example, is based on RTMP and addresses these problems for the web by relying on STUN and TURN, two separate but related approaches to "NAT negotiation." Essentially every two-way real-time media application must take a similar approach, which is particularly unfortunate since TURN introduces appreciable overhead as well as privacy and security implications.

Complexity at the Edge

This is a far looser argument, but I think one that is nonetheless easy to make confidently: the concept of complexity at the edge has largely been abandoned. This happens more at the application layer than at lower layers, since the lower layers ossified decades ago. But almost all software has converged on the web platform, and the web platform is inherently client-server. Trends such as SPAs have somewhat reduced the magnitude of the situation as even in web browsers some behavior can happen on the client-side, but a look at the larger ecosystem of commercial software will show you that there is approximately zero interest in doing anything substantial on an end-user device. The modern approach to software architecture is to place all state and business logic "in the cloud."

Like the shift away from P2P, this has benefits but also has decided disadvantages. Moreover, it has been normalized to the extent that traditional desktop development methods that were amenable to significant complexity at the client appear to be atrophying on major platforms.

As the web platform evolves we may regain some of the performance, flexibility, and robustness associated with complexity at the edge, but I'm far from optimistic.

Peer-to-Peer and Distributed Applications Today

My contention that the internet is not P2P might be surprising to many as there is certainly a bevy of P2P applications and protocols. Indeed, one of the wonders of software is that with sufficient effort it is possible to build a real-time media application on top of a best-effort packet-switched network... the collective ingenuity of the software industry is great at overcoming the limitations of the underlying system.

And yet, the fundamentally client-server nature of the modern internet cannot be fully overcome. P2P systems rely on finding some mechanism to create and maintain open connections between participants.

Early P2P systems such as BitTorrent (and most other file sharing systems) relied mostly on partial centralization and user effort. In the case of BitTorrent, for example, trackers are centralized services which maintain a database of available peers. BitTorrent should thus be viewed as a partially centralized system, or perhaps better as a distributed system with centralized metadata (this is an extremely common design in practice, and in fact the entire internet could be described this way if you felt like it). Further BitTorrent assumes that the inbound connection problem will be somehow solved by the user, e.g. by configuring appropriate port forwarding or using a local network that supports automated mechanisms such as UPnP.

Many P2P systems in use today have some sort of centralized directory or metadata service that is used for peer discovery, as well as configuration requirements for the user. But more recent advances in distributed methods have somewhat alleviated the needs for this type of centralization. Methods such as distributed hole punching (both parties initiating connections to each other at the same time to result in appropriate conntrack entries at the NAT devices) allow two arbitrary hosts to connect, but require that there be some type of existing communications channel to facilitate that connection.

Distributed hash tables such as the Kademlia DHT are a well understood method for a distributed system to share peer information, and indeed BitTorrent has had it bolted on as an enhancement while many newer P2P systems rely on a DHT as their primary mechanism of peer discovery. But for all of these there is a bootstrapping problem: once connected to other DHT peers you can use the DHT to obtain additional peers. But, this assumes that you are aware of at least a single DHT peer to begin with. How do we get to that point?

You could conceivably search the entire internet space, but given the size of the internet that's infeasible. The next approach you might reach for is some kind of broadcast or multicast, but for long-standing abuse, security, and scalability reasons broadcast and multicast cannot be routed across the internet. Anycast offers similar potential and is feasible on the internet, but it requires the cooperation of an AS owner which would be both centralized and require a larger up-front investment than most P2P projects are interested in.

Instead, real P2P systems address this issue by relying on a centralized service for initial peer discovery. There are various terms for this that are inconsistent between P2P protocols and include introducer, seed, bootstrap node, peer helper, etc. For consistency, I will use the term introducer, because I think it effectively describes the concept: an introducer is a centralized (or at least semi-centralized) service that introduces new peers to enough other peers that they can proceed from that point via distributed methods.

As a useful case study, let's examine Bitcoin... and we have come full circle back to the introduction, finally. Bitcoin prefers the term "seeding" to refer to this initial introduction process. Dating back to the initial Nakamoto Bitcoin codebase, the Bitcoin introduction mechanism was via IRC. New Bitcoin nodes connected to a hardcoded IRC server and joined a harcoded channel, where they announced their presence and listened for other announcements. Because IRC is a very simple and widely implemented message bus, this use of IRC had been very common. Ironically it was its popularity that lead to its downfall: IRC-based C2 was extremely common in early botnets, at such a level that it has become common for corporate networks to block or at least alert on IRC connections, as the majority of IRC connections out of many corporate networks are actually malware checking for instructions.

As a result, and for better scalability, the Bitcoin project changed the introduction mechanism to DNS, which is more common for modern P2P protocols. The Bitcoin Core codebase includes a hardcoded list of around a dozen DNS names. When a node starts up, it queries a few of these names and receives a list of A records that represent known-good, known-accessible Bitcoin nodes. The method by which these lists are curated is up to the operators of the DNS seeds, and it seems that some are automated while some are hand-curated. The details don't really matter that much, as long as it's a list that contains a few contactable peers so that peer discovery can continue from there using the actual Bitcoin protocol.

Other projects use similar methods. One of the more interesting and sophisticated distributed protocols right now is Hypercore, which is the basis of the Beaker Browser, in my opinion the most promising distributed web project around... at least in that it presents a vision of a distributed web that is so far not conjoined at the hip with Ethereum-driven hypercapitalism. Let's take a look at how Hypercore and its underlying P2P communications protocol Hyperswarm address the problem.

Well, it's basically the exact same way as Bitcoin with one less step of indirection. When new Hypercore nodes start up, they connect to bootstrap1.hyperdht.org through bootstrap3.hyperdht.org, each of which represents one well-established Hypercore node that can be used to get a toehold into the broader DHT.

This pattern is quite general. The majority of modern P2P systems bootstrap by using DNS to look up either a centrally maintained node, or a centrally maintained list of nodes. Depending on the project these introducer DNS entries may be fully centralized and run by the project, or may be "lightly decentralized" in that there is a list of several operated by independent people (as in the case of Bitcoin). While this is slightly less centralized it is only slightly so, and does not constitute any kind of real distributed system.

Part of the motivation for Bitcoin to have multiple independently operated DNS seeds is that they are somewhat integrity sensitive. Normally the Bitcoin network cannot enter a "split-brain" state (e.g. two independent and equally valid blockchains) because there are a large number of nodes which are strongly interconnected, preventing any substantial number of Bitcoin nodes being unaware of blocks that other nodes are aware of. In actuality Bitcoin enters a "split-brain" state on a regular basis (it's guaranteed to happen by the stochastic proof of work mechanism), but as long as nodes are aware of all "valid" blockchain heads they have an agreed upon convention to select a single head as valid. This method can sometimes take multiple rounds to converge, which is why Bitcoin transactions (and broadly speaking other blockchain entries) are not considered valid until multiple "confirmations"---this simply provides an artificial delay to minimize the probability of a transaction being taken as valid when the Bitcoin blockchain selection algorithm has not yet converged across the network.

But this is only true of nodes which are already participating. When a new Bitcoin node starts for the first time, it has no way to discover any other nodes besides the DNS seeds. In theory, if the DNS seeds were malicious, they could provide a list of nodes which were complicit in an attack by intentionally not forwarding any information about some blocks or advertisements of nodes which are aware of those blocks. In other words, in practice the cost of a sybil attack is actually reduced to the number of nodes directly advertised by the DNS seeds, but only for new users and only if the DNS seeds are complicit. In practice the former is a massive limitation. The Bitcoin project allows only trusted individuals, but also multiple such individuals, to operate DNS seeds in order to mitigate the latter. In practice the risk is quite low, mostly due to the limited impact of the attack rather than its difficulty level (very few people are confirming Bitcoin transactions using a node which was just recently started for the first time).

Multicast

One of the painful points here is that multicast and IGMP make this problem relatively easy on local networks, and indeed mDNS/Avahi/Bonjour/etc solve this problem on a daily basis, in a reasonably elegant and reliable way, to enable things like automatic discovery of printers. Unfortunately we cannot use these techniques across the internet because, among other reasons, IGMP does not manageably scale to internet levels.

P2P systems can use them across local networks, though, and there are P2P systems (and even non-P2P systems) which use multicast methods to opportunistically discover peers on the same local network. When this works, it can potentially eliminate the need for any centralized introducer. It's, well, not that likely to work... that would require at least one, preferably more than one, fully established peer on the same local network. Still, it's worth a shot, and Hypercore for example does implement opportunistic peer discovery via mDNS zeroconf.

Multicast presents an interesting possibility: much like the PGP key parties of yore, a P2P system can be bootstrapped without dependence on any central service if its users join the same local network at some point. For sufficiently widely-used P2P systems, going to a coffee shop with a stable, working node once in order to collect initial peer information will likely be sufficient to remain a member of the system into the future (as long as there are enough long-running peers with stable addresses that you can still find some and use them to discover new peers weeks and months into the future).

Of course by that point we could just as well say that an alternative method to bootstrapping is to call your friends on the phone and ask them for lists of good IP addresses. Still, I like my idea for its cypherpunk aesthetics, and when I inevitably leave my career to open a dive bar I'll be sure to integrate it.

Hope for the future

We have seen that all P2P systems that operate over the internet have, somewhere deep down inside, a little bit of centralized original sin. It's not a consequence of the architecture of the internet so much as it's a consequence of the fifty years of halting change that has brought the internet to its contemporary shape... changes that were focused around the client-server use cases that drive commercial computing for various reasons, and so had the network shaped in their image to the extent of exclusion of true P2P approaches.

Being who I am it is extremely tempting to blame the whole affair on capitalism, but of course that's not quite fair. There are other reasons as well, namely that the security, abuse, stability, scalability, etc. properties of truly distributed systems are universally more complex than centralized ones. Supporting fully distributed internet use cases is more difficult, so it's received lower priority. The plurality of relatively new P2P/distributed systems around today shows that there is some motivation to change this state of affairs, but that's mostly been by working around internet limitations rather than fixing them.

Fixing those limitations is difficult and expensive, and despite the number of P2P systems the paucity of people who actually use them in a P2P fashion would seem to suggest that the level of effort and cost is not justifiable to the internet industry. The story of P2P networking ends where so many stories about computing end: we got here mostly by accident, but it's hard to change now so we're going to stick with it.

I've got a list of good IP addresses for you though if you need it.

[1] WU's digital automatic routing network was developed in partnership with RCA and made significant use of microwave links as it expanded. Many have discussed the way that the US landscape is littered in AT&T microwave relay stations, fewer know that many of the '60s-'80s era microwave relays not built by AT&T actually belonged to Western Union for what we would now call data networking. The WU network was nowhere near as extensive as AT&T's but was particularly interesting due to the wide variety of use-cases it served, which ranged from competitive long distance phone circuits to a a very modern looking digital computer interconnect service.

[2] We should not get the impression that any of these problems are in any way specific to Windows. Many of the earliest computer worms targeted UNIX systems which were, at the time, more common. UNIX systems were in some ways more vulnerable due to their relatively larger inventory of network services available, basically all of which were designed with no thought towards security. Malware developers tended to follow the market.

>>> 2022-01-01 secret military telephone buttons

It's the first of the new year, which means we ought to do something momentous to mark the occasion, like a short piece about telephones. Why so much on telephones lately? I think I'm just a little burned out on software at the moment and I need a vacation before I'm excited to write about failed Microsoft ventures again, but the time will surely come. Actually I just thought of a good one I haven't mentioned before, so maybe that'll be next time.

Anyway, let's talk a little bit about phones, but not quite about long distance carriers this time. Something you may or may not have noticed about the carriers we've discussed, perhaps depending on how interesting you find data communications, is that we have covered only the physical layer. So far, there has been no consideration of how switches communicated in order to set up and tear down connections across multiple switches (i.e. long distance calls). Don't worry, we will definitely get to this topic eventually and there's plenty to be said about it. For the moment, though, I want to take a look at just one little corner of the topic, and that's multifrequency tone systems.

Most of us are at least peripherally familiar with the term "dual-tone multifrequency" or "DTMF." AT&T intended to promote Touch-Tone as the consumer friendly name for this technology, but for various reasons (mainly AT&T's trademark) most independent manufacturers and service providers have stuck to the term DTMF. DTMF is the most easily recognizable signaling method in the telephone system: it is used to communicate digital data over phone lines, but generally only for "meta" purposes such as connection setup (i.e. dialed digits). An interesting thing about DTMF that makes it rather recognizable is that it is in-band, meaning that the signals are sent over the same audio link as the phone call itself... and if your telephone does not mute during DTMF (some do but most do not), you can just hear those tones.

Or, really, I should say: if your phone just makes the beep boop noises for fun pretend purposes, like cellphones, which often emit DTMF tones during dialing even though the cellular network uses entirely on-hook dialing and DTMF is not actually used as part of call setup. But that's a topic for another day.

DTMF is not the first multi-frequency signaling scheme. It is directly based on an earlier system called, confusingly, multifrequency or MF. While DTMF and MF have very similar names, they are not compatible, and were designed for separate purposes.

MF signaling was designed for call setup between switches, mostly for long-distance calling. Whenever a call requires a tandem switch, so say you call another city, your telephone switch needs to connect you to a trunk on a tandem switch but also inform the tandem switch of where you intend to call. Historically this was achieved by operators just talking to each other over the trunk before connecting it to your local loop, but in the era of direct dialing an automated method was needed. Several different techniques were developed, but MF was the most common for long-distance calling in the early direct dial era.

An interesting thing about MF, though, is that it was put into place in a time period in which some cities had direct long distance dialing but others did not. As a result, someone might be talking to an operator in order to set up a call to a city with direct dial. This problem actually wasn't a new one, the very earliest direct dialing implementations routinely ran into this issue, and so it became common for operators switchboards to include a telephone dial mounted at each operator position. The telephone dial allowed the operator to dial for a customer, and was especially important when connecting someone into a direct dial service area.

MF took the same approach, and so one could say that there were two distinct modes for MF: in machine-to-machine operation, a telephone switch automatically sent a series of MF tones after opening a trunk, mainly to forward the dialed number to the next switch in the series. At the same time, many operators had MF keypads at their positions that allowed them to "dial" to a remote switch by hand. The circuitry that implemented these keypads turned more or less directly into the DTMF keypads we see on phones today.

Like DTMF, MF worked by sending a pair of two frequencies [1]. The frequencies were selected from the pool of 700, 900, 1100, 1300, 1500, and 1700Hz. That's six frequencies, and it is required that two frequencies always be used, so the number of possible symbol is 6c2 or 15. Of course we have the ten digits, 0-9, but what about the other five? The additional five possibilities were used for control symbols. For reasons that are obscure to me, the names selected for the control symbols were Key Pulse or KP and Start or ST. Confusingly, KP and ST each had multiple versions and were labeled differently by different equipment. The closest thing to a universal rule would be to say that MF could express the symbols 0-9, KP1-KP2, and ST1-ST3.

Part of the reason that the labeling of the symbols was inconsistent is that their usage was somewhat inconsistent from switch to switch. Generally speaking, an operator would connect to a trunk and then press KP1, the number to be called, and then ST1. KP1 indicated to the far-side switch that it should set up for an incoming connection (e.g. by assigning a sending unit or other actions depending on the type of switch), while ST1 indicated that dialing was complete. Most of the time telephone switches used other means (digit-matching based on "dial plans") to determine when dialing was complete, but since tandem switches handled international calls MF was designed to gracefully handle arbitrary length phone numbers (due to both variance between countries and the bizarre choice of some countries to use variable-length phone numbers).

The additional KP and ST symbols had different applications but were most often used to send "additional information" to the far side switch, in which case the use of one of the control symbols differentiated the extra digits (e.g. an account number) from the phone number.

MF keypads were conventionally three columns, two columns of digits (vertically arranged) and one of control symbols on the right.

This is a good time to interject a quick note: the history of MF signaling turns out to be surprisingly obscure. I had been generally aware of it for years, I'm not sure why, but when I went to read the details I was surprised by... how few details there are. Different sources online conflict about basic facts (for example, Wikipedia lists 6 frequencies which is consistent with the keypad I have seen and the set of symbols, but a 1960 BSTJ overview article says there were only five...). So far as I can tell, MF was never formally described in BSTJ or any other technical journal, and I can't find any BSPs describing the components. I suspect that MF was an unusually loose standard for the telephone system, and that the MF implementation on different switches sometimes varied significantly. This is not entirely surprising since the use of MF spanned from manual exchanges to modern digital exchanges (it is said to still be in use in some areas today, although I am not aware of any examples), covering around 80 years of telephone history.

I didn't really intend to go into so much detail on MF here, but it's useful to understand my main topic: DTMF. MF signaling went into use by the late 1940s (date unclear for the reasons I just discussed), and by 1960 was considered a main contender for AT&T's goal of introducing digital signaling not just between switches but also from the subscriber to the switch [2]. A few years later, AT&T introduced Touch-Tone or DTMF dialing. Unsurprisingly, DTMF is really just MF with some problems solved.

MF posed a few challenges for use with subscriber equipment. The biggest was the simple placement of the frequencies. The consistent 200 Hz separation meant that certain tones were subject to harmonics and other intermodulation products from other tones, requiring high signal quality for reliable decoding. That wasn't much of a problem on toll circuits which were already maintained to a high standard, but local loops were routinely expected to work despite very poor quality, and there was a huge variety of different equipment in use on local loops, some of which was very old and easily picked up spurious noise.

Worse, the MF frequencies were placed in a range that was fairly prominent in human speech. This resulted in a high risk that a person talking would be recognized by an MF decoder as a symbol, which could create all kind of headache. This wasn't really a problem for MF because MF keypads were designed to disconnect the subscriber when digits were pressed. DTMF, though, was intended to be simpler to implement and convenient to use while in a call, which made it challenging to figure out how to disconnect or "mute" both parties during DTMF signaling.

To address these issues, a whole new frequency plan was devised for DTMF. The numbers and combinations all seem a bit odd, but were chosen to avoid any kind of potential intermodulation artifacts that would be within the sensitivity range of the decoder. DTMF consisted of eight frequencies, which were organized differently, into a four by four grid. A grid layout was used, in which there is one set of "low" frequencies and one set of "high" frequencies and "low" was never mixed with "low" and vice versa, because it allowed much tighter planning of the harmonics that would result from mixing the frequencies.

So, we can describe DTMF this way: there are four rows and four columns. The four rows are assigned 697, 770, 852, and 941 Hz, while the four columns are 1209, 1336, 1477, and 1633 Hz. Each digit consists of one row frequency and one column frequency, and they're laid out the same way as the keypad.

Wait a minute... four rows, four columns?

DTMF obviously needed to include the digits 0-9. Some effort was put into selecting the other available symbols, and for various reasons * and # were chosen as complements to the digits (likely owing to their common use in typewritten accounting and other business documents at the time). That makes up 12 symbols, the first three columns. The fourth column, intended mostly for machine-to-machine applications [3], was labeled A, B, C, and D.

Ever since, DTMF has featured the mysterious symbols A-D, and they have seen virtually no use. It is fairly clear to me that they were only included originally because DTMF was based directly on MF and so tried to preserve the larger set of control symbols and in general a similar symbol count. The engineers likely envisioned DTMF taking over as a direct replacement for MF signaling in switch-to-switch signaling, which did happen occasionally but was not widespread as newer signaling methods were starting to dominate by the time DTMF was the norm. Instead, they're essentially vestigial.

One group of people which would be generally aware of the existence of A-D are amateur radio operators, as the DTMF encoders in radios almost always provide a full 4x4 keypad and it is somewhat common for A-D to be used for controlling telephone patches---once the telephone patch is connected, 0-9, *, and # will be relayed directly to the phone network, A-D provide an opportunity for four symbols that are reserved for the patch itself to respond to.

Another group of people to which this would be familiar is those in the military from roughly the '70s to the '90s, during the period of widespread use of AUTOVON. While AUTOVON was mostly the same as the normal telephone network but reserved for military use, it introduced one major feature that the public telephone system lacked: a precedence, or priority system.

Normally dialed AUTOVON calls were placed at "routine" priority, but "priority," "immediate," "flash," and "flash override" were successively higher precedence levels reserved for successively more important levels of military command and control. While it is not exactly true, it is almost true, and certainly very fun to say, that AUTOVON telephones feature a button that only the President of the United States is allowed to press. The Flash Override or FO button was mostly reserved for use by the national command authority in order to invoke a nuclear attack, and as you would imagine would result in AUTOVON switches abruptly terminating any other call as necessary to make trunks available.

AUTOVON needed some way for telephones to indicate to the switch what the priority of the call was, and so it was obvious to relabel the A, B, C, and D DTMF buttons as FO, F, I, and P respectively. AUTOVON phones thus feature a full 4x4 keypad, with the rightmost column typically in red and used to prefix dialed calls with a precedence level. Every once in a while I have thought about buying one of these phones to use with my home PABX but they tend to be remarkably expensive... I think maybe restorers of military equipment are holding up prices.

And that's what I wanted to tell you: the military has four extra telephone buttons that they don't tell us about. Kinda makes you take the X-files a little more seriously, huh?

In all seriousness, though, they both do and don't today. Newer military telephone systems such as DSN and the various secure VoIP systems usually preserve a precedence feature but offer it using less interesting methods. Sometimes it's by prefixing dialing with a numeric code, sometimes via feature line keys, but not by secret DTMF symbols.

[1] This was technically referred to as a "spurt" of MF, a term which I am refusing to use because of my delicate sensibilities.

[2] One could argue that pulse dialing was "digital," but because it relied on the telephone interrupting the loop current it was not really "in-band" in the modern sense and so could not readily be relayed across trunks. Much of the desire for a digital subscriber signaling system was for automated phone systems, which could never receive pulses since they were "confined" to the local loop. Nonetheless DTMF was also useful for the telephone system itself and enabled much more flexible network architectures, especially related to remote switches and line concentrators, since DTMF dialing could be decoded by equipment "upstream" from wherever the phone line terminated without any extra signaling equipment needed to "forward" the pulses.

[3] This might be a little odd from the modern perspective but by the '60s machine-to-machine telephony using very simple encodings was becoming very popular... at least in the eyes of the telephone company, although not always the public. AT&T was very supportive of the concept of telephones which read punched cards and emitted the card contents as DTMF. In practice this ended up being mostly used as a whimsical speed-dial, but it was widely advertised for uses like semi-automated delivery of mail orders (keypunch them in the field, say going door to door, and then call an electromechanical order taking system and feed them all through your phone) and did see those types of use for some time.