>>> 2021-12-13 coaxial goes to war (PDF)

Last time we perambulated on telephones, we discussed open-wire long-distance telephone carriers and touched on carriers intended for cables. Recall that, in the telephone industry, cable refers to an assembly of multiple copper twisted pairs (often hundreds) in a single jacket. There is a surprising amount of complexity to the detailed design of cables, but that's the general idea. Cables were not especially popular for long-distance service because the close proximity of the pairs always lead to crosstalk problems. Open-wire was much better in that regard but it was costly to install and maintain, and the large physical size of open-wire arrays limited the channel capacity of long-distance leads.

The system of carriers, multiplexing multiple lines onto a single set of wires, allowed for a significant improvement in the capacity of both cables and open-wire. However, even the highest quality open-wire circuits could offer only a very limited bandwidth for multiplexing. In practice, anything near 100kHz became hopelessly noisy as the balanced transmission system used on these cables was simply ineffective at high frequencies. Because phone conversations required around 15kHz of bandwidth (assuming no companding, which was not yet done at the time) this imposed a big limit... which helps to explain why open-wire carriers basically topped out at 17 total channels.

Fortunately, in the early 1930s AT&T engineers [1] began to experiment with a then obscure type of cable assembly called a coaxial line (I will stick to this terminology to avoid confusion with the more specific industry meaning of "cable"). Coaxial lines were first proposed in the 19th century and are in widespread use today for all manner of RF applications, although people tend to associate them most with cable television. At the time, though, there were no commercial applications for long, high-frequency coaxial lines, and so AT&T's efforts covered considerable new ground.

The basic concept of a coaxial line is this: a center conductor is surrounded first by a dielectric (typically air in earlier lines with separation achieved by means of wrapping a non-conductive fiber cord around the center conductor) and then by a cylindrical metal outer conductor. Unlike open-wire lines, the outside conductor is connected to ground. This system has two particularly useful properties: first, because the signal is carried through the internal space between the conductors which is effectively a capacitor, the system acts much like the old loaded telephone lines (but more effective) and can carry very high frequencies. Second, the skin effect causes the outer conductor to serve as a very effective shield: undesired RF energy follows the outside of the outer conductor to ground, and is thus kept well isolated from the RF energy following the inside of the outer conductor. Coaxial lines can support a high bandwidth with low noise, and for this reason they are still the norm today for most RF applications.

The high-bandwidth property of coaxial lines has an interesting implication that the 1934 BSTJ article introducing the concept must lay out explicitly, since the technology was not yet familiar. Because a coaxial line can carry a wide range of frequencies simultaneously, it can be used by a radio system much like the air. We have previously discussed "wired radio," but coaxial lines provide a much more literal "wired radio." Modern CATV systems, for example, are basically an entire broadcast RF spectrum, up to about 1GHz, but contained inside of a coaxial line instead of traveling through free air. The implication is that coaxial lines can carry a lot of payload, using conventional radio encoding methods, but are isolated such that two adjacent lines can use the same frequencies for different purposes with no (or in practice minimal) interference with each other.

We can presumably imagine how telephone over coaxial line works: much like the open-wire carriers, telephone calls are modulated to higher frequencies, and thus many telephone calls at different frequencies can be multiplexed onto one line. The principle is simple, although in the 1930s the electronics necessary to perform the modulation, amplification, and demodulation were rather complex.

Adding further complexity, the early coaxial lines which could be manufactured at the time had rather high attenuation compared to modern cables, requiring frequent amplification of the signal (you will be surprised, when we get to it, by just how frequent). Further, the RF properties of the cables (in terms of frequency response and attenuation) turned out to be significantly related to the temperature of the cable, likely mostly because of expansion and contraction of the outer conductor which was physically relatively large (1/2" common in early telephony experiments) and secured loosely compared to modern designs.

Another important trend occurring at the same time was the creation of national television networks. Radio networks already often used leased telephone lines to distribute their audio programming to various member stations, and this had by the 1930s already become a profitable service line for AT&T. Television networks were now looking to do the same, but the far higher bandwidth required for a television signal posed a challenge to AT&T which had few options technically capable of carrying them. This was a huge incentive to develop significantly higher bandwidth carriers.

AT&T first created a 2600' test cable at a facility in Phoenixville, Pennsylvania. Tests conducted on this length of copper pipe in 1929 validated the concept and lead to the 1930s project to fully realize the new carrier scheme. In 1936, AT&T committed, building a real coaxial long-distance lead between New York and Philadelphia that supported 240 voice channels or a single television channel. The design bandwidth of this line was what we now call 1MHz, but in AT&T documents it is widely referred to as the "million-cycle system" or, less picturesque, the 1,000 kc system. Because of the rather high attenuation in the line, repeaters were required every 10.5 miles, and the design of suitably wide-band repeaters was one of the greater challenges in development of this experimental toll lead.

Powering these repeaters proved a challenge. Previous carrier systems had usually had a local utility install three-phase power to each repeater station; it was undesirable to run power along with the signal wires because AC hum in telephone calls had been an ongoing frustration with telephone lines run along with power. With repeaters as frequently as 10 miles, though, the cost of adding so many new power lines would have been excessive. Instead, the decision was made to bundle the coaxial signal cable along with wires used for high-voltage DC. "Primary power supply stations," later called "main stations," had grid connections (and typically backup batteries and generators) along with rectification equipment to inject HVDC onto the cable. Repeaters between main stations ran off of this DC power. Much the same technique is used today for transoceanic cables.

Following experiments performed on this early coaxial route, the frequency division carrier on coaxial cable was productionized as the L-carrier, or specifically L1. The first proper L1 route was installed from Stevens Point, Wisconsin to Minneapolis in 1941. L1 combined voice channels into groups of 12, called banks. Five banks were then modulated to different carriers to form a group. Finally, eight groups were modulated onto carriers to form a "supergroup" of 480 channels, which was transmitted on the cable [2]. The end result spanned 68 kHz to 2044 kHz on the line, and some additional carriers at higher frequencies were used as "pilots" for monitoring cable attenuation to adjust amplifiers.

As L1 equipment and installation methods improved, additional supergroups were added to reach 600 total channels, and it became the norm to combine multiple coaxial lines into a single assembly (along with power wires and often several basic twisted pairs, which served as order wires). Late L1 installations used four coaxial lines, for a total of 2400 channels.

AT&T briefly experimented with an L2 carrier, a variation that was intended to be simpler and lower cost and thus suitable for shorter toll leads (e.g. within metro areas). The effort quickly proved to be uncompetitive with conventional cables and was canceled, which is simply to explain why most accountings of L-carrier history totally skip L2.

In 1952, a major advancement to the technology came in the form of the L-3 carrier, initially installed between New York and Philadelphia for testing. L-3 carried three "mastergroups" spanning approximately 200kHz to 8.3 MHz. Each mastergroup contained two submastergroups, which each contained six supergroups, which matched L1 in containing eight groups of five banks of twelve channels. This all combined onto 8 coaxial lines in a typical assembly yielded a total of 14,880 voice channels per route, although both more and fewer coaxial lines were used for some routes. As an additional feature, L-3 could optionally replace two mastergroups with a TV channel, allowing one TV channel and 600 channels on a cable.

One of the larger improvements to L-3, besides its increased capacity, was a significant expansion of the supporting infrastructure considered part of the carrier installation. This included repeaters at 4 mile intervals, some of which were fitted with additional signal conditioning equipment (namely for equalization of the balanced pairs). Main stations were required to inject power at roughly 100-mile intervals.

For an additional quality improvement, L-3 used a technique called "frogging" in which the supergroups were periodically "rearranged" or swapped between frequency slots. This prevented excessive accumulation of intermodulation products in any one supergroup frequency range, and was done at some main stations, typically about every 800 miles.

A more interesting feature, though, was L-3's substantial automatic protection capabilities. Equipment at each main station (where power was injected) monitored a set of pilot signals on each coaxial line, and each route was provided with a spare coaxial line. A failure of any coaxial line triggered an automatic switching of its modulation and demodulation equipment onto the spare pair, restoring service in about 15ms. L-3 also contained several other automatic recovery features, and an extensive alarm system that detected and reported faults in the cable or at repeaters.

Here, at L-3, we will spend more time discussing what is perhaps the most interesting part of the L-carrier story: the physical infrastructure. L-3 was the state of the art in long-distance toll leads during an especially portentous period in US telecom history: the 1960s or, more to the point, the apex of the cold war.

Beginning in 1963, the military began construction of the Automatic Voice Network, or AUTOVON. AUTOVON, sometimes called "the DoD's phone company," was a switched telephone network very much like the regular civilian one but installed and operated for the military. But, of course, the military did not really build telephone infrastructure... they contracted it to AT&T. So, in practice, AUTOVON was a "shadow" or "mirror" telephone system that was substantially similar to the PSTN, and operated by AT&T on largely the same equipment, but only terminated at various military and government installations.

A specific goal for AUTOVON was to serve as a hardened, redundant system that would survive an initial nuclear attack to enable coordination of a reprisal. In other words, AUTOVON was a critical survivable command and control system. To that end, it needed a survivable long-distance carrier, and L-carrier was the state of the art.

Previous L-carrier routes, including the initial L-3 routes, had enclosed their repeaters and power feeds in existing telephone exchange buildings and brick huts alongside highways. For AUTOVON, though, AT&T refined the design to L-3I, or L-3 Improved. The "Improved," in this case, was a euphemism for nuclear hardening, and L-3I routes consisted entirely of infrastructure designed to survive nuclear war (within certain parameters, such as a maximum 2 PSI blast overpressure for many facilities). Built in the context of the cold war, nearly all subsequent L-carrier installations were nuclear hardened.

The first major L-3I project was the creation of a hardened transcontinental route for AUTOVON. Like the 1915 open-wire route before it, the first L-3I connected the east coast to the west coast via New York and Mojave---or rather, connected the military installations of the east coast to the military installations of the west coast.

L-3I routes consisted of repeater stations every 4 miles, which consisted of a buried concrete vault containing the repeater electronics and a sheet-metal hut on the surface, directly over the vault manhole, containing test equipment and accessories to aid maintenance technicians. Because repeaters were powered by the line itself, no utility power was required, although many repeater huts had it added at a later time to allow use of the lights and ventilation blower without the need to run a generator.

Every 100 miles, a main station injected power onto the cable and contained automatic protection equipment. Some main stations contained additional equipment, up to a 4ESS tandem switch. Main stations also served as interconnect points, and often had microwave antennas, cables, and sometimes "stub" coaxial routes to connect the L-3I to nearby military and civilian telephone exchanges (L-3I routes installed for AUTOVON were also used for civilian traffic on an as-available basis). A few particularly large main stations had even more equipment, as they were capable of serving as emergency network control facilities for the AUTOVON system.

A typical main station consisted of a five to twenty thousand square foot structure buried underground, with all sensitive equipment mounted on springs to provide protection from shocks transmitted through the ground. Vent shafts from the underground facility terminated at ground-level vents with blast deflectors. A gamma radiation detector on the surface (and, in later installations, a "bhangmeter" type optical detector) triggered automatic closure of blast valves on all vents when a nearby nuclear detonation was detected. Several diesel generators, either piston or turbine depending on the facility, were backed by buried diesel tanks to provide a two-week isolated runtime. Water wells (with head pit underground), water tanks, and supplies of sealed food supported the station's staff for the same two week duration. This was critical, as main stations required a 24/7 staff for monitoring and maintenance of the equipment.

At those facilities with interconnections to microwave routes, even the microwave antennas were often a variant of the KS-15676 modified for hardening against blast overpressures by the addition of metal armor. L-3I main stations, being hardened connection points to AUTOVON with a maintenance staff on-hand, were often used as ground stations for the ECHO FOX and NORTH STAR contingency communications networks that supported Air Force One and E-4B and E-6 "Doomsday planes."

This first transcontinental L-3I ran through central New Mexico and had a main station at Socorro, where I used to live. In fact, the Socorro main station [3] housed a 4ESS tandem switch, a master synchronization time source for the later digital upgrade of the system, and served as the contingency network control center for the western half of AUTOVON, making it one of the larger main stations. You would have no idea from the surface, as the surface structures are limited to a short microwave tower (for interconnection to the Rio Grande corridor microwave route) [4], a small surface entry building, and a garage for vehicle storage. The only indication of the facility's nature as cold war nuclear C2 infrastructure are the signs on the perimeter fence which bear a distinctive warning about the criminality of tampering with communications infrastructure used for military purposes. And the gamma radiation detector, if you know what they look like.

As an aside, a poorly maintained page that includes photos of some of these locations can be found on my personal website: https://jbcrawford.us/history/bellsystem/socorro

Hopefully you can see why I have always found this fascinating. Rumors about secret underground military facilities abound, and yet few really exist... but somewhat secret underground telephone facilities are actually remarkably common, as not only L-3I but following L-4 and L-5 main stations were all hardened, buried facilities that were, at least initially, discreet. The fact that such an important part of the network infrastructure was located in such a rural area might be a surprise to you, for example (at least if you are familiar with New Mexico geography), but this was an explicit goal: L-3I main stations were required to be located at least 20 miles from any major population centers, since they were designed based on a nuclear detonation at 5 miles distance. So not only are these sorts of underground facilities found throughout the nation, they're almost always found in odd places... off the side of rural US highways between modest towns.

Given the lengths I have already reached, I will spend less time on L-4 and L-5. This isn't much of a loss to you, because L-4 and L-5 were mostly straightforward incremental improvements on L-3I. L-4 reached up to 72,000 channels per route while L-5E ("L5 Enhanced," which if you read the relevant BSTJ articles appears to be merely the original L5 scheme with a limitation in the multiplexing equipment resolved) reached up to 108,000 channels, using 66 MHz of bandwidth on each coaxial line.

Somewhat counter-intuitively, AT&T achieved these increases in capacity at the cost of increased attenuation, so the repeater frequency actually increased as the L-carrier technology evolved. L-4 required a repeater every 2 miles and a power feed every 150, while L-5 required a repeater every 1 mile and a power feed every 75. Some L-3I routes, such as the segment between Socorro and Mojave, were upgraded to L-4, resulting in an L-4 repeater added between each pair of original L-3I repeaters. Most L-4 routes were upgraded to L-5, resulting in "alternating" main stations as smaller L-5 power-feed-only main stations were added between the older L-4 main stations with more extensive protection equipment.

Both L-4 and L-5 made use of entirely underground repeaters (e.g. no hut at the surface), although L-4 repeaters sometimes had huts above them... usually at mid-span equalizing repeaters (every 50 miles) and occasionally randomly at others. The L-4 huts are said to have been almost entirely empty, serving only to give technicians a workspace out of the wind and rain.

These L-carrier systems were entirely analogue as you have likely gathered, and started out as tube equipment that transitioned to transistorized in the L-3I era. But the analogue limitation was not undefeatable, and Philips designed a set of systems for digital data on L-carrier referred to as P-140 (140 Mbps on L-4) and P-400 (400 Mbps on L-5). Most L-carrier routes still in service in the '80s were upgraded to digital.

What came of all of this infrastructure? Not long after the development of L-5E in 1975, fiber optics began to reach maturity. By the 1980s fiber optics had become the obvious future direction in long-distance telecoms, and L-carrier began to look obsolete. L-carrier routes generally went out of service in the '90s, although some survived into the '00s. Many, but not all, L-carrier rights of way were reused to trench fiber optic cable, and some L-carrier repeater vaults were reused as land for fiber add/drop and regeneration huts (typically every 24 miles on telco fiber lines).

More interestingly, what about the main stations? Large underground facilities have long proven difficult to repurpose. At least twice a month someone declares that it would be a good idea to purchase an underground bunker and operate it as a "high security data center," and sometimes they even follow through on it, despite the fact that these ventures have essentially never been successful (and the exceptions seem to be the type that prove the rule, since they are barely surviving and/or criminal enterprises). The nation is studded with AT&T main stations and Atlas and Titan missile silos that suffer from extremely haphazard remodeling started, but not finished, by a "data center" operator before going bankrupt. There are two examples just in the sparsely populated state of New Mexico (both surrounding Roswell in the former Walker AFB missile silo complex).

In practice, the cost of restoring and upgrading a Cold War underground facility for modern use usually exceeds the cost of building a new underground facility. The rub of it is that no one actually wants to put their equipment in an underground data center anyway. These Cold War facilities cannot practically be upgraded to modern standards for redundant HVAC, power, and connectivity, and are never operated by ventures with enough money to hire security, add vehicle protections, and obtain certifications. Ironically, they are less secure and reliable than your normal above-ground type. Most of them are highly prone to flooding [5].

Many main stations, L-4 and L-5 in particular, have been sold into private ownership. Some owners have tried to make use of the underground facility, but most have abandoned it and only use the surface land (for example because it is adjacent to their farm). A few are being restored but these restoration efforts quickly become very expensive and usually fail due to lack of funds, meaning these often come up on the market with odd quirks like new kitchen appliances but a foot of water on the lower level.

On the other hand, because L-carrier main stations sat on high-capacity long-distance lines and had a staff and space for equipment, they naturally became junction points for other types of long-distance technology. Many L-carrier main stations are still in use today as switching centers for fiber routes, but in most cases the underground has been placed in mothballs and new surface buildings contain the actual equipment (the cost of modernizing the electrical infrastructure and adding new cable penetrations to the underground areas is very high). Mojave is a major example, as the old Mojave L-3I main station remains one of Southern California's key long-distance telephone switching centers.

Still others exist somewhere in-between. I have heard from a former employee that Socorro, for example, is no longer in use for any long-distance application and is largely empty. But CenturyLink, the present owner, still performs basic caretaking on the structure at least in part because they know that details of the lease agreement (most western L-carrier facilities are on land leased from the BLM or Forest Service) will require them to perform site remediation that is expected to be very costly. As happens all too often with old infrastructure, it's cheaper to keep the lights on and little else than to actually close the facility.

I am not aware of any former main stations that are not fairly well secured. Repeaters are a different story. L-4 and L-5 seldom lead to interesting repeater sites since they were underground vaults that were often filled and in any case are very dangerous to enter (hydrogen sulfide and etc). L-3I, on the other hand... nearly all visible signs of the L-3I transcontinental in California were removed due to the environmental sensitivity of the Mojave Desert region. Many other L-3I routes, though, in their more rural sections, feature completely abandoned repeater huts with the doors left to flap in the wind.

Even in the absence of visible structures, L-carrier has a very visible legacy. In the desert southwest, where the land is slow to heal, the routes are clearly visible in aerial images to this day. A 100' wide swath cleared of brush with a furrow in the center is perhaps more likely to be a petroleum pipeline, but some of them are old L-carrier routes. You can identify AT&T's routes in some cases by their remarkable dedication to working in completely straight lines, more so than the petroleum pipelines... perhaps an accommodation to the limited surveying methods of the 1960s that created a strong desire for the routes to be easy to pace out on the ground.

Since the genesis of the L-carrier system AT&T has maintained a practice of marking underground lines to discourage backhoe interference. There are many types of these markers, but a connoisseur learns certain tricks. Much like the eccentric wording "Buried Light Guide" indicates one of the earliest fiber optic routes, signs reading "Buried Transcontinental Telephone Line" usually indicate L-carrier. Moreover, L-carrier routes all the way to L-5 used AT&T's older style of ROW markers. These were round wooden posts, about 4" in diameter and 4-8' tall, with one to three metal bands painted orange wrapped around the top. Lower on the post, a rectangular sign gave a "Buried telephone line, call before digging" warning and a small metal plate affixed to the post near the sign gave a surveyor's description of the route each way from the post (in terms of headings and distances). Maintenance crews would locate the trench if needed by sighting off of two posts to find the vector between them, so they are usually tall enough and close enough together that you can see more than one at a time.

It's hard to find or put together maps of the entire system, as routes came and went over time and AT&T often held maps close to their chest. Some are available though, and a 1972 map [6] depicts L-3I and major microwave routes. L-4 and L-5 routes were more common but fewer maps depict them; on all maps of long-distance routes in the 1980s time period there is a high chance that any given non-microwave route is L-4 or L-5.

At the peak of the L-carrier network, there were over 100 hardened underground main stations, 1000 underground repeater vaults, and at least 5,000 miles of right of way. For a period of around two decades, L-carrier was the dominant far long-distance technology in the United States, and the whole thing was designed for war.

There is so much more to say about both L-carrier and AT&T's role in Cold War defense planning, and I have already said a lot here. My next post will probably be on a different topic, but I will return to long-distance systems in order to discuss microwave. Microwave relay systems were extensively built and covered many more route-miles than L-carrier. The lower cost of installation made microwave better for lower-capacity routes, and also spurred competitive long distance carriers like MCI and Verizon to use almost entirely microwave. Later, we will get to fiber, although I have previously written about SONET.

I will also return to AT&T and nuclear war, one of my greatest interests. The practicalities of the Cold War---that it consisted primarily of an enormous planning exercise in preparation for nuclear attack---meant that AT&T functioned effectively as another branch of the military. Nearly every nuclear scenario involved AT&T's infrastructure, and AT&T and its subsidiaries, partners, and successors were knee deep in secret planning for the end of the world. They still are today.

P.S.: I have a fairly extensive collection of information on L-carrier routes, and particularly those built for AUTOVON. There is a surprisingly large community of people interested in this topic, which means that many resources are available. Nonetheless it has always been my intent to put together one of the most comprehensive sources of information on the topic. For various reasons I had put this project down for years, but I am picking it back up now and hope to produce something more in the format of a book over the next year. I will perhaps share updates on this from time to time.

[1] You have hopefully, by now, realized that I am using "AT&T" to refer to the entirety of the Bell System apparatus, which at various time periods consisted of different corporate entities with varying relationships. Much of the work I attribute to AT&T was actually performed by AT&T Long Lines, Bell Laboratories, and the Western Electric Company, but some of it was also performed by various Bell Operating Companies (BOCs, although they became the somewhat more specifically defined RBOCs post-breakup). All of these entities have been through multiple rounds of restructuring and, often, M&A and divestitures, with the result that you sort of need to settle on using one name for all of them to avoid spending a lot of time explaining the relationships. The same organizations usually exist today in the forms of AT&T, Alcatel-Lucent, Nokia, Avaya, CenturyLink, etc., but often not recognizably.

[2] This hierarchy of multiple levels of multiplexing was used both to make the multiplexing electronics more practical and to allow L-carrier's 12-channel banks to "appear" the same as the J and K-carrier banks. The concept had a lot of staying power, and virtually all later telephone-industry multiplexing scheme used similar hierarchies, e.g. DS0, DS1, etc.

[3] If you are following along at home it is technically the Luis Lopez or "Socorro #2" main station, located just south of Socorro, as the Socorro name was already used within AT&T Long Lines for an en-route microwave relay located somewhat north of Socorro.

[4] If you're one of the few who has seen my few YouTube videos, you might find it interesting that documentation refers to a direct microwave connection between the Socorro #2 main station and the Manzano Base nuclear weapons repository (now disused and part of Kirtland AFB). It's unclear if this was a dedicated system or merely reserved capacity on the Rio Grande route, although the latter seems more likely since multiple relays would be required and there's no evidence of any.

[5] Like most underground facilities, these main stations are often below the water table and have always required sump pumps for water extraction. As they get older the rate of water ingress tends to increase, and so if the pumps are out of operation for any period of time they can quickly turn into very-in-ground swimming pools.

[6] https://jbcrawford.us/_media/history/bellsystem/1972_l-carrier_map.jpg