>>> 2023-05-07 electrical characteristics of telephone lines

Let's take a break from our boring topic of regional history to focus instead on an even more boring topic: implementation details of telephone lines.

The conventional "copper pair" analog telephone line is fading away. The FCC has begun to authorize abandonment of copper outside plant in major markets, and telcos are applying to perform such abandonment in more and more areas. The replacement is IP, part of the overall trend of "over the top" delivery, meaning that all communications utilities can be delivered by the use of IP as a common denominator.

This type of service is usually branded as "digital voice." Historically this seems to have come about to evade VoIP's bad reputation; in the early days of Vonage and the charmingly sketchy, back-of-magazine-ad Magic Jack, VoIP products often delivered subpar service. Today, I think "digital voice" has mostly just become part of price differentiation for carrier-offered VoIP, since independent VoIP services tend to cost considerably less. Still, there is some logic to differentiating digital voice and VoIP: because digital voice service is offered by the operator of the underlying IP network, it benefits from QoS measures that general internet traffic doesn't. On consumer internet connections, especially slower ones, digital voice is still likely to be more reliable than VoIP due to QoS policy.

Ultimately the move to digital voice is probably a good thing, as the abandonment of copper plant will kill off DSL in urban markets and make way for faster offerings---from telcos, usually PON. I'll take the opportunity to eulogize the conventional copper pair, though, by going in to a bit of detail about how it actually worked.

To start, a large disclaimer: the details of the telephone network have varied over time as technology and the industry evolved. Details often varied from manufacturer to manufacturer, and because Western Electric had a practical monopoly on the manufacturing of telephone instruments for many decades, it's pretty much the case that the "standards" for telephone lines in the US were "whatever Western Electric did," which varied over time. There were some independent organizations that promulgated telephone standards (such as the railroads which had their own extensive telephone plants), but they were almost always completely deferential to the Bell System. Independent telephone companies initially had to use different conventions than Bell because much of the Bell telephone system was under patent; after the expiration of these patents they mostly shifted to doing whatever Western Electric did to benefit from the ready availability of compatible equipment.

After divestiture, Western Electric's de facto standards-making power was vested to Bellcore, later Telcordia, today iconectiv, which after the end of AT&T monopoly was owned by defense contractor SAIC and is owned today by AT&T's erstwhile competitor Ericsson. iconectiv continues to promulgate some of the standards used in the telephone industry today, through contracts awarded by the FCC.

This is all to explain that the telephone system is actually surprisingly poorly standardized in the United States. You might expect some hefty ISO specification for analog telephone lines, but there isn't really one outside of equipment specifications published by manufacturers. Many international markets have much more detailed engineering specifications from independent bodies, but they're usually based directly on Western Electric's practices. To make things more confusing, it's not unusual for international telephone standards to either be based on older US practices that are now rare in the US, or to have standardized on "in practice" properties of the US system instead of nominal values, or to have mixed conventions from Western Electric with conventions from European telephone manufacturers like Ericsson. All of these standards end up being mostly the same, but with a dizzying number of slight differences.

Today, the FCC imposes requirements on telephone lines as part of its regulatory oversight of telcos. The FCC's requirements are basically to "keep doing whatever Western Electric did," and are often surprisingly loose. Phones are really very robust, and the basic design of the system is over 100 years old. Local loops are routinely in poor condition which throws things out of spec anyway, and then subscribers use all kinds of weird phones that are not always that well designed (the history of regulation of telephone instruments could fill its own post). It's actually fairly intentional that the electrical specifications in the system are all soft targets.

For the purpose of this article I am mostly going to describe the state of a fairly modern local loop, such as one connected to a 5ESS or DMS-100 digital switch. But I will definitely not describe either of these switches totally accurately. I'm trying to give the right general idea without getting too bogged down in the details like the last few paragraphs. I'll take the topic of electrical specifications (potential and current on telephone lines) as a chance to give some examples of the variation you see in practice.

First, let's talk about the very general architecture of an analog local loop.

Somewhere in town there is a telephone exchange, and somewhere in your house there is a telephone. The local loop is fundamentally two long copper wires that go directly from your phone to the exchange. This distance varies greatly. It's advantageous to keep it under a few miles (mostly for DSL), but in rural areas especially it can be far longer.

There's a little more detail to what goes on at the two ends of the line. In your house, you have one or more telephones that you use to make and receive calls. In the parlance of the industry, these are often referred to as "instruments" or "subscriber terminals" depending on the era and organization. Historically, instruments were considered part of the telephone system proper and were property of your telco. Today, you are allowed to purchase and use your own telephones. This comes with some downsides. Along with the phone being your property (and thus your problem), the telephone wiring inside of your home is your property (/problem).

The telephone wiring in your house runs from jack to jack. In the United States, all of the telephone jacks in a home are connected in parallel. This is one of the differences you will find if you look in other countries: because of exact details of the electrical design of the exchange and the phones, and where different components are placed, some countries such as the UK require slightly more complex household wiring than just putting all jacks in parallel. But in the US, that's all we do.

If you crack open a wall and look at your household telephone wiring, you will almost certainly find a surprising number of wires. Not two, but four. This, of course, corresponds with the four pins on a modular telephone jack. Your telephone only uses two wires (one pair), but dating back to the '60s it has been a widespread convention to wire homes for two separate telephone lines. It doesn't cost much more, but it gives the telco the opportunity to upsell you to a second line. This convention is reflected not only in the wiring but, as I mentioned, the connector.

The connector used for modern telephones is often called RJ-11, although that term is not exactly correct in a pedantic way that rarely matters. It's a little more correct, when speaking of the connector itself, to call it a 6P4C modular connector. 6P4C means six positions, four contacts---it could have six pins, but only four of the pins are actually populated. Two are for one phone line, two are for the other. If you actually have two phone lines fitted to your house, you will find that the single-line phones common in homes always use the same pair, so you'll either need an adapter cable or some jacks wired the other way around to use two lines. If you've ever lived in a house with two phone jacks right next to each other, it's likely that one of them is wired with the pairs reversed (or only has one pair at all) so that a standard single line phone can be used on the second line.

The phone wiring in your house joins your phones in parallel with a device formally called a Network Interface Device (NID), but often referred to as the demarc or demarcation point. This is because the NID is not just a technical object but an administrative concept: it is the border between your property and the telco's property. Inside of the NID there is usually a short phone cable (connected to your house wiring) plugged directly into a jack (connected to the telco's wiring). If your phone ever malfunctions, the telco will likely ask you to take it directly to the NID, unplug your household wiring, and plug your phone straight into the jack. If the problem goes away, it is somewhere in your household wiring, and therefore not the telephone company's problem. This is their preferred outcome, and you will be told to use your non-functioning phone to call an electrician.

The NID may be located in different places depending on the details of your house, when it was built, and when telephone service was installed. Most commonly it is somewhere outside of the house mounted on an exterior wall or skirting, often somewhere near the electrical service entry. There are plenty of exceptions, and especially in older houses the NID may be in the basement or crawl space. In some cases, mostly mobile and manufactured homes, the NID may actually be mounted to the telephone pole at the street or your property line, and the overhead or underground connection to your house is also your problem.

From the NID, the telephone line makes way to the exchange. In most cases today this will be as part of a telephone cable. A telephone cable is an assembly of many telephone pairs bundled into one sleeve, and they're either run along utility poles (lower than the electrical lines for isolation) or underground. Either way, your telephone line will be connected to the cable inside of a splice closure. For overhead wiring, splice closure are usually black plastic cylinders hung alongside the cable. For underground wiring, they're usually gray-green pedestals sticking out of the ground. Either one provides space for the many pairs in a cable to be spliced together, some to another cable but some to a drop line that runs to your house.

Most of the time, the cable does not run directly to the exchange. This is not exactly modern practice, but a common convention is to have two levels of "feeder" cables. The F1 cable is a very large cable that runs from the telephone exchange to a neighborhood. There, the F1 cable is spliced onto multiple smaller F2 cables that run along neighborhood streets.

The splice between F1 and F2 cables, and in general any splice between multiple cables, is usually done in a larger splice cabinet. Historically splice cabinets were sometimes mounted up utility poles, but this made them more difficult to safely work on so this arrangement is pretty much gone to history. Instead, modern splice cabinets are larger ground-level pedestals, usually a good 4-8 feet wide with large double doors.

There are several advantages to these splice points. First, they are obviously necessary for the original installation of the telephone infrastructure. Second, splice closure and cabinets are intentionally made for easy modification. This gives the telco a lot of flexibility in fixing problems. The world of local telephone loops is a dirty one, full of dirt and rain. Despite precautions, water has a way of working its way into telephone cables and can cause corrosion which makes pairs unreliable. When you complain, and the NID test shows the problem is on the telco side, they will likely just resplice your home telephone service onto a different pair back to the exchange. This swap from one pair to the other avoids the problem, which is a whole lot easier than fixing it. Actually fixing problems inside of telephone cables is a whole lot of work, and with subscriber numbers dwindling in cities there are usually lots of unused pairs so it's easy to swap them out.

In some areas, your local loop may not actually go directly to an exchange. It might go to something like a remote line concentrator, or a serving area cabinet, or a loop extender. These are all variations on the idea of putting some of the exchange-side equipment in a big curb cabinet, closer to your house. These arrangements are most common in suburban areas where local loop lengths are long and subscriber density is fairly high. I'll mostly ignore this, but know that some of the parts of the telephone switch may actually be in a curb cabinet in your case. These curb cabinets usually function as remote components of the switch and connect back by ISDN or fiber.

Once your telephone loop makes it from your phone, through your house wiring, down a drop cable, through an F2 cable, and then through an F1 cable, it arrives at the telephone exchange. There it often passes through an area called the cable vault, usually an underground space in or adjacent to the basement where cables enter the building, seeping water is drained, and pairs come out of the armored cable jacket. Everything before this point has been "outside plant," and is the realm of outside plant engineers. Now, we have entered the sanctum of the inside plant, and a different department of the company.

From the basement, pairs go to the main frame, basically a really big splice cabinet inside of the telephone exchange. The main frame allows exchange technicians to connect pairs to the switch as they please. If an outside plant technician has fixed your telephone problem by resplicing your house to a different pair, they will submit a ticket (originally a paper slip) to have the exchange technicians perform the same remapping on the main frame. Originally, if you stopped paying your bill, a ticket would be generated for an exchange technician to un-splice your phone line at the main frame. Today, both of these are often done digitally instead by leaving pairs connected to the switch's line cards and reconfiguring the line card mapping.

Which takes us to your local loop's next stop: the actual switch. The many local loops that a class-5 or exchange switch serves terminate (in the case of modern electronic switch) at what are called "line cards." The line card is responsible for managing all of the electrical aspects of a small set of local loops connected to it. Depending on the type of switch, the line card may perform ADC and DAC to convert your analog local loop to digital signaling for further handling by digital means. Or, it may connect your local loop to a device called a hybrid transformer that separates your call into two pairs (one for audio each direction) for further handling in analog form.

And that is your local loop. It's called a loop because the two wires, connected at one end by the switch and at the other end by your phone, allow current to flow all the way around. "Current loop" is the term in electrical engineering for this type of arrangement, and it's such a common means of conveying information by electrical signals that it's often only implied.

For there to be current through the loop, though, someone has to put some potential onto the line. Historically, phones were expected to provide power, but this practice had become obsolete by the end of WWII. In modern phone systems the loop power is provided by the switch. In the case of phones providing power, the phone contained a battery which was occasionally replaced by the telco. In the case of switch-provided power, AC-DC rectification was an imperfect art and there was a need for a backup capability in any case, and so the telephone switch would get its loop power from a very large battery. Because of this history, the normal potential on your phone line is known as battery power. People will sometimes shorten this to say that the switch "provides battery," especially in situations like test equipment or military field phones where it isn't always obvious which end battery power will come from. As another bit of telephone terminology, a telephone line with battery applied is sometimes called "wet," while one without battery applied is called "dry."

Battery power in the United States nominally comes from a series of lead-acid batteries producing a nominal 48v. In practice, there is some considerable variation. In older phone switches, a float charger was continually connected to the batteries and so held the battery voltage higher (around 52-54v) whenever the exchange had utility power available. Likely because of this, some countries such as Japan actually standardized 50v or 52v as the nominal off-hook potential. In newer equipment, battery voltage often comes not from batteries at all but from a regulated switch-mode power supply (that runs either off of external AC power or a battery bank of a potentially different voltage). It may therefore be exactly 48v, but some of these power supplies are actually regulated to 50v to match the typical behavior of older equipment. It really just depends on the device, and most telephones will function acceptably with well below 48v off-hook.

For historic reasons, telephone switches are mostly frame-positive. This means that battery voltage is often described as -48v. The difference doesn't really matter that much on the telephone end, but can be confusing if you are looking at documents that use different conventions. Owing to the 1/4" jacks originally used for telephone exchanges, the two wires in a telephone pair are called "tip" and "ring." Compared to ground, "tip" should be about 0v while "ring" should be about -48v. This "ring" is not to be confused with ringing power, to be discussed later. It's just the naming convention for the wires. Some phones, especially rotary types, will function fine with the polarity backwards. Most newer phones won't.

This talk of voltages, so far, has all been assuming that the phone is on hook. When a phone is on hook, the "hookswitch" disconnects the two sides of the local loop from each other, leaving an open circuit. The voltage across the phone thus settles at whatever voltage is applied by the switch. There are two conditions under which this voltage changes significantly.

First, when the phone is off-hook and you receive a call, the exchange needs some way to ring your phone. An age-old method of ringing phones is a magnetic solenoid (arranged so that its core will strike bells when it moves) with the line passing through it. The coil provides a very high resistance and so passes little current under normal off-hook conditions. When the switch wants your phone to ring, it applies a much higher voltage, and alternates it. In fact, this AC voltage is superimposed on the normal DC battery voltage, and is known as "ringing power" or "ringing voltage." Ringing voltage is AC at 20Hz by wide agreement (this works well for the simple mechanical solenoid ringers), but the potential has varied over time and by manufacturer. It also just tended to vary based on equipment condition, since it was originally produced by electromechanical means such as a motor-generator. 90v is fairly typical, but up to 110v AC is acceptable.

The fact that a telephone line can carry 110v AC is a surprise to many, and should discourage licking loose phone wiring, but from a fire safety perspective it's reassuring to know that the current of the ringing power is rather limited. The details of how it's limited, and to what level, depend on the telephone switch in use, but the current-limiting at the switch allows telephone lines to be handled as class 2 or power-limited for purposes of the electrical code.

The astute reader might notice an interesting problem here. Multiple telephones may be connected in parallel, but ringing voltage provides only limited power. If you have enough phones, will there not be enough ringing power?

Yes.

The implementation of this problem is sort of an odd thing. Federal standards created the concept of a "ringer equivalence number" or REN. When the REN was developed, the standard telephone instrument from Western Electric was the model 500, and other Western Electric phones for PSTN use were made to match the 500. So, a REN is defined as the impedance and resistance imposed by a single Western Electric model 500. Local loops are expected to provide enough ringing power for up to 5 REN. In practice, this is rarely a concern today. Few modern phones use the power-intensive electromechanical ringer of the 500, and if you look at the literature that came with a digital phone like a DECT cordless set you will likely find that it is specified as 0.1 REN. You can have a lot of modern phones before you run into problems.

Ringing current only applies when there is an incoming call and the phone is on hook. When you take the phone off hook, the hookswitch connects the two sides of your local loop together via the phone's "voice circuit." This prompts the switch to stop applying ringing current. Once the voice circuit is connected, the voltage across your phone drops considerably. Power is now flowing and so Ohm's law has taken control. Local loops are long and made of small-gauge wire; just the resistance of the telephone line itself is often over a thousand ohms.

Besides the wiring itself, there are two other notable components the loop current must pass through. First, there is some resistance imposed by the switch. The details of this resistance depend on the exact switch in use but the most common situation is that the phone line passes through a coil in a "line relay" assembly as long as your phone is off hook. This relay informs the switch logic of the state of your telephone line. The resistance imposed by the line relay varies by switch, and on many switches it's adjustable, at least in two or three steps. This allows an exchange technician to make an adjustment if your loop is very short or very long, to keep the loop current more in a normal range. 600 ohms is fairly typical for line relays.

The third important component is the voice circuit of your phone itself, which also varies by phone but is also typically around 200 ohms. Because the phone is in fact powered by the loop current, there is a certain requirement for enough power for the phone to operate. AT&T specified that the maximum local loop resistance should be 2400 ohm. At 2400 ohm and 48v battery power, the loop current will be 20mA. 20mA was about the lower bound at which the voice circuit in typical Western Electric models (such as the 500) performed acceptably. Modern electronic phones are often workable at lower currents (and thus higher loop resistance), but audio quality will become worse.

Too high of loop current can also be a problem. There isn't a widely accepted upper limit that I know of, but it was known by the '60s at least that high loop currents due to low resistance loops (usually found when the subscriber lived close to the telephone exchange) caused higher temperatures in the line relay which could result in early failure of the line card. This is one of the reasons line cards often provide an adjustment, so that short loops can have a higher line relay resistance to reduce the current. Modern line cards often monitor the current on a loop and raise a trouble ticket if it is abnormally low or high. Either one tends to indicate that there is a problem somewhere in the line. Modern line cards also provide better over current protection and will usually automatically disconnect battery power (and raise a trouble ticket) if the current goes significantly above normal. This is why you can't get greedy if you are trying to charge your laptop on Ma Bell's dollar. Older equipment may have just used a fuse, so at least newer systems have the forgiveness of auto-reset a few seconds after laptop-charging mistakes.

Under off-hook conditions, with loop current flowing, the voltage across your telephone will be much lower. 3-5v is pretty typical, but the value will vary with voice modulation as you use the phone and isn't very important.

It's important to emphasize that nothing in this system is very well regulated. Well, with modern equipment the battery voltage is often precisely regulated, but that's more a natural consequence of switch-mode power supplies than any aspect of the design of the telephone system. Battery voltage is 48v in the same sense that automotive electronics are 12 or 24v, in practice it's often higher. Loop current is limited but fairly loosely. The acceptable range is pretty wide in either case.

To put some more solid numbers on it, here are some specs taken from the manual for a PABX. They seem pretty typical, except the upper limit for off-hook voltage which seems unusually low. Many sources say 4-9v. Note the two different ringing voltage specifications, these are to accommodate two different national standards.

• On-hook voltage: 40-50 V DC
• Off-hook voltage: 4 to 6 V DC
• Ringing: 90-100 V AC 20Hz or 60-90 V AC 25Hz.
• Loop current: 25-40 mA (up to 80 mA in "unusual circumstances")

And for flavor, numbers I took from the spec sheet of a different business telephony product:

• On-hook voltage: -54v to 40v DC
• Off-hook voltage: -20v to -5v DC
• Current: 23-35 mA

Note the different voltage convention, and that all the numbers are, well, similar, but different. That's just how it is!

>>> 2023-04-30 los alamos - compound to county pt II

Where we left off, Los Alamos had become a county, but the town itself continued to be directly administered by the Atomic Energy Commission (AEC). The Atomic Energy Communities Act (AECA) mandated the AEC to dispose of the towns it owned by transferring the property to private owners or government agencies. This included not just the individual houses (which had all been rented by AEC) but utilities, parks, and other municipal property.

In 1963, shortly after the addition of Los Alamos to the AECA, the AEC started the process of transferring public resources. The schools were a relatively simple case, as a county school board had existed since the creation of the county in 1949, but it still took until 1966 for the AEC to give the school board title to the real estate the schools occupied. A case that seemed more complex, but also moved faster, was the electrical supply.

It's likely that the AEC had their own motivations. Electricity was fairly expensive and limited in Los Alamos, and the lab had an almost perpetual need for more power. Transferring the electrical utility to an outside agency might have benefited the AEC by moving system expansions into state and municipal financing and out of the federal budget. In any case the electrical system was actually reasonably cut-and-dry since New Mexico had a well established system of electrical franchise agreements and there were plenty of utility operators with experience in power distribution. As mandated by the AECA, the AEC put the electrical system out to bid.

One of the bidders was entirely unsurprising: The Public Service Company of New Mexico, today known as PNM [1]. The politics of Los Alamos utilities turn out to be one of the most complex issues surrounding the transition from AEC control, and PNM's attempt to purchase the electrical system is a great example of the debate.

First, it should be understood that electrical service in Los Alamos was expensive to offer due to the remote location and difficult terrain around the town, but Los Alamos residents had been largely insulated from this issue. It was the AEC that really footed the bill for electrical power, and residents were charged a rate that was designed to be similar to other electrical rates in the area---not to recoup the AEC's cost. There was a clear risk that electrical rates in Los Alamos would increase significantly after transfer away from the AEC, and much of the politics of mid-'60s Los Alamos was animated by this concern.

The debate over ownership of utilities played out in a series of delightfully mid-century newspaper ads taken out not just by PNM but also by organizations with names like "People's Committee for Private and Professional Utilities" and "Los Alamos County Citizens for Utilities." The debate was something like this: PNM argued that their large size and experience in the electrical industry would allow them to sustain low rates in Los Alamos. Some Los Alamos residents favored this argument, taking the view that PNM was an established, experienced organization that would operate the utility based on sound analysis and long-term planning. This kind of thinking seems to have been particularly popular in Los Alamos, where nearly all the residents worked for another huge bureaucracy that considered technical expertise its defining identity. Further, it was thought that if the county owned the utilities, the financial shortfall would likely be made up by increasing county property taxes---effectively increasing electrical cost, even if the rates stayed the same.

On the other hand, advocates of county ownership felt that a public operator, without the motive of returning profit to its shareholders, could ultimately keep costs lower. In their support, consulting engineers hired by both the AEC (to evaluate bids) and the county (to support their bid) concluded that it would be possible to operate the system profitably without exorbitant rates. County ownership was often seen as the more conservative option anyway, since it would allow for the privatization of the utility in the future by franchise agreement. To further assuage concerns about the county's competence to run the utilities, the county board advertised in the newspaper that it intended to directly hire staff from the Zia Company, the private contractor the AEC had used for maintenance and operations of the utilities.

Finally, the county had a rather unusual motivation to assume control of utilities. Because the laboratory was essentially the only industry in Los Alamos and was exempt from property tax due to federal ownership, the county had only a very limited ability to raise revenue. The county felt that the profit it could make from utility operations would become a key revenue source, particularly since the lab itself would be a customer. In the words of state senator Sterling Black, county-owned utilities would become "a club we can wave over the AEC." Considering the AEC's tendency to completely steamroll county decisions when they conflicted with federal policy, this kind of political power was extremely important.

The idea of AEC decision input by referendum was not a new one. Just a bit earlier in 1963, Los Alamos residents had gone to the polls to choose who would receive the Los Alamos Medical Center. Their choice was between the Lutheran Hospitals and Homes Society (an out-of-state concern but known to operate the hospital in Clayton at the time), the Los Alamos Medical Center Inc. (an organization formed by Los Alamos residents expressly to operate the hospital locally), or "neither." This issue had been substantially less contentious as, it turns out, the Lutheran organization had submitted their proposal to the AEC before it knew about the local organization. Once it came to their knowledge they apparently deferred to the local initiative, telling the Santa Fe New Mexican that they supported local operation of the hospital and would pursue ownership only if the county invited them to do so.

In the case of the hospital, though, we can already see the formation of the movement against local organizations and towards larger, more experienced organizations that were likely to offer a more predictable and stable---even if less locally driven---operation. The Lutheran organization won the referendum, and controlled the hospital until 2002, when another attempt at establishing a local hospital operator failed.

The story of the Los Alamos Medical Center illustrates some other challenges of the military-to-municipal transition as well. When the AEC built the Los Alamos Medical Center, it included housing for staff as part of the complex. As a result, the hospital had two, 24-unit apartment buildings directly attached to it. This seems to have established a tradition in which another condo building, owned by a group of physicians, was attached to the hospital in 1982. The odd mix of hospital and housing, and the resulting complex lease agreements, complicate ownership transfers of the hospital to this day.

The hospital referendum provided a clear example for the utility referendum, but the utility question attracted significantly more debate. This is perhaps ironic considering that the utility referendum was also less significant in the actual selection process. Because of the precise wording of the AECA, the AEC had determined that it would follow the outcome of the hospital referendum exactly. In the case of utilities, though, the AECA mandated that the AEC consider a set of technical and economic factors as well. The referendum would not be absolute, it would only be one of the factors in the AEC's decision.

A further complication in the decision was the exact nature of the county's legal authority. New Mexico counties are not normally authorized to operate utilities (although several do today under various provisions, such as Bernalillo County's partnership with the City of Albuquerque to operate the Albuquerque-Bernalillo County Water Utility Authority). In the legislative session at the beginning of 1963, the New Mexico legislature passed a constitutional amendment intended to facilitate the Los Alamos transfer, without further fanning local debates by requiring the City of Los Alamos to incorporate [2].

Although the amendment didn't take effect until 1964 following its approval in a statewide referendum, it was very much on the minds of the county government as they considered utilities. We will revisit this amendment in more detail shortly, but it had an important impact on the utility debate. In the 1960s, counties in New Mexico granted utility franchises under state regulations. This meant that such utilities had their rates reviewed by the state Public Service Commission. Because of the unusual status of Los Alamos county as the only "category 7" county (by the '60s now called an "H-class" county), the situation was somewhat more complex, and Los Alamos County probably had the opportunity to function as a municipal government instead, which allowed for complete home rule of utilities. In other words, it seemed that Los Alamos utilities could be regulated by the Public Service Commission, but they didn't have to be, depending on exactly how the county wrote up the paperwork. The upcoming constitutional amendment in fact solidified this situation.

One argument for county ownership was then that it would give the public better control of the utility, since it would not necessarily be subject to state rate regulation. One argument the other way was that a private utility operator could still be state regulated, if so desired. The regulation issue really only served to confuse the debate, as did a number of other events of 1963, including minor scandals of the county withholding documents and accusations of harassment and illegal campaigning by various individuals and political organizations in Los Alamos. Notably, the League of Women Voters, long the most important and highly regarded non-partisan political organization in New Mexico, was widely accused of compromising its impartial, non-partisan values by openly advocating for county control. In some ways this was actually not a very partisan position, as in a rather unusual political turn Los Alamos's Democratic Committee and Republican Committee joined together in endorsement of county ownership. The Santa Fe New Mexican quipped that the situation had become David vs. Goliath, except that somehow David was the mighty PNM (and Southern Union Gas, the company that had previously bought out PNM's gas operations) and Goliath was tiny Los Alamos.

A full accounting of the politics around the utility acquisition could quite possibly fill a book, as very nearly every aspect of Los Alamos life was somehow a factor in the debate. It only adds color to the situation that the federal officials involved, being AEC personnel, included well-known scientists like Glenn Seaborg. Just the newspaper editorials written in support of either position would make for a hefty binder. I will try not to get stuck on this forever and jump to the conclusion: David did not fell Goliath, and Los Alamos voters favored county control of utilities by 71%. Although the full transfer process would take several years to complete, the decision was made and the electrical and gas infrastructure were transferred to Los Alamos County, where they remain today.

There were yet more utilities to consider. Telephone service in Los Alamos had been provided by the AEC, with all service ultimately connected to the exchange at the laboratory. This, too, needed to be transferred, and in 1964 the AEC put out a request for bids. There was little precedent for government operation of telephone systems, and it was viewed as legally difficult, so it was clear that some private operator would step in. Two made an offer: Universal Telephone and Mountain States Telephone and Telegraph.

Universal Telephone was born of Milwaukee, Wisconsin. Wisconsin was known in the middle part of the 20th century for its particularly skeptical stance on telephone monopolization under AT&T. Wisconsin was among the first states to introduce pro-competition telephone regulation, namely the requirement that competitive telephone carriers interconnect so that their customers could call each other. Universal Telephone was thus among the most ambitious of AT&T's competitors, and no doubt saw Los Alamos as an opportunity to capture a brand new telephone market and demonstrate the capabilities of those outside the Bell system.

Mountain States, though, was an AT&T company, and brought with it the full force of AT&T's vast empire. The complex landscape of telephone regulation immediately became the key point of this fight, with AT&T arguing that the state telephone regulation agreements gave Mountain States the exclusive legal authority to operate telephone service in Los Alamos county. Far from a referendum debated in the newspapers, the future of Los Alamos telephony was mostly debated by lawyers behind closed doors. The AEC's lawyers sided with Mountain States, disqualifying Universal Telephone as a bidder. By the end of 1964 the writing was on the wall Mountain States would be the new telephone company (besides, AT&T rarely lost a fight), although legal arguments would continue into the next year. In early 1966, the deal was complete and Mountain States began to staff its Los Alamos operation. They had a lot of work to do.

Even into the '60s the telephone exchange in Los Alamos supported only 5-digit dialing. While 5-digit dialing is not unusual in some large private phone systems (and indeed is still in use by LANL today), the use of 5-digit dialing across a town was exceptional and meant that Los Alamos telephone users could not directly dial outside of the town. In 1968, Mountain States began construction on a new telephone building with a distinctive microwave tower for direct connectivity to the broader AT&T network. The new switch installed there, a 1ESS, offered for the first time touch-tone dialing, direct-dial long distance, and even three-way calling. At the same time, numerous new lines were installed to take advantage of the higher capacity of the new exchange. Telephone service became far easier to order in Los Alamos and on Barranca Mesa (not yet always considered part of the town), although White Rock would have to wait almost another decade for improved service availability.

Electronic telephone switching wasn't the only new idea in Los Alamos. The entire "H-class" county designation of Los Alamos was extremely confusing, and more and more the nature of the county was considered a mistake going back to its start in 1949. State legislators had tried to make Los Alamos a special kind of county that had the authority to do certain things normally withheld to municipal governments, but the piecemeal way they had done this created a lot of problems and required some sort of state legislation to address one problem or another nearly every year from 1949 to 1969. This included two cases of significantly reworking the county's enabling legislation, which is part of why it went from "7th category" to "class H" even though H isn't the 7th letter. Los Alamos was created as a whole new kind of county in 1949, it was made yet another whole new kind of county in 1955, major changes were made in 1963, and now it was still a mess. Something had to change, and the people of Los Alamos along with the state legislature were increasingly leaning towards the idea of incorporating like a city---but as a county.

In 1963, the state legislature passed Constitutional Amendment #4. In 1964, it was approved by state voters and took effect. This amendment, now article X section 5 of the New Mexico Constitution, created yet another brand new kind of county: the incorporated county.

The applicability of X § 5 was intentionally limited to Los Alamos, although the realities of legislation discouraged writing it that way. Instead, it was made applicable to any county with an area under 140 miles and a population greater than 10,000 (this latter rule mostly just to agree with other rules on incorporating municipalities). You can no doubt guess the number of counties under 140 square miles in New Mexico, a state known for its vast counties. Even tiny Bernallilo County, sized for its dense urban population, measures over 1,000 square miles. Despite its ostensibly general criteria, X § 5 applies uniquely to Los Alamos. It's even written such that any new counties that small would be excluded, only counties that existed at the time of its passage are eligible.

The amendment allowed Los Alamos county to incorporate, just like a municipality:

An incorporated county may exercise all powers and shall be subject to all limitations granted to municipalities by Article 9, Section 12 of the constitution of New Mexico and all powers granted to municipalities by statute.

In other words, it would resolve the problem of Los Alamos County's confusing city-county nature by making it, well, a real city-county. This concept is not unique nationally and some states, such as Virginia, rely on it extensively with "primary cities." San Francisco is a well known example of a city-county in the West. Los Alamos has much the same status today, but due to the history of the city being closely tied to AEC administration, the city-county title is rarely used. The Los Alamos government today identifies itself only as Los Alamos County (slogan: "Where Discoveries Are Made"), and its nature as a municipal government is relegated to the legal details.

Los Alamos's administrative situation is actually oddly obscure today. Wikipedia refers to Los Alamos proper only as a census-designated place and county seat, making no mention that the county it is seat of is actually a municipality as well. If I could mount one odd political campaign in Los Alamos it would be to restyle the government as The City and County of Los Alamos for aesthetic reasons, but then the strict limitations in X § 5 make it unlikely that the idea will catch on more broadly in New Mexico.

But what about the actual incorporation process? An incorporated municipality, under state law, requires a charter. The constitutional amendment provided that such a charter could be developed by a charter committee and then approved by a majority of the voters. This detail of the incorporation process would become a very tricky one.

The first major charter effort gained momentum in 1966, when a charter committee completed a draft county charter and began the process of gaining electoral support. In doing so, they brought the debate over public utilities back to life.

Incorporation would normalize the county's authority to franchise, operate, and regulate utilities, but changed the legal situation enough that the county would need to rebuild its administrative bureaucracy for utility oversight. The 1963 concern that utilities would become a "political football" (in the words of one county commissioner at least) were back in the spotlight, as the new charter proposed a utility advisory committee but gave it little actual authority and mostly deferred utility matters to the elected county commission.

Utility operations might seem like a small detail of municipal government but, as the original 1963 referendum shows, it was a very big detail in Los Alamos. The problem of the 1966 charter and its utility advisory committee lead to many newspaper articles and opinion columns, and a group calling itself the "Citizens for Representative Government" formed a major opposition to the '66 charter. While their name makes it sound like a broader issue, the representative government they sought was mostly the ability to put utility rates to referendum, denied by the draft charter. Another, less decisive problem was the charter's general increase in the powers of the county manager. Several county positions which were previously elected would be appointed by the manager, mostly positions related to finance and tax administration.

In a cunning political move, the Citizens for Representative Government seem to have made themselves the principal source of information on the draft charter. They mailed a copy of the proposed charter to every Los Alamos resident, including their phone number for questions. A nearly full-page newspaper ad featured the charter represented as arrows shot towards the hapless Los Alamos public, captioned "There's still time - duck now! Vote no February 8th." Major political points made in the papers included limited recall powers, the possible appointment of judges by the county council, and the lack of a requirement for competitive bidding on county contracts... but still focused on utility management and utility rates as the single largest issue.

Whether through good political organizing or the genuine public fear of politicization of town utilities, the '66 charter failed its referendum vote. The general consensus, reflected in the papers, is that the charter was mostly acceptable but there was widespread opposition to a few specific sections. Given the situation, there was not much to do except to start over again, and that they did. At the end of 1967, a new charter committee was formed. As an example, they worked from the new City of Albuquerque charter, one of several which rotated Albuquerque through various forms of city government in the mid century.

The second charter committee had been asked to work quickly, and they delivered on a tight timeline. In early 1968 a draft had been published and was the subject of much debate once again, and as the end of that year approached it had reached the form of a final draft. In many ways, this second charter was more conservative, leaving much of the county government to function the same way it had before. Like the City of Albuquerque at that time, it was based on a manager-commission model without an elected executive.

This time around, proponents of the charter leaned heavily on its amendment mechanism as a selling point. Being such a new municipality, and with an unusual administrative situation, it seemed clear that Los Alamos government would require flexibility. In an effort to address the specific complaints about the '66 charter, this new charter also formed an independent utility board with members appointed by the county council on a staggered basis... and direct authority over utilities, alongside a professional utilities director hired by the commission.

This new charter gained public support much more easily, apparently benefiting greatly from the lessons learned during the last attempt. In September it was approved by the county commission, and in December a referendum was held for public approval. This time, it passed---with lower turnout than the last effort, suggesting some mix of greater support with, well, less opposition. Still, not making enemies is a kind of success, and at the beginning of 1969 Los Alamos took its identity as New Mexico's only "incorporated county."

It took a lot of attempts for New Mexico to make the transition, perhaps five depending on how you count, but the townsite eventually did convert from AEC property to a municipality. We've even talked about the transition of the school district and electrical, gas, and telephone service. But what of water service? That story is complex enough I could go on about it for a while, and I probably will, in a part III.

[1] This "public service" naming is extremely common in older, central US electrical utilities, but should not be taken as implying public ownership. Nearly all of the "public service companies" are private corporations with no history of public ownership.

[2] It may be important here to understand that New Mexico is one of the states that achieves most major legislation through constitutional amendments. The New Mexico Secretary of State publishes a convenient, pocket-sized copy of the constitution, which is 218 pages long.

>>> 2023-04-27 los alamos - compound to county pt I

This weekend, I found myself staying in Los Alamos for a volunteer role in which I judge children on the quality of their software. Clearly this is not the kind of opportunity I would turn down, but I also always take an excuse to drive back up the hill. I only lived in Los Alamos briefly, but it left a big impression. It's a unique place in many ways, born of a rather unusual history.

The role that Los Alamos as a place, Project Y and the Los Alamos Scientific Laboratory as an institution, and more broadly the Manhattan Engineering Works played in World War II is widely documented. The Manhattan Project-era history of Los Alamos is actually surprisingly uninteresting to me, because few physical remnants of it exist in publicly accessible areas. There has been a concerted effort to offer more frequent public tours of places like Omega Canyon as part of the Manhattan Project National Historic Park, but the reality of DoE security requirements and complete paralysis of DoE outreach efforts during COVID mean that these have not made much forward progress.

Unfortunately, LANL remains littered with historic buildings---demarcated by a brown sign bearing an "H" placed by the laboratory historian---that no one other than employees will ever see. And employees don't tend to make much note, because when you work in the institution it seems rather unremarkable. The nuclear weapons program poses a fascinating historiographical puzzle that way, and I sometimes lament that the laser-focus of most nuclear history on the pre-1945 period leaves even many nuclear history enthusiasts oddly unaware of the evolution of DoE facilities in even the 1950s. While unfortunate, this blindness to the Cold War era is quite practical: once the arms race kicks in, the proportion of historical record that remains classified shoots steeply upwards.

This is all sort of a preamble to explain that I am not going to discuss the history of Los Alamos as it was founded, there are plenty of good sources on that. Instead, I'm going to get into some of the fine details of how Los Alamos became the place that it is today. I will also ponder as to whether or not Omega Bridge is, was, or could be rigged to explode, so stay tuned for that bit of urban legend.

There are aspects of the post-WWII story that ought to be fairly easy to research, though: the areas of nuclear weapons installations that are now quite public: formerly secret towns that are, today, just towns. From a formal perspective these sites are Oak Ridge, Tennessee; Richland, Washington; and Los Alamos, New Mexico. Los Alamos is, in my opinion, the most interesting, and it's certainly the most the one I'm staying in right now, and so I want to talk a bit today about part of the nuclear story that you don't often hear: The transition of a secret military installation to a mostly (but not entirely) normal town. A mid-sized one, by New Mexico standards.

Los Alamos was originally created as a military installation, most of it hastily constructed of temporary buildings and reused facilities of the ranch school that had formerly occupied the site. Not just housing but all municipal services were managed by the military, and of course during WWII the fact that Los Alamos existed at all was a secret (although not an especially well-kept one in Santa Fe). Shortly after the war it became clear that this posed a problem. From our perspective, many years later, WWII might seem to crossfade seamlessly into the Cold War, but that was not the case at the time. The conclusion of WWII was seen sometimes as the conclusion of war entirely.

In the years shortly after the Japanese surrender, the War Department was tasked with a disarmament of incredible scale. The War Surplus Administration was tasked with most of the liquidation, and repurposed several military installations (including Sandia Army Base, which it shared tenuously with what would become Sandia National Laboratories) as storage yards to hold the numerous assets (including not just airplanes but entire buildings) that it was expected to auction off. The clearance of military assets declared excess after WWII took decades to run its course and lead to the phenomena of the military surplus store, which have now largely faded away as our modern military industrial complex (and particularly its inability to produce replacements for anything) has eliminated the concept of military surplus.

The Atomic Energy Commission, formed almost immediately after the closure of the war to assume responsibility for the former Manhattan Project, was not at all immune. Especially the late '40s and '50s, when the Missile Gap had not yet become a leading political issue, the AEC was expected to tighten its belt. This meant finding a way to economize the enormous industrial complex it had created, unwinding all of the costly special measures that had been justified as wartime expediencies. One of these was the whole matter of having installed the Corps of Engineers as a functionally socialist administrative authority over entire towns.

In 1993, Los Alamos established a sister city relationship with Kremlyov, now called Sarov, and formerly a series of number-suffixed codenames typical of Soviet "closed cities." This unusual kinship of American military company town and Russian military company town is presumably rather strained by today's geopolitical events, but the sibling relationship of the cities is a profound one. Sarov is, to put it in simple terms, the Los Alamos of Russia. Los Alamos is still dominated (politically, culturally, and often physically) by a large government laboratory that employs most of its residents. Sarov is much the same. Sarov features rows of dull, government-constructed housing projects to which its residents were assigned. Los Alamos is much the same, except that today, those government housing complexes are sold and leased at rather exorbitant prices.

Here is a critical difference: while Russia has made some efforts towards normalizing its closed cities, the United States has eliminated them. Instead of a government installation, Los Alamos is today a democratically governed town that just happens to have an intensive dependence on the Department of Energy. It all started in 1949.

Most of what is now Los Alamos County was obtained by the Army by either transfer from other federal agencies (mostly the Forest Service, as Los Alamos is firmly within the Santa Fe National Forest) or purchase from the private holders. Construction of the laboratory began in late 1942. At this point in time, Los Alamos remained a steep, forested site on the border of two counties. The Army operated by the chain of command was not especially concerned with administrative boundaries. The entire area was simply considered to be a federal reservation---completely under the control of the federal government, which was not an unusual arrangement for military sites at the time.

From the beginning, Los Alamos enjoyed some very limited form of municipal government. A "town council," formed in 1943, consisted of seven members that served in an advisory role only. They represented the municipal needs of residents to the Army, the actual government of the installation.

Post-war, the federal enclave began to cause confusion [1]. Were Los Alamosans residents of New Mexico? During the Manhattan Project secrecy they had all semi-officially been considered residents of Santa Fe, but as Los Alamos came to fame they started to use their Los Alamos addresses on paperwork. Consequently, Los Alamosans were denied the right to vote, on account of their not residing within the State of New Mexico. This matter was thought resolved when the State Legislature specifically authorized Los Alamos residents to vote in state elections in 1947, but that decision didn't hold up to review by the New Mexico Supreme Court---a review prompted in part by a state court having refused to hear a Los Alamos divorce proceeding due to questions of jurisdiction, highlighting yet more problems with the federal situation. Adding insult to injury, the state Attorney General ruled that Los Alamosans would have to pay non-resident rates for hunting and fishing licenses. The legal environment was quite unsatisfactory and lead to an escalating campaign sometimes referred to as "Civil Rights for Los Alamos."

In an effort to resolve the conflict, US Rep. Fernandez of New Mexico introduced HR 54 to the 81st congress. The bill was actually quite simple: it would "retrocede" the jurisdiction over Los Alamos to the state of New Mexico, conditional upon acceptance of jurisdiction by the state. The matter was seen as quite urgent and so HR 54 was rushed through to pass before the New Mexico legislature ended its own session. Almost simultaneously, State Senators Roach (hailing from Hot Springs, which would later be known as Truth or Consequences) and De Vargas (Española, Los Alamos's closest neighbor) introduced SB 110. This bill authorized the governor to accept jurisdiction over the area, and quickly after the passage of both pieces of legislation, Governor Mabry wrote to President Truman indicating the state's acceptance.

The normalization of Los Alamos proceeded at a pace that is hard to imagine in modern government. The next month, a duo of bills continuing the process reached Mabry's desk. One created a new category of county, notable mostly for the fact that its officials would receive no state salary. The second sliced the Los Alamos Reservation out of Santa Fe and Sandoval Counties and established Los Alamos County in the resulting hole. Hot on the heels of the new county, the state legislature authorized the incorporation of the City of Los Alamos. A slew of other bills addressed more minor matters ranging from regulation of bus service between Española and Los Alamos to, well, the fishing licenses.

One might wonder why Los Alamos was established as a county, instead of being merely a city in Santa Fe or Sandoval county. The reason, unsurprisingly, was a matter of politics: Los Alamos residents wanted representation at the state level, and state senators and representatives were allocated by county. The formation of a new county would guarantee positions in the state legislator, the mere incorporation of a city would not. There had been an effort towards incorporating a city starting in 1948, but the proposed city charter was set aside in favor of the county. Ultimately, the project to create the county determined the structure of both the county and city governments.

Over the course of April and May of 1949, the newborn County, and its School District elected their first councils. While still largely a military installation, Los Alamos now had a government other than the AEC and Corps of Engineers. The fact that the AEC site manager is virtually always quoted in newspaper articles about the elections reveals the ongoing importance of federal largess. Indeed, while Los Alamos residents now elected their own county government, the actual governance of the city remained largely unchanged: the municipal government was the AEC, advised (but only advised) by the town council. In some senses, the county was now the municipal authority, but the AEC owned the entirety of the county, granting it superior power in many senses. The path to normalization would prove rocky, not least because of this city-county contradiction.

First, a legal issue relating to the state constitution prevented Los Alamos residents having a vote for what were ostensibly their state senator and representative in 1950. Even more, the creation of Los Alamos as a new type of county produced more problems than anticipated. In the years after 1949, numerous legal questions had to be resolved. The state legislator had to be asked to grant Los Alamos County the authority to pass ordinances: the state attorney general interpreted the 1949 act establishing it as providing only for the enforcement of state law, perhaps because the AEC was the de facto municipal authority. But as Los Alamos became less of a military installation, it seemed more important for it to have a body of municipal law outside of AEC policy.

In 1951, equipped by the state with the power of ordinances, Los Alamos County began to institute traffic law. This is not to say that Los Alamosans had been free to drive whatever speed they wanted, but town police had been enforcing regulations that were really more AEC rules of personnel conduct than laws. This was just one of the many small steps required to remove the AEC as the local authority, and the legal challenges of 1949-1951 paved the way for Los Alamos independence.

The operation of a town proved rather difficult, though, when considering that the town was directly interspersed with AEC offices and research facilities. First, the AEC had security concerns about what were now townsfolk, not necessarily laboratory personnel, wandering about. Second, the Corps of Engineers and AEC were rather explicitly looking to get out of the business of recreational facilities, roads, and even churches. In order to separate the operation of the laboratory from the operation of the town, they needed to be physically separated. The saying goes that good fences make good neighbors, but in Los Alamos it took a canyon.

I should explain, here, that the town of Los Alamos now consists of a series of mesas. Today, these are Los Alamos Mesa, North Mesa, and Barranca Mesa, tracking the town's expansion towards the north. Between the mesas are canyons, so steep that parts of them remain dimly lit during the day. The canyons were cut out of what was once a plateau by erosion, leaving the mesas behind. to the west a mountain ridge provided harder ground, stopping the progress of the expanding canyons. There, Diamond Drive skirts the ends of the canyons, connecting the mesas together. The overall layout is a bit like a hand, not in the sense of Michigan, but with separate fingers. The "palm," though, the west part of the city, is steep and rugged---as much so as the canyons in some places, limiting urban expansion.

Los Alamos Mesa, which we might call the eponymous mesa, is the oldest part of the town. During WWII, the core of both housing and laboratory operations shared the narrow protrusion, edging towards the cliffs as more space was needed. South of Los Alamos Mesa is Omega Canyon, and its great depths provided a level of isolation from the rest of the laboratory that made it a relatively safe place for experimentation. Los Alamos's first reactors operated there. As the lab expanded its nuclear and high explosive operations, both more facilities and more isolation were needed, prompting expansion both down canyons and to neighboring mesas. The lab expanded particularly to the south, onto South Mesa, across Omega Canyon from Los Alamos Mesa. South Mesa, already becoming a major part of the lab, was the perfect refuge for the reclusive installation.

In 1953, the lab up and moved. It was not so much a move as a replacement. Most of the original WWII structures on Los Alamos Mesa were already in poor or questionable condition, having been thrown together as temporary facilities for what was then a temporary project. Nearly all laboratory facilities on Los Alamos Mesa were demolished, their users moving to newer buildings primarily on a nearby part of South Mesa known as Technical Area 3, or TA-3 [2]. Facilitating this move, a handsome arched bridge over Omega Canyon, known as Omega Bridge, was built in 1951 to provide an easy trip between the two mesas that were otherwise separated by a particularly steep and difficult crossing. From 1953 onward, Omega Bridge mostly (but not at all completely) separated the lab from what came to be called the townsite.

Los Alamos was not exactly made normal, though. Even after the separation of town and lab the townsite continued to be a secure area in the eyes of the AEC, which operated a checkpoint (depicted in one of the most famous Los Alamos photographs) on the "Hill Road" up the side of Los Alamos mesa where visitors had to show identification and a plausible justification for their entry---a "closed city," just like its Soviet counterparts. Oddly, the opening of Los Alamos to the general public came with little fanfare or even discussion. In 1957, the AEC announced an administrative decision to discontinue the checkpoint, allowing just anyone to drive up the hill. It solicited comments from residents and received only three, two for and one against. The main objection seemed to be that residents had become accustomed to the "gated community" life and regretted having to start locking their doors. One newspaper article snuck in a bit of a jab about how Los Alamosans would now have to fear unexpected visits by their in-laws, a comment that seemed to reflect the general view that opening Los Alamos was inevitable and not that important of an occasion.

At the same time, Los Alamos began to grow. The post-WWII stand-down did not last long, and as relations with the Soviet Union chilled into the Cold War, the AEC suddenly became a top priority once again. One of the clearest symbols of the post-war expansion of the lab is White Rock. Los Alamos county consists mostly of the lab, the townsite, and an area south of the lab known as White Rock. White Rock is sometimes considered a "resurrected ghost town." Originally constructed as a temporary camp for construction workers, White Rock was completely abandoned for most of a decade, during which all of the temporary structures were demolished. In the 1960s, increased staffing at the lab created a housing crunch in Los Alamos County, and a brand new White Rock was built on the same site.

White Rock is an odd place---seldom has the term "bedroom community" fit a town better. Los Alamos is small even today, but White Rock is much smaller, and almost purely residential. It's curiously suburban, considering how far it is from the nearest large urbanization. While it fits into small Los Alamos county with the townsite and is just as close to the lab by some measurements, White Rock is also pretty far away. The most direct road between White Rock and the townsite passes directly through the lab. Today, it is restricted to badged employees only, although that's a fairly recent situation. Even so, it has always been the case that Pajarito Road between the two towns was somewhat restricted and prone to closures. The next best route between the two means driving all the way back to "the Y" of highways 502 and 4, basically leaving Los Alamos county in order to drive back into it again. It's a ten mile trip over some fairly difficult terrain.

The matter of White Rock's isolation provides a good example of the tensions between public administration and dependence on the AEC. White Rock has never been incorporated, but fell under the governance of Los Alamos County. While the County was nominally independent throughout the '50s and '60s, it continued to rely on the lab and the AEC to provide most basic services. Residents paid utility bills to the AEC for power and water and, in case of emergency, called the AEC fire department.

In the '60s, a house in White Rock burned to the ground. The response by the AEC fire department was very slow, a situation attributed both to the lack of a full staff at White Rock's small fire station and the fact that White Rock had never been connected to the lab's central fire alarm system (presumably a Gamewell fire telegraph network, common equipment at the time). The lack of a fire telegraph was compounded by some sort of dispute between White Rock residents and Mountain States Telephone and Telegraph. I have had a hard time finding more details on this (and you know it's the kind of thing I badly want to find details on), but the general situation seems to have been that there were very few telephone lines in White Rock and Mountain States had required exorbitant rates to install more.

This is a pretty typical matter of small-town politics, but here the politics were not between the residents and the county but between the residents, the county's barely teenage government, and the vast bureaucracy of a secretive agency of the executive branch. Town hall meetings were held, but progress was slow. There were many examples of disputes like this, often over rates, the promptness of repairs, and AEC's cooperation with new housing construction. The whole thing was very similar to another phenomenon of rural New Mexico politics, the various small towns and ranch houses that had received their running water (and sometimes electricity) from the railroad by consequence of their close proximity to a station or water tower. In both cases, residents seeking lower rates and modernized service found themselves fighting an uphill battle against a huge organization that saw them as a low priority, if not an outright expense.

These problems didn't exactly surprise anyone---the military has found itself in similar situations many times, and in some ways this was a military problem, Los Alamos having been willed into existence by General Leslie Groves. Nuclear weapons, though, had been divorced from the military by the creation of the AEC. Despite the military legacy, it was now a civilian operation, and the federal government was not keen to see the AEC build a "force support" and base operations organization that paralleled that of the military [3].

In 1955, congress passed the Atomic Energy Community Act. The Act ordered the AEC to begin to sell off their housing stock, transfer commercial, recreational, and religious property to outside companies and organizations, and hand off the operation of utilities to local governments or private providers. The goal of the AECA was to eliminate the AEC towns, encouraging residents to purchase the housing they occupied (by giving them priority status and a slight pricing discount) and incorporate governments or start organizations to assume control of public services. The number of words the act spends on the issue of government-owned churches highlights the extent to which the AEC had completely operated AEC towns, but under the AECA the AEC was set to wash its hands of the whole municipal government exercise within five years.

There was one catch: the AECA did not apply to Los Alamos. In its original form, it applied only to Richland, WA (at the Hanford site) and Oak Ridge, TN (at Site Y-12).

The trouble was this: not only residents of AEC towns but also the counties and states that contained them feared the consequences of an AEC withdrawal. Not only was the AEC responsible for just about all public operations, AEC towns were mostly found in remote areas where the cost of these services was quite high. This was true of Richland and Oak Ridge, but it was especially true of Los Alamos, which remains difficult for utilities to reach today. The end of AEC water and electrical service might be the end of water and electrical service at all, the residents being unable to afford to continue them.

The AECA tried to address this problem on a few fronts. First, it created a pool of funding from which the AEC would pay not only expenses related to the transfer but also "local assistance funding" for ten years. The amount of "local assistance funding" was to be determined by the AEC, but it was specified that it should cover reasonable costs incurred by the newly "self-sufficient" communities. Tellingly, the Act says that in determining the payments the AEC should consider "the fiscal problems peculiar to the governmental entity by reason of the construction at the community as a single purpose national defense installation under emergency conditions." It was understood that AEC towns were not exactly financially efficient, and in most cases were left to resolve considerable infrastructure and housing problems resulting from the Army's emphasis on speed over sustainability.

The AECA even included an interesting sort of guarantee of a future for these towns. It stated that purchasers of housing from the AEC would be repaid from AEC funds if, at any point in the following fifteen years, the AEC failed to maintain certain levels of employment and population in those towns.

Still, for Los Alamos the problems were seen as too great. It took the Army to build Los Alamos and it just about kept the army to keep it running. I will likely write a follow-up post on the specific matter of water and electrical supply in Los Alamos, but consider that drinking water for Los Alamos had to be obtained from wells that were drilled 1,500 to 3,100 feet deep... starting from the canyon floors. The water produced by those wells was then pumped upwards of 1,000 feet uphill through rugged terrain. Just the cost of the electricity to operate the system was considerable.

Los Alamos's exclusion from the AECA did not last long, though. Likely the relative success of the program in Richland and Oak Ridge was encouraging, and in 1962 congress amended the AECA to add Los Alamos to the list. The enthusiasm was measured, though: most of the 5-year deadlines in the AECA were extended in the case of Los Alamos, with the deadline for transfer of utilities stretched to 1998.

The land transfer program was slow getting off the ground. It took until 1965 for the first sales of AEC-owned housing to begin, but that year many Los Alamos residents became homeowners rather than renters. The process was not entirely straightforward. Army housing built in Los Alamos consisted heavily of duplexes. The AECA stated that the "senior" resident of a duplex, typically the one who had lived there longer, had the priority right to purchase the whole building. In practice, this involved a frustrating paperwork process of obtaining an application from the AEC office and submitting it, then waiting to hear the AEC's determination of priority purchaser status, and then putting in another application to purchase and waiting yet longer.

Still, by early 166 several hundred housing units had been sold, with a little more than ten additional sales being handled each day. With a total of over 1,500 units to sell the process would take most of the rest of the year, but the pace was still impressive considering the complexities of real estate transactions---especially with the federal government. In many cases the AEC was financing the sales, offering a mortgage backed by the AEC itself to the buyers, and the 15% discount against appraised value for the single-family units made it a hard deal to turn down. Many of these duplexes are still standing today, often with surprisingly ambitious additions.

The fervor of the housing sale process was accelerated by the housing crisis. It is difficult to find housing in Los Alamos today, and this has been true almost since Oppenheimer first selected the site. '60s reporting on the housing sale process is invariably accompanied by coverage of the general housing shortage and other projects intended to alleviate it, like a planned new apartment complex in White Rock, the demolition of a number of four-plex "Sundt" units to make room for more dense housing, and the planning of a new apartment complex on the edge of Los Alamos Mesa. The concept of developing the bottoms of some of the wider canyons as neighborhoods was already in play at this point in time, and as we will see later this plan (and its challenges and controversies) has been an ongoing part of housing politics.

Urban transformation brought its inevitable controversies. During the sale process, residents of some Los Alamos homes (such as the "Denver Steel" homes found near the high school and named for their origin and steel construction) complained that the AEC was expecting similar prices for homes in their "more congested" area. AEC management rebutted to the newspaper with a set of statistics on the ratio of home sizes to lot areas, revealing that even in cramped Los Alamos residents enjoyed lot sizes that would make a downtown Albuquerque resident such as myself jealous [4].

The concern over increasing Los Alamos density was just one aspect of the urban planning confusion. The AEC's administration of the town had erred on the side of the pragmatic, and on account of the housing shortage the AEC continued its policy of growth even as the sales process was ongoing. On the other hand, the residents tended to favor preservation of the town's forested, green space studded nature---a wonder of mountainous northern New Mexico. One particular flashpoint was a large grassy lot near the Los Alamos Community Center (today the Fuller Lodge Art Center). The AEC platted the lot for commercial use, while the county ruled that the space should be reserved as a park. The controversy went on for months, with such contributions as a letter from the New Mexico Garden Clubs calling for preservation. The fact that the AEC owned the land gave it a lot of authority over the matter, but by the next year the process of selling the Community Center area, and the Community Center itself, to the county and private interests had already begun. Ultimately the AEC tended to pursue its original policies for as long as it owned a section of town, with priorities shifting to those of the county once an area had been transferred.

The AECA had an express goal of making AEC towns self-sufficient, while still being comfortable enough places to live that the AEC could readily attract staff. While this provided some built-in motive for the AEC to support the town, there remained inherent tensions between the interests of the bomb builders and those of a small town in northern New Mexico. The AEC, after all, was not exactly democratic. Despite the incorporation of the county, the people of Los Alamos still felt that they had a limited ability to actually influence policy. Besides, as the AEC began to sell off property, its surprisingly informal, ad-hoc authority in the town began to subside. The transfer of the schools to the ownership of the county school board in 1966 was taken as one major sign of the AEC's withdrawal from civil life, but that withdrawal left a rather significant hole: the city government.

The mid '60s made it clear that Los Alamos needed a city government, but given the politics of the area there was surprisingly little appetite towards incorporating Los Alamos as a city. It's always hard to divine the reasoning for these political decisions when they were made so long ago, but numerous newspaper articles and letters suggest one clear reason: Los Alamos county consisted almost entirely of the town of Los Alamos by population, except for White Rock. The residents of White Rock were already frustrated by the perceived lack of consideration they received, and incorporating Los Alamos to leave White Rock as "unincorporated Los Alamos County" seemed certain to worsen the problem. Besides, the AEC transfer program had to give land to an entity that existed, not one that might exist in the future, so essentially all public land and resources were being transferred to Los Alamos County. In many ways, such as the operation of parks and schools, the County was the municipal government. The movement of Los Alamos politics shifted away from city government entirely, and towards the amplification of the County's power. Just a few years later, this would lead to one of the more curious events in New Mexico's municipal law.

Let's save that for part II.

[1] Federal enclaves always do, and the concept of a "federal enclave" was mostly eliminated by SCOTUS in 1953, Howard v. Commissioners, a case dealing with another WWII war expedient federal installation. That one wasn't nuclear in nature, but instead a Navy ordnance plant in Louisville. The City of Louisville thought its employees ought to pay city taxes, the employees didn't want to, and so legal history was made. They had to pay the taxes.

[2] The practice of organizing land into "Technical Areas" comes from the Corps of Engineers and was historically seen at a number of military installations. The term "Technical Area" was originally intended to distinguish areas used for research and experimentation from cantonment areas, used mostly for personnel services and housing. Today, the convention has mostly faded from military use but is almost universal at nuclear weapons sites. Because renumbering TAs is seen as potentially confusing or at least distasteful, the TA numbers often reflect history more than geography, such as LANL being headquartered in TA-3. Sandia is headquartered in TA-1, but has no TA-2 at all, etc.

[3] Not many decades later, the military would also try to get out of the business of operating bases, significantly reducing the amount of military housing and privatizing the operation of what remained. The current state of most military bases suggests that the decision for the AEC to no longer provide housing to employees was probably the right one, despite the challenges it has created at Los Alamos today.

[4] My house doesn't actually even fit on my lot, a perennial cause of extra paperwork and frustrated surveyors. Fortunately most of Los Alamos was platted by the Corps of Engineers as part of the process of contracting housing construction, so the surveying situation is relatively straightforward.

>>> 2023-04-20 different kinds of differential

On the front page of HN today was an [article with a confusing headline], "Farmers 'crippled' by satellite failure as GPS-guided tractors grind to a halt." Of course the headline doesn't actually say this, but it does seem to imply that there has been some kind of failure of GPS.

This is not quite as outlandish as it might sound. The GPS system, maintained by the US Space Force, has occasionally suffered from serious funding shortages and bureaucratic paralysis. In 2009, the GAO issued a concerning report. They found that the program to launch a new generation of GPS satellites was so far behind schedule that the system was in danger of falling below the required 24 satellites. This would result in partial outages of GPS in different parts of the world, and could potentially take years to resolve. In response the Air Force issued a set of strongly worded statements insisting that they had kept GPS working for fifteen years and would consider to do so, and moreover made some changes to accelerate the GPS-III program.

Fortunately, we are now largely out of the woods on this issue, as not just GPS-III but subsequent designs are in service and the GPS constellation has been restored to its full planned size---including on-orbit spares for use in case of an unexpected failure. This is not to say that the administration of GPS is all good news; the GAO continues to issue more or less annual reports on how military programs to acquire more advanced GPS receivers (supporting the newer "M-code" signal) are badly mismanaged. But at least the civilian aspect of GPS should be quite reliable for years to come.

In any case, any significant failure of GPS would become major international news. Another perennial topic of GAO reports is the failure of the US government and infrastructure operators to develop any meaningful backup or alternative for GPS. A GPS failure would cause huge swaths of transportation and communications infrastructure to malfunction. While I remain a huge advocate for the construction of the terrestrial PNT (position, navigation, time) technology called eLORAN, it would cost money, and not in the F-35 kind of way, so it's unlikely to happen.

What happened in Australia, the article tells us about halfway through, was actually a problem with Inmarsat-41. Inmarsat is a bit of a historical oddity, in the same category as ARINC (Aviation Radio Inc.). It was founded as the International Maritime Satellite Organization, a non-profit entity sponsored by the UN's International Maritime Organization to develop a satellite network for emergency communications at sea. In 1998, though, Inmarsat was privatized, becoming a British company. Despite its historic legacy, Inmarsat is today just one of several major commercial satellite communications networks. It offers two-way telephony and data service, and carries both first and third-party broadcast services.

One of these services is SBAS, the Space-Based Augmentation System. "Augmentation System" is a common, if not very specific, term for GPS correction systems. The topic of GPS correction systems is sort of complicated, and we're going to celebrate 04/20 by going into them in some depth. Speaking of celebrating 04/20, this might be more of a ramble than a well-formed narrative, but there's a lot of interesting ideas related to PNT correction.

First, let's start with the concept of GPS augmentation systems. GPS fixes are subject to a number of sources of error. A high-quality GPS receiver with a sufficiently long observation time can eliminate most of the receiver-based positioning error, but there remain some pernicious errors which are hard to eliminate. Two of the most prominent are orbital ephemera and atmospheric effects.

First, orbital ephemera: to produce a GPS fix, receivers need to know the locations of the GPS satellites in orbit. In order to provide this information, ground stations observe the locations of GPS satellites and produce orbital ephemera, sets of parameters that describe the satellite's elliptical paths in relation to the earth. As the term "ephemera" suggests, these are "point in time" measurements that describe the current path of the satellites. Details of how space works like gravitational perturbations mean that satellite orbits are prone to changes, and satellites sometimes navigate to correct their orbits. This is why observations are used to determine the ephemera, and these observations are carried out on a continuous basis.

The GPS operators at the Second and Nineteenth Space Operations Squadrons regularly upload updated ephemera to the GPS satellites which transmit them for receivers to use. For practical reasons the ephemera are transmitted at a very low bitrate and can take some time to receive, which can contribute to "cold start" times on traditional GPS receivers of upwards of ten minutes. Most modern GPS receivers use some form of "assisted GPS" to reduce this time, with the most common example being the widespread practice of smartphone GPS receivers obtaining current ephemera from a web service instead of "waiting for it to come around" on the low-rate GPS data feed.

The problem is that new ephemera are only produced hourly, and the orbits of satellites varies on a minute level too rapidly for hourly observations to keep up with. Additional latency in the upload process means that the ephemera transmitted by GPS satellites can be several hours old. The observations are somewhat limited in precision anyway, considering the incredible precision modern GPS receivers are capable of. So, variation in the actual orbit of satellites from the ephemera leads to ephemeris error in GPS fixes.

Ephemeris error isn't actually that big of a source of error, but it's an interesting one so I wanted to talk about it anyway. Besides, it has introduced us to the term "assisted GPS" or "AGPS," which is unrelated to but often confused with augmentation systems. Assisted GPS just refers to the use of an alternate data path, such as IP, to obtain the current GPS network information that receivers need to make a first fix.

Second are atmospheric effects. GPS positioning relies on extremely accurate time-of-flight measurements. Unfortunately, the atmospheric is a weirdly complex thing, and numerous effects (such as "weather") result in radio frequency radiation traveling by indirect, non-linear paths. This problem is especially acute when dealing with satellites since, well, they're up in space, and so the received signals have to travel through a whole lot of atmosphere to get down here. This includes the ionosphere, which interacts with RF in particularly strange ways and can add significant travel time.

There are a number of approaches to reducing these errors, but one of the most common is differential GPS or DGPS. DGPS relies on a convenient property of most sources of GPS error: satellite position, satellite clock drift, and atmospheric conditions all tend to contribute a pretty similar error across a region. This means that if you can measure the error of GPS fixes in one place, the error at other nearby places is probably pretty similar. This is exactly what DGPS does: a reference station with a precisely surveyed location uses a high-quality GPS receiver to obtain very accurate fixes. The difference between the GPS fix and known location is then distributed as a correction signal that can be applied by other GPS receivers in the same area, canceling much of the inaccuracy. DGPS is one of the most common types of augmentation system, and is widely used by higher-quality GPS receivers.

Numerous augmentation systems exist, which makes them more complicated to talk about. The big issue is how the correction signal is distributed. Historically, one of the earliest major DGPS systems was the NDGPS or Nationwide DGPS. NDGPS is operated by the US Coast Guard (having originally been developed for maritime navigation) and uses a network of site across the US that transmit correction signals at around 300kHz. NDGPS has a great historical detail: when it was expanded from coastal stations only to a nationwide system, a great deal of the new inland correction stations were installed on recently retired sites of the Ground Wave Emergency Network (GWEN). GWEN was a short-lived survivable radio system operated by the military for continuity of government and command and control purposes, and much of the modern conspiracy theory around 5G can be directly traced to historic conspiracy theories surrounding GWEN as a government mind control system.

NDGPS is now being retired, though, as WAAS is viewed as a replacement. WAAS, the Wide Area Augmentation System, is the FAA's version of the same concept, intended for aviation use. Most WAAS reference stations are installed on the roofs of air route traffic control centers, with some others added as needed to fill gaps. WAAS differs from NDGPS in an important way: WAAS correction signals are actually distributed by three different commercial communications satellites. WAAS comes from space, just like GPS itself.

This is a good time to point out an odd bit of terminology: DGPS and augmentation systems are essentially the same thing, but for largely historic reasons "DGPS" usually refers to correction signals distributed by terrestrial radio while "augmentation system" usually refers to correction signals distributed by satellite.

You can probably see where this goes with Inmarsat. WAAS provides complete coverage of North America, but not of elsewhere. Inmarsat operates equivalent space-based augmentation signals as a contractor to several national governments, including Australia, New Zealand, and likely soon the UK. And that's what broke: some kind of problem with the satellite caused a disruption of the SBAS feed for Australia.

While we're talking about GPS augmentation, we should also talk briefly about RTK, Real-Time Kinematic correction. RTK, DGPS, and augmentation are sometimes used almost synonymously, but once again there is a conventional difference in the meanings. RTK most literally refers to the observation of the phase of the the GPS signals. The phase can be used as additional timing information to estimate the distance between the receiver and the satellite. Microwave GPS signals have a short wavelength (about 19cm), and electronics for observing phase difference can be made very precise, so this method can produce extremely accurate fixes. The problem is that RTK only tells you where the receiver is within a wavelength, or in other words it tells you where you are in a 19cm window, but not which 19cm window you're in.

To resolve this mystery, RTK is almost always used with a reference station so that the phase can be compared between a fixed point (with a location determined through long, 48-hour+ observation) and the moving receiver used for surveying in the field. Since RTK is mostly used by surveyors who are striving for huge accuracy and aren't in a hurry, it's most often done with a portable reference station installed nearby for the duration of the surveying project. If you've ever run into a skywards antenna connected to a chained-down Jobox with a solar panel, it's very likely an RTK reference station installed for some highway construction project. While RTK tends to imply DGPS, it's a separate technique, and produces even higher precision than DGPS or augmentation. It's also possible to use RTK without a reference station at all, by analyzing fixes from multiple locations later... a method referred to as virtual reference station.

So how much does this stuff actually impact our lives? Well, for the most part we tend to use smartphones for PNT in our daily lives, and smartphones have a somewhat different bag of tricks that relies on the cellular network to assist positioning. The thing is, smartphone GPS receivers really aren't very accurate at all. They rely on network assistance not for precision but in order to obtain any fix at all. Smartphones are constantly used in situations like urban environments and even indoors where GPS reception is poor and subject to huge multipath error. Cellular receivers bootstrap based on hints from the network (using the known locations of cellular towers) to accelerate GPS fix, and to produce a fix at all when they can't detect signals from a sufficient number of GPS satellites.

And that's it on PNT for today, but maybe I'll come back to shill more for eLORAN in the future.

>>> 2023-04-10 solving problems with chatgpt

One of the foundational goals of computer technology, at least as understood by popular culture, is to automate away our jobs. When your job is (put simply) to get computers to work correctly, there's sort of an irony to this view of things. Still, just within the span of my career there have been technological advancements that have both automated away parts of my job (Puppet might be an example here, the prior art of CFEngine notwithstanding) and given me plenty more to bill time for (Kubernetes).

With the rise of large language models (LLMs), there is a large camp of people who think these ML applications are going to automate away larger portions of more jobs. I find myself curious: should I fear for my employment?

The thing is, a large portion of my profession could be accurately described as "Googling it," although lately I'm racking up a Kagi bill instead. Obviously this is an important strategy for things like error messages where it's often faster to see if someone has solved the same problem before than to figure it out from first principles. But it's sort of true for routine tasks as well... maybe my memory is just unusually poor, but there's a huge set of tasks where I will repeatedly look up the same Digital Ocean guide because I just want to copy-paste their config file snippets instead of having to look in docs to remember the names of the right directives.

Of course, what makes me a "professional" is that I have opinions about the correct ways certain things should be done, so I often ignore parts of these guides or make changes to suit my preferences on important issues like Unix domain sockets or localhost network sockets for communication with application servers. In a way it's sort of silly to follow some guide on best-linux-tutorials.biz and then just ignore the parts I disagree with but, well, that's what I tend to do. So I wonder: would a better approach be an LLM?

In today's episode, I'm going to ask ChatGPT for guides for some increasingly complex Linux sysadmin and DevOps tasks and then see whether or not I agree with its output. I don't really intend this as that serious of a critique of the use of ML in operations but hey, maybe we'll learn something useful!

The LEMP Stack

Back when I was a child first learning to hate software, the dominant stack for websites was "LAMP." This stood for Linux, Apache, MySQL, PHP. Nowadays it seems that at least a few people are saying "LEMP" for Linux, eNginx, Mymariadb, PHP? I don't like this, aesthetically. In any case, there's a number of things in PHP that I tend to deploy a lot, Dokuwiki being a prime example. This means that I am repeatedly wanting to set up Nginx to proxy requests to PHP-FPM, which is not difficult but does involve remembering a couple of config directives and where php-fpm puts its socket by default. Most of the time I don't actually want an RDBMS (personally I usually just use sqlite for everything) so for a long time I've googled for some guide and copied their snippets while ignoring the parts about MySQL/MariaDB. Let's see if ChatGPT has something more to the point for this very common and simple task.

Prompt: "how do I set up nginx and php?"

To set up Nginx and PHP on your server, follow these steps:

1. Install Nginx on your server:

For Ubuntu/Debian:

 sudo apt update
 sudo apt install nginx

For CentOS/RHEL:

 sudo yum update
 sudo yum install nginx

The output starts off strong by offering snippets for both "Ubuntu/Debian" and "CentOS/RHEL." These two cover the good majority of the Linux server landscape, and while I could quibble with the label "CentOS/RHEL" rather than something that doesn't invoke the mostly-dead CentOS project like "RHEL/Fedora," ChatGPT is following the same convention most people do.

The instructions it provides are pretty reasonable, but there's something interesting here that those of you familiar with the two dominant Linux package manager families might detect. For Debian, it tells us to 'update' and then 'install.' for RHEL, it tells us to 'update' and then 'install.' These are neatly parallel except that the 'update' subcommand of apt and yum do fairly different things! On Debian it will retrieve the repo metadata and then install nginx (upgrading software is a different subcommand), on RHEL it will upgrade all of the managed software on the system and then install nginx (repo metadata is retrieved automatically by yum when it exceeds a configured 'stale' timeout).

I'm not sure how much we can make of this. It seems likely to me that we are seeing ChatGPT's lack of understanding of the underlying material: it is extremely common for people to 'update' and then 'install' on both platforms, so each in isolation is pretty reasonable, but it's odd for it to put them in parallel without noting that they will do different things. If the 'yum update' were an intentional choice we should see 'apt upgrade,' but we don't.

1. Install PHP and the required PHP extensions:

For Ubuntu/Debian:

 sudo apt install php-fpm php-mysql php-mbstring

For CentOS/RHEL:

 sudo yum install php-fpm php-mysql php-mbstring

This is a little odder. It says "the required PHP extensions" which are apparently mysql and mbstring. It's not clear where or why these are required, which makes me think ChatGPT got this command from some guide on deploying specific software. WordPress might be to blame here as it requires mbstring, and the sheer number of LAMP/LEMP guides for the assumption that we will need the mysql extension.

1. Configure Nginx to use PHP:

Create a new server block configuration file in /etc/nginx/sites-available directory with the following content:

 server {
   listen 80;
   server_name example.com;
   root /var/www/html;
   index index.php index.html index.htm;

   location / {
       try_files $uri $uri/ /index.php?$query_string;
   }

   location ~ \.php$ {
       include snippets/fastcgi-php.conf;
       fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
   }
 }

Replace example.com with your domain name and php7.4 with your installed PHP version.

This is pretty much what you would find in any guide. One interesting thing is the 'try_files' directive. There are about a million ways to set this up in nginx and some people prefer different approaches, but what we see here is pretty typical of php setup guides and should work fine. The catch is explicitly passing the $query_string which sort of gets eaten otherwise when nginx internally redirects to use index.php.

1. Test your Nginx configuration:

   sudo nginx -t

2. Reload Nginx to apply the new configuration:

   sudo systemctl reload nginx

Hey, nice work! A surprising number of people don't know about '-t' to validate config files.

1. Create a PHP file to test if everything is working:

   sudo nano /var/www/html/info.php

Add the following content:

 <?php
   phpinfo();
 ?>

1. Open your web browser and navigate to http://your-server-ip/info.php. You should see the PHP information page.

That's it! You now have Nginx and PHP installed and configured on your server.

So yeah, this isn't exactly part of the install process, but still very normal for guides.

Overall I'd say this is about an 8/10. The only things I disliked were:

1. Telling the user to upgrade packages on RHEL... not an unusual thing in guides to update first "for good measure" but I personally feel that it's not a great practice to tell the user to 'yum update' without explaining that will update anything.
2. Telling the user to install mysql and mbstring, which aren't really going to do any harm, but if you're going to include a "common" set of PHP extensions in the instructions it should be bigger than just these two. It's just odd.

'du' and 'df' disagree

This is actually an interview question I like to use for Linux-intensive positions. It might seem a little in the weeds but to be honest I think I run into this situation in real life probably at least once a month. The problem looks like this: tools like 'df' tell you that a partition is 99% or 100% full, and you might be having problems because of the full partition. But when you use 'du' to investigate, it shows a total usage that is appreciably less than 'df' does. Why do these two-letter commands not get along? is dd somehow the answer?

Prompt: On Linux, a partition is full but du doesn't show as much data as df does.

If you are seeing a discrepancy between the output of du and df on a Linux system, where df reports that a partition is full but du does not show as much data, it's possible that there are files that are being held open by processes and therefore are not being deleted even though they have been unlinked (deleted).

This actually isn't the answer that I expect but it's a reasonable answer. This could very well be the cause, I just find that this situation is less common than having a mount that "shadows" data on the volume at the next higher mount point. For example, say you have a volume mounted at / and a volume mounted at /home. If you have some kind of problem that results in /home failing to mount, or changed the filesystem setup at some point, or even have weird boot-order problems, you can sometimes end up with data in /home/ on the volume mounted at /. This data counts against the usage of the volume at / but won't show up in tools like 'du' since it's "shadowed" by /home/ now being a mountpoint to another volume. Easy problem to have if you are rearranging the volumes/mounts on a system that's already been in use.

I'll omit the rest of the output, but it makes sense. It tells us to use 'lsof' to figure out what process is keeping the files open and then kill that process. The guide does include the requisite paragraph that you need to investigate to determine how this happened in the first place.

Overall I'm giving this a 5/10, because it gave a reasonable answer but omitted the possibility of mount shadowing. Which of these two causes is more likely depends on the system and workload, but I think a thorough answer should include both possibilities.

Per-process network policy

One sort of common "advanced" Linux networking scenario is when you are using a full-tunnel VPN and want to route all traffic through it, but you have to get the VPN itself to connect to its endpoint without trying to go through itself. There are a few ways to solve this problem, but one of the less common and (in my opinion) more elegant approaches is to get the VPN service to use its own special routing table. This same concept also makes it easy to set up firewall policy around the VPN to prevent leakage. Now there are a lot of caveats to this and I'm really just talking about userspace VPNs here, but that probably makes it a good challenge for ChatGPT.

prompt: How do I configure per-process route tables on Linux

To configure per-process route tables on Linux, you can use the ip command and the setsockopt() system call.

Here are the steps to follow:

1. Create a new routing table:

   sudo echo "200 mytable" >> /etc/iproute2/rt_tables

This will create a new routing table with the name mytable and the ID 200.

It maybe seems a little weird that this is the way to create a routing table but it's fine. The iproute2 stuff that facilitates multiple routing tables without multiple headaches is pretty new and there isn't a lot of porcelain around it... and I'm not sure there ever will be, this isn't a very common task and while editing the file seems a little old-school compared to most of the contemporary network tooling it works just fine. There's very little need to expose this over a bus or anything.

1. Configure the routes for the new table:

   sudo ip route add default via dev table mytable

This command will add a default route to the new table with the specified gateway and interface.

I will take this moment to make a couple of funny observations about the mechanics of ChatGPT's output. First, every step in this guide is numbered 1. Some things here are probably copy-paste errors on my part (I'm reformatting the output to look better in plaintext), but that isn't, this output has four step ones. Another is the code blocks. I'm just using plaintext but ChatGPT uses nice embedded code blocks with syntax highlighting. Syntax highlighting means that it needs to know (or guess) the language it's highlighting and the codeblocks show what language they're set to. It is... seemingly random? The above shell command is 'csharp', but 'sql' seems to be a more common guess for shell commands than even 'bash'. This just means that most of the time the syntax highlighting doesn't really make sense.

I'm not sure where either of these are going wrong, but I get the feeling (without doing any actual research) that ChatGPT is outputting markdown. Markdown numbered lists can be fiddly and there are a few ways to end up with all step 1 like this. When it comes to the code blocks, I'm not sure if chatGPT is specifying the language or if the code block service is guessing it based on contents. I tend to guess the former because otherwise the guesses are very weird. I can't see any reason it would think one of these commands is csharp and the other is sql, they both have the same set of characters in them.

And yeah, this instruction makes perfect sense. You need to add some routes to the new route table.

1. Configure the process to use the new table:

   sudo ip rule add from table mytable

This command will add a new rule to use the mytable routing table for traffic originating from the specified IP address.

:thinking_emoji:

There's some clear semantic confusion here. I think ChatGPT is just going too readily to the many guides it's read that create a route table and then apply that route table based on ip rules. This is a very common thing to do, I would wager the most common way to use custom route tables, but it doesn't help in our case. The intro of this step is also just wrong (this has nothing to do with the process), but the sentence after correctly explains what it does.

1. Finally, start the process with the setsockopt() system call to bind it to the new table:

   int table_id = 200; setsockopt(socket_fd, SOL_SOCKET, SO_BINDTODEVICE, "eth0", strlen("eth0")+1); setsockopt(socket_fd, SOL_SOCKET, SO_MARK, &table_id, sizeof(table_id));

This code will bind the process to the eth0 interface and mark its packets with the mytable routing table ID.

This is, well, it's not exactly wrong but it's also not right.

If I fiddled with the prompt I'm sure I could get it to output something that makes more sense, but I also feel like this is an unreasonable way to interpret the prompt I did provide. To be clear, I expected that it would use cgroups and iptables to apply the new custom route table to some existing application. This seems like a way more common situation than to... write or modify an existing application in C. On the upside, the code block correctly has the language as 'c'!

All that said, if we start from the assumption that we are writing a new application this still doesn't work exactly. The option we see here, SO_MARK, applies a 'mark' in the sense of netfilter to traffic on the socket. Marks are a super useful feature that allows us to set an arbitrary tag on each packet that we can then refer to in our other network policy. A very common use for marks is to match the traffic up with an iptables rule that sets the routing table for that traffic... but ChatGPT didn't tell us to set that rule, it had us set the rule based on source port instead. I think maybe ChatGPT was trying to cover this in step 1 number 3 but it didn't quite choose the right rule.

Repeat steps 3 and 4 for each process that needs to use the new routing table.

There are no steps 3 and 4! There is only step 1!

Overall I'd give this a 3/10. Honestly I think ChatGPT did better than I expected when considering that this is not a very common use-case, but ChatGPT's output is worse than what you get if you google the same question.

Takeaways

This isn't intended to be a super carefully thought out research piece, but like others I've seen reporting that suggests that ChatGPT is pretty good at generating source code but has a tendency to leave subtle errors. I wondered if the same thing might be true in the old-school system administration space, but the feeling I walked away with is a little different: ChatGPT does make a lot of errors, but they aren't particularly subtle. I suspect this has to do with the different nature of the training material for these kind of questions, which is more in the form of narrative guides and documentation that keep steps simple but leave a lot of assumptions.

I pretty much covered every question I thought to ask in a short time spent playing around. There was one question that I was just unable to get ChatGPT to generate a good answer for... an AWS network architecture question involving load balancing for both HTTP(S) and arbitrary TCP services on the same elastic IP when you want layer 7 behavior. I could mark this as a 0/10 for ChatGPT but it might be a case of more "prompt engineering" being required... it kept producing output that was reasonable but failed to address one or more of my requirements, so I kept making the requirements more explicit, and then it would just miss a different set of requirements. It may very well be possible to get ChatGPT to produce a correct solution but it was definitely getting to be less useful than a search engine, even with how difficult the AWS documentation can be to use for architecture questions.

On the whole, I felt that ChatGPT was performing more poorly than Google for similar queries. It's possible to get ChatGPT to refine its output by adding more detail (at least if you don't get too deep into AWS networking capabilities), which is a big plus over a conventional search engine, but honestly it still didn't feel to me like this was a savings of effort over reading a few different articles and synthesizing.

One of the reasons this was on my mind is because I'm working with a client right now who has an interesting habit of copying and pasting all the error messages they get into ChatGPT, while still screen sharing. From this sort of eavesdropping on AI I have not been very impressed with its output, which has often been high-level to the point of uselessness. It sort of has the feel of AWS documentation, actually... he would put a very specific error from, let's say, Terraform into ChatGPT and it would answer with a few paragraphs about how Terraform works at a high level. I'm sure this can be improved with a more specific training corpus but I'm not sure what that corpus would be, exactly, which continues to stymie my plans to just forward all the emails I get from clients to ChatGPT and give them the response.

I mean, I think it would keep them going back and forth for a while, but they might feel like it's a good value for money.