COMPUTERS ARE BAD is a newsletter semi-regularly issued directly to your doorstep to enlighten you as to the ways that computers are bad and the many reasons why. While I am not one to stay on topic, the gist of the newsletter is computer history, computer security, and "constructive" technology criticism.
I have an MS in information security, more certifications than any human should, and ready access to a keyboard. These are all properties which make me ostensibly qualified to comment on issues of computer technology. When I am not complaining on the internet, I work in professional services for a DevOps software vendor. I have a background in security operations and DevSecOps, but also in things that are actually useful like photocopier repair.
You can read this here, on the information superhighway, but to keep your neighborhood paperboy careening down that superhighway on a bicycle please subscribe. This also contributes enormously to my personal self esteem. There is, however, also an RSS feed for those who really want it. Fax delivery available by request.
--------------------------------------------------------------------------------
Let's take a break from our boring topic of regional history to focus instead
on an even more boring topic: implementation details of telephone lines.
The conventional "copper pair" analog telephone line is fading away. The FCC
has begun to authorize abandonment of copper outside plant in major markets,
and telcos are applying to perform such abandonment in more and more areas. The
replacement is IP, part of the overall trend of "over the top" delivery,
meaning that all communications utilities can be delivered by the use of IP as
a common denominator.
This type of service is usually branded as "digital voice." Historically this
seems to have come about to evade VoIP's bad reputation; in the early days of
Vonage and the charmingly sketchy, back-of-magazine-ad Magic Jack, VoIP
products often delivered subpar service. Today, I think "digital voice" has
mostly just become part of price differentiation for carrier-offered VoIP,
since independent VoIP services tend to cost considerably less. Still, there is
some logic to differentiating digital voice and VoIP: because digital voice
service is offered by the operator of the underlying IP network, it benefits
from QoS measures that general internet traffic doesn't. On consumer internet
connections, especially slower ones, digital voice is still likely to be more
reliable than VoIP due to QoS policy.
Ultimately the move to digital voice is probably a good thing, as the
abandonment of copper plant will kill off DSL in urban markets and make way
for faster offerings---from telcos, usually PON. I'll take the opportunity
to eulogize the conventional copper pair, though, by going in to a bit of
detail about how it actually worked.
To start, a large disclaimer: the details of the telephone network have varied
over time as technology and the industry evolved. Details often varied from
manufacturer to manufacturer, and because Western Electric had a practical
monopoly on the manufacturing of telephone instruments for many decades, it's
pretty much the case that the "standards" for telephone lines in the US were
"whatever Western Electric did," which varied over time. There were some
independent organizations that promulgated telephone standards (such as the
railroads which had their own extensive telephone plants), but they were almost
always completely deferential to the Bell System. Independent telephone
companies initially had to use different conventions than Bell because much
of the Bell telephone system was under patent; after the expiration of these
patents they mostly shifted to doing whatever Western Electric did to benefit
from the ready availability of compatible equipment.
After divestiture, Western Electric's de facto standards-making power was vested
to Bellcore, later Telcordia, today iconectiv, which after the end of AT&T
monopoly was owned by defense contractor SAIC and is owned today by AT&T's
erstwhile competitor Ericsson. iconectiv continues to promulgate some of the
standards used in the telephone industry today, through contracts awarded by
the FCC.
This is all to explain that the telephone system is actually surprisingly
poorly standardized in the United States. You might expect some hefty ISO
specification for analog telephone lines, but there isn't really one outside of
equipment specifications published by manufacturers. Many international markets
have much more detailed engineering specifications from independent bodies, but
they're usually based directly on Western Electric's practices. To make things
more confusing, it's not unusual for international telephone standards to
either be based on older US practices that are now rare in the US, or to have
standardized on "in practice" properties of the US system instead of nominal
values, or to have mixed conventions from Western Electric with conventions
from European telephone manufacturers like Ericsson. All of these standards end
up being mostly the same, but with a dizzying number of slight differences.
Today, the FCC imposes requirements on telephone lines as part of its
regulatory oversight of telcos. The FCC's requirements are basically to "keep
doing whatever Western Electric did," and are often surprisingly loose. Phones
are really very robust, and the basic design of the system is over 100 years
old. Local loops are routinely in poor condition which throws things out of
spec anyway, and then subscribers use all kinds of weird phones that are not
always that well designed (the history of regulation of telephone instruments
could fill its own post). It's actually fairly intentional that the electrical
specifications in the system are all soft targets.
For the purpose of this article I am mostly going to describe the state of a
fairly modern local loop, such as one connected to a 5ESS or DMS-100 digital
switch. But I will definitely not describe either of these switches totally
accurately. I'm trying to give the right general idea without getting too
bogged down in the details like the last few paragraphs. I'll take the topic
of electrical specifications (potential and current on telephone lines) as a
chance to give some examples of the variation you see in practice.
First, let's talk about the very general architecture of an analog local loop.
Somewhere in town there is a telephone exchange, and somewhere in your house
there is a telephone. The local loop is fundamentally two long copper wires
that go directly from your phone to the exchange. This distance varies greatly.
It's advantageous to keep it under a few miles (mostly for DSL), but in rural
areas especially it can be far longer.
There's a little more detail to what goes on at the two ends of the line. In
your house, you have one or more telephones that you use to make and receive
calls. In the parlance of the industry, these are often referred to as
"instruments" or "subscriber terminals" depending on the era and organization.
Historically, instruments were considered part of the telephone system proper
and were property of your telco. Today, you are allowed to purchase and use
your own telephones. This comes with some downsides. Along with the phone being
your property (and thus your problem), the telephone wiring inside of your home
is your property (/problem).
The telephone wiring in your house runs from jack to jack. In the United
States, all of the telephone jacks in a home are connected in parallel. This is
one of the differences you will find if you look in other countries: because of
exact details of the electrical design of the exchange and the phones, and
where different components are placed, some countries such as the UK require
slightly more complex household wiring than just putting all jacks in parallel.
But in the US, that's all we do.
If you crack open a wall and look at your household telephone wiring, you will
almost certainly find a surprising number of wires. Not two, but four. This, of
course, corresponds with the four pins on a modular telephone jack. Your
telephone only uses two wires (one pair), but dating back to the '60s it has
been a widespread convention to wire homes for two separate telephone lines.
It doesn't cost much more, but it gives the telco the opportunity to upsell you
to a second line. This convention is reflected not only in the wiring but, as I
mentioned, the connector.
The connector used for modern telephones is often called RJ-11, although that
term is not exactly correct in a pedantic way that rarely matters. It's a
little more correct, when speaking of the connector itself, to call it a 6P4C
modular connector. 6P4C means six positions, four contacts---it could have six
pins, but only four of the pins are actually populated. Two are for one phone
line, two are for the other. If you actually have two phone lines fitted to
your house, you will find that the single-line phones common in homes always
use the same pair, so you'll either need an adapter cable or some jacks wired
the other way around to use two lines. If you've ever lived in a house with two
phone jacks right next to each other, it's likely that one of them is wired with
the pairs reversed (or only has one pair at all) so that a standard single line
phone can be used on the second line.
The phone wiring in your house joins your phones in parallel with a device
formally called a Network Interface Device (NID), but often referred to as the
demarc or demarcation point. This is because the NID is not just a technical
object but an administrative concept: it is the border between your property
and the telco's property. Inside of the NID there is usually a short phone
cable (connected to your house wiring) plugged directly into a jack (connected
to the telco's wiring). If your phone ever malfunctions, the telco will likely
ask you to take it directly to the NID, unplug your household wiring, and plug
your phone straight into the jack. If the problem goes away, it is somewhere in
your household wiring, and therefore not the telephone company's problem. This
is their preferred outcome, and you will be told to use your non-functioning
phone to call an electrician.
The NID may be located in different places depending on the details of your
house, when it was built, and when telephone service was installed. Most
commonly it is somewhere outside of the house mounted on an exterior wall
or skirting, often somewhere near the electrical service entry. There are
plenty of exceptions, and especially in older houses the NID may be in the
basement or crawl space. In some cases, mostly mobile and manufactured homes,
the NID may actually be mounted to the telephone pole at the street or your
property line, and the overhead or underground connection to your house is
also your problem.
From the NID, the telephone line makes way to the exchange. In most cases today
this will be as part of a telephone cable. A telephone cable is an assembly of
many telephone pairs bundled into one sleeve, and they're either run along
utility poles (lower than the electrical lines for isolation) or underground.
Either way, your telephone line will be connected to the cable inside of a
splice closure. For overhead wiring, splice closure are usually black plastic
cylinders hung alongside the cable. For underground wiring, they're usually
gray-green pedestals sticking out of the ground. Either one provides space for
the many pairs in a cable to be spliced together, some to another cable but
some to a drop line that runs to your house.
Most of the time, the cable does not run directly to the exchange. This is not
exactly modern practice, but a common convention is to have two levels of
"feeder" cables. The F1 cable is a very large cable that runs from the
telephone exchange to a neighborhood. There, the F1 cable is spliced onto
multiple smaller F2 cables that run along neighborhood streets.
The splice between F1 and F2 cables, and in general any splice between multiple
cables, is usually done in a larger splice cabinet. Historically splice
cabinets were sometimes mounted up utility poles, but this made them more
difficult to safely work on so this arrangement is pretty much gone to history.
Instead, modern splice cabinets are larger ground-level pedestals, usually a
good 4-8 feet wide with large double doors.
There are several advantages to these splice points. First, they are obviously
necessary for the original installation of the telephone infrastructure.
Second, splice closure and cabinets are intentionally made for easy
modification. This gives the telco a lot of flexibility in fixing problems.
The world of local telephone loops is a dirty one, full of dirt and rain.
Despite precautions, water has a way of working its way into telephone cables
and can cause corrosion which makes pairs unreliable. When you complain, and
the NID test shows the problem is on the telco side, they will likely just
resplice your home telephone service onto a different pair back to the
exchange. This swap from one pair to the other avoids the problem, which is a
whole lot easier than fixing it. Actually fixing problems inside of telephone
cables is a whole lot of work, and with subscriber numbers dwindling in cities
there are usually lots of unused pairs so it's easy to swap them out.
In some areas, your local loop may not actually go directly to an exchange. It
might go to something like a remote line concentrator, or a serving area
cabinet, or a loop extender. These are all variations on the idea of putting
some of the exchange-side equipment in a big curb cabinet, closer to your
house. These arrangements are most common in suburban areas where local loop
lengths are long and subscriber density is fairly high. I'll mostly ignore
this, but know that some of the parts of the telephone switch may actually be
in a curb cabinet in your case. These curb cabinets usually function as remote
components of the switch and connect back by ISDN or fiber.
Once your telephone loop makes it from your phone, through your house wiring,
down a drop cable, through an F2 cable, and then through an F1 cable, it
arrives at the telephone exchange. There it often passes through an area called
the cable vault, usually an underground space in or adjacent to the basement
where cables enter the building, seeping water is drained, and pairs come out
of the armored cable jacket. Everything before this point has been "outside
plant," and is the realm of outside plant engineers. Now, we have entered the
sanctum of the inside plant, and a different department of the company.
From the basement, pairs go to the main frame, basically a really big splice
cabinet inside of the telephone exchange. The main frame allows exchange
technicians to connect pairs to the switch as they please. If an outside plant
technician has fixed your telephone problem by resplicing your house to a
different pair, they will submit a ticket (originally a paper slip) to have the
exchange technicians perform the same remapping on the main frame. Originally,
if you stopped paying your bill, a ticket would be generated for an exchange
technician to un-splice your phone line at the main frame. Today, both of these
are often done digitally instead by leaving pairs connected to the switch's
line cards and reconfiguring the line card mapping.
Which takes us to your local loop's next stop: the actual switch. The many
local loops that a class-5 or exchange switch serves terminate (in the case of
modern electronic switch) at what are called "line cards." The line card is
responsible for managing all of the electrical aspects of a small set of local
loops connected to it. Depending on the type of switch, the line card may
perform ADC and DAC to convert your analog local loop to digital signaling for
further handling by digital means. Or, it may connect your local loop to a
device called a hybrid transformer that separates your call into two pairs (one
for audio each direction) for further handling in analog form.
And that is your local loop. It's called a loop because the two wires,
connected at one end by the switch and at the other end by your phone, allow
current to flow all the way around. "Current loop" is the term in electrical
engineering for this type of arrangement, and it's such a common means of
conveying information by electrical signals that it's often only implied.
For there to be current through the loop, though, someone has to put some
potential onto the line. Historically, phones were expected to provide power,
but this practice had become obsolete by the end of WWII. In modern phone
systems the loop power is provided by the switch. In the case of phones
providing power, the phone contained a battery which was occasionally replaced
by the telco. In the case of switch-provided power, AC-DC rectification was an
imperfect art and there was a need for a backup capability in any case, and so
the telephone switch would get its loop power from a very large battery.
Because of this history, the normal potential on your phone line is known as
battery power. People will sometimes shorten this to say that the switch
"provides battery," especially in situations like test equipment or military
field phones where it isn't always obvious which end battery power will come
from. As another bit of telephone terminology, a telephone line with battery
applied is sometimes called "wet," while one without battery applied is called
"dry."
Battery power in the United States nominally comes from a series of lead-acid
batteries producing a nominal 48v. In practice, there is some considerable
variation. In older phone switches, a float charger was continually connected
to the batteries and so held the battery voltage higher (around 52-54v)
whenever the exchange had utility power available. Likely because of this, some
countries such as Japan actually standardized 50v or 52v as the nominal
off-hook potential. In newer equipment, battery voltage often comes not from
batteries at all but from a regulated switch-mode power supply (that runs
either off of external AC power or a battery bank of a potentially different
voltage). It may therefore be exactly 48v, but some of these power supplies are
actually regulated to 50v to match the typical behavior of older equipment. It
really just depends on the device, and most telephones will function
acceptably with well below 48v off-hook.
For historic reasons, telephone switches are mostly frame-positive. This means
that battery voltage is often described as -48v. The difference doesn't really
matter that much on the telephone end, but can be confusing if you are looking
at documents that use different conventions. Owing to the 1/4" jacks originally
used for telephone exchanges, the two wires in a telephone pair are called
"tip" and "ring." Compared to ground, "tip" should be about 0v while "ring"
should be about -48v. This "ring" is not to be confused with ringing power, to
be discussed later. It's just the naming convention for the wires. Some phones,
especially rotary types, will function fine with the polarity backwards. Most
newer phones won't.
This talk of voltages, so far, has all been assuming that the phone is on hook.
When a phone is on hook, the "hookswitch" disconnects the two sides of the
local loop from each other, leaving an open circuit. The voltage across the
phone thus settles at whatever voltage is applied by the switch. There are two
conditions under which this voltage changes significantly.
First, when the phone is off-hook and you receive a call, the exchange needs
some way to ring your phone. An age-old method of ringing phones is a magnetic
solenoid (arranged so that its core will strike bells when it moves) with the
line passing through it. The coil provides a very high resistance and so passes
little current under normal off-hook conditions. When the switch wants your
phone to ring, it applies a much higher voltage, and alternates it. In fact,
this AC voltage is superimposed on the normal DC battery voltage, and is known
as "ringing power" or "ringing voltage." Ringing voltage is AC at 20Hz by wide
agreement (this works well for the simple mechanical solenoid ringers), but the
potential has varied over time and by manufacturer. It also just tended to vary
based on equipment condition, since it was originally produced by
electromechanical means such as a motor-generator. 90v is fairly typical, but
up to 110v AC is acceptable.
The fact that a telephone line can carry 110v AC is a surprise to many, and
should discourage licking loose phone wiring, but from a fire safety
perspective it's reassuring to know that the current of the ringing power is
rather limited. The details of how it's limited, and to what level, depend on
the telephone switch in use, but the current-limiting at the switch allows
telephone lines to be handled as class 2 or power-limited for purposes of the
electrical code.
The astute reader might notice an interesting problem here. Multiple telephones
may be connected in parallel, but ringing voltage provides only limited power.
If you have enough phones, will there not be enough ringing power?
Yes.
The implementation of this problem is sort of an odd thing. Federal standards
created the concept of a "ringer equivalence number" or REN. When the REN was
developed, the standard telephone instrument from Western Electric was the
model 500, and other Western Electric phones for PSTN use were made to match
the 500. So, a REN is defined as the impedance and resistance imposed by a
single Western Electric model 500. Local loops are expected to provide enough
ringing power for up to 5 REN. In practice, this is rarely a concern today.
Few modern phones use the power-intensive electromechanical ringer of the 500,
and if you look at the literature that came with a digital phone like a DECT
cordless set you will likely find that it is specified as 0.1 REN. You can have
a lot of modern phones before you run into problems.
Ringing current only applies when there is an incoming call and the phone is on
hook. When you take the phone off hook, the hookswitch connects the two sides
of your local loop together via the phone's "voice circuit." This prompts the
switch to stop applying ringing current. Once the voice circuit is connected,
the voltage across your phone drops considerably. Power is now flowing and so
Ohm's law has taken control. Local loops are long and made of small-gauge wire;
just the resistance of the telephone line itself is often over a thousand ohms.
Besides the wiring itself, there are two other notable components the loop
current must pass through. First, there is some resistance imposed by the
switch. The details of this resistance depend on the exact switch in use but
the most common situation is that the phone line passes through a coil in a
"line relay" assembly as long as your phone is off hook. This relay informs the
switch logic of the state of your telephone line. The resistance imposed by the
line relay varies by switch, and on many switches it's adjustable, at least in
two or three steps. This allows an exchange technician to make an adjustment if
your loop is very short or very long, to keep the loop current more in a normal
range. 600 ohms is fairly typical for line relays.
The third important component is the voice circuit of your phone itself, which
also varies by phone but is also typically around 200 ohms. Because the phone
is in fact powered by the loop current, there is a certain requirement for
enough power for the phone to operate. AT&T specified that the maximum local
loop resistance should be 2400 ohm. At 2400 ohm and 48v battery power, the loop
current will be 20mA. 20mA was about the lower bound at which the voice circuit
in typical Western Electric models (such as the 500) performed acceptably.
Modern electronic phones are often workable at lower currents (and thus higher
loop resistance), but audio quality will become worse.
Too high of loop current can also be a problem. There isn't a widely accepted
upper limit that I know of, but it was known by the '60s at least that high
loop currents due to low resistance loops (usually found when the subscriber
lived close to the telephone exchange) caused higher temperatures in the line
relay which could result in early failure of the line card. This is one of the
reasons line cards often provide an adjustment, so that short loops can have a
higher line relay resistance to reduce the current. Modern line cards often
monitor the current on a loop and raise a trouble ticket if it is abnormally
low or high. Either one tends to indicate that there is a problem somewhere in
the line. Modern line cards also provide better over current protection and will
usually automatically disconnect battery power (and raise a trouble ticket) if
the current goes significantly above normal. This is why you can't get greedy
if you are trying to charge your laptop on Ma Bell's dollar. Older equipment
may have just used a fuse, so at least newer systems have the forgiveness of
auto-reset a few seconds after laptop-charging mistakes.
Under off-hook conditions, with loop current flowing, the voltage across your
telephone will be much lower. 3-5v is pretty typical, but the value will vary
with voice modulation as you use the phone and isn't very important.
It's important to emphasize that nothing in this system is very well regulated.
Well, with modern equipment the battery voltage is often precisely regulated,
but that's more a natural consequence of switch-mode power supplies than any
aspect of the design of the telephone system. Battery voltage is 48v in the
same sense that automotive electronics are 12 or 24v, in practice it's often
higher. Loop current is limited but fairly loosely. The acceptable range is
pretty wide in either case.
To put some more solid numbers on it, here are some specs taken from the
manual for a PABX. They seem pretty typical, except the upper limit for
off-hook voltage which seems unusually low. Many sources say 4-9v. Note
the two different ringing voltage specifications, these are to accommodate
two different national standards.
On-hook voltage: 40-50 V DC
Off-hook voltage: 4 to 6 V DC
Ringing: 90-100 V AC 20Hz or 60-90 V AC 25Hz.
Loop current: 25-40 mA (up to 80 mA in "unusual circumstances")
And for flavor, numbers I took from the spec sheet of a different business
telephony product:
On-hook voltage: -54v to 40v DC
Off-hook voltage: -20v to -5v DC
Current: 23-35 mA
Note the different voltage convention, and that all the numbers are, well,
similar, but different. That's just how it is!
Where we left off, Los Alamos had become a county, but the town itself
continued to be directly administered by the Atomic Energy Commission (AEC).
The Atomic Energy Communities Act (AECA) mandated the AEC to dispose of the
towns it owned by transferring the property to private owners or government
agencies. This included not just the individual houses (which had all been
rented by AEC) but utilities, parks, and other municipal property.
In 1963, shortly after the addition of Los Alamos to the AECA, the AEC started
the process of transferring public resources. The schools were a relatively
simple case, as a county school board had existed since the creation of the
county in 1949, but it still took until 1966 for the AEC to give the school
board title to the real estate the schools occupied. A case that seemed
more complex, but also moved faster, was the electrical supply.
It's likely that the AEC had their own motivations. Electricity was fairly
expensive and limited in Los Alamos, and the lab had an almost perpetual need
for more power. Transferring the electrical utility to an outside agency might
have benefited the AEC by moving system expansions into state and municipal
financing and out of the federal budget. In any case the electrical system was
actually reasonably cut-and-dry since New Mexico had a well established system
of electrical franchise agreements and there were plenty of utility operators
with experience in power distribution. As mandated by the AECA, the AEC put
the electrical system out to bid.
One of the bidders was entirely unsurprising: The Public Service Company of New
Mexico, today known as PNM [1]. The politics of Los Alamos utilities turn out to
be one of the most complex issues surrounding the transition from AEC control,
and PNM's attempt to purchase the electrical system is a great example of the
debate.
First, it should be understood that electrical service in Los Alamos was
expensive to offer due to the remote location and difficult terrain around the
town, but Los Alamos residents had been largely insulated from this issue. It
was the AEC that really footed the bill for electrical power, and residents
were charged a rate that was designed to be similar to other electrical rates
in the area---not to recoup the AEC's cost. There was a clear risk that
electrical rates in Los Alamos would increase significantly after transfer away
from the AEC, and much of the politics of mid-'60s Los Alamos was animated by
this concern.
The debate over ownership of utilities played out in a series of delightfully
mid-century newspaper ads taken out not just by PNM but also by organizations
with names like "People's Committee for Private and Professional Utilities" and
"Los Alamos County Citizens for Utilities." The debate was something like this:
PNM argued that their large size and experience in the electrical industry
would allow them to sustain low rates in Los Alamos. Some Los Alamos residents
favored this argument, taking the view that PNM was an established, experienced
organization that would operate the utility based on sound analysis and
long-term planning. This kind of thinking seems to have been particularly
popular in Los Alamos, where nearly all the residents worked for another huge
bureaucracy that considered technical expertise its defining identity. Further,
it was thought that if the county owned the utilities, the financial shortfall
would likely be made up by increasing county property taxes---effectively
increasing electrical cost, even if the rates stayed the same.
On the other hand, advocates of county ownership felt that a public operator,
without the motive of returning profit to its shareholders, could ultimately
keep costs lower. In their support, consulting engineers hired by both the AEC
(to evaluate bids) and the county (to support their bid) concluded that it
would be possible to operate the system profitably without exorbitant rates.
County ownership was often seen as the more conservative option anyway, since
it would allow for the privatization of the utility in the future by franchise
agreement. To further assuage concerns about the county's competence to run the
utilities, the county board advertised in the newspaper that it intended to
directly hire staff from the Zia Company, the private contractor the AEC had
used for maintenance and operations of the utilities.
Finally, the county had a rather unusual motivation to assume control of
utilities. Because the laboratory was essentially the only industry in Los
Alamos and was exempt from property tax due to federal ownership, the county
had only a very limited ability to raise revenue. The county felt that the
profit it could make from utility operations would become a key revenue source,
particularly since the lab itself would be a customer. In the words of state
senator Sterling Black, county-owned utilities would become "a club we can wave
over the AEC." Considering the AEC's tendency to completely steamroll county
decisions when they conflicted with federal policy, this kind of political
power was extremely important.
The idea of AEC decision input by referendum was not a new one. Just a bit
earlier in 1963, Los Alamos residents had gone to the polls to choose who would
receive the Los Alamos Medical Center. Their choice was between the Lutheran
Hospitals and Homes Society (an out-of-state concern but known to operate the
hospital in Clayton at the time), the Los Alamos Medical Center Inc. (an
organization formed by Los Alamos residents expressly to operate the hospital
locally), or "neither." This issue had been substantially less contentious as,
it turns out, the Lutheran organization had submitted their proposal to the AEC
before it knew about the local organization. Once it came to their knowledge
they apparently deferred to the local initiative, telling the Santa Fe New
Mexican that they supported local operation of the hospital and would pursue
ownership only if the county invited them to do so.
In the case of the hospital, though, we can already see the formation of the
movement against local organizations and towards larger, more experienced
organizations that were likely to offer a more predictable and stable---even if
less locally driven---operation. The Lutheran organization won the referendum,
and controlled the hospital until 2002, when another attempt at establishing a
local hospital operator failed.
The story of the Los Alamos Medical Center illustrates some other challenges of
the military-to-municipal transition as well. When the AEC built the Los Alamos
Medical Center, it included housing for staff as part of the complex. As a
result, the hospital had two, 24-unit apartment buildings directly attached to
it. This seems to have established a tradition in which another condo building,
owned by a group of physicians, was attached to the hospital in 1982. The odd
mix of hospital and housing, and the resulting complex lease agreements,
complicate ownership transfers of the hospital to this day.
The hospital referendum provided a clear example for the utility referendum,
but the utility question attracted significantly more debate. This is perhaps
ironic considering that the utility referendum was also less significant in the
actual selection process. Because of the precise wording of the AECA, the AEC
had determined that it would follow the outcome of the hospital referendum
exactly. In the case of utilities, though, the AECA mandated that the AEC
consider a set of technical and economic factors as well. The referendum would
not be absolute, it would only be one of the factors in the AEC's decision.
A further complication in the decision was the exact nature of the county's
legal authority. New Mexico counties are not normally authorized to operate
utilities (although several do today under various provisions, such as
Bernalillo County's partnership with the City of Albuquerque to operate the
Albuquerque-Bernalillo County Water Utility Authority). In the legislative
session at the beginning of 1963, the New Mexico legislature passed a
constitutional amendment intended to facilitate the Los Alamos transfer,
without further fanning local debates by requiring the City of Los Alamos to
incorporate [2].
Although the amendment didn't take effect until 1964 following its approval in
a statewide referendum, it was very much on the minds of the county government
as they considered utilities. We will revisit this amendment in more detail
shortly, but it had an important impact on the utility debate. In the 1960s,
counties in New Mexico granted utility franchises under state regulations. This
meant that such utilities had their rates reviewed by the state Public Service
Commission. Because of the unusual status of Los Alamos county as the only
"category 7" county (by the '60s now called an "H-class" county), the situation
was somewhat more complex, and Los Alamos County probably had the opportunity
to function as a municipal government instead, which allowed for complete home
rule of utilities. In other words, it seemed that Los Alamos utilities could
be regulated by the Public Service Commission, but they didn't have to be,
depending on exactly how the county wrote up the paperwork. The upcoming
constitutional amendment in fact solidified this situation.
One argument for county ownership was then that it would give the public better
control of the utility, since it would not necessarily be subject to state rate
regulation. One argument the other way was that a private utility operator
could still be state regulated, if so desired. The regulation issue really only
served to confuse the debate, as did a number of other events of 1963,
including minor scandals of the county withholding documents and accusations of
harassment and illegal campaigning by various individuals and political
organizations in Los Alamos. Notably, the League of Women Voters, long the most
important and highly regarded non-partisan political organization in New
Mexico, was widely accused of compromising its impartial, non-partisan values
by openly advocating for county control. In some ways this was actually not a
very partisan position, as in a rather unusual political turn Los Alamos's
Democratic Committee and Republican Committee joined together in endorsement of
county ownership. The Santa Fe New Mexican quipped that the situation had
become David vs. Goliath, except that somehow David was the mighty PNM (and
Southern Union Gas, the company that had previously bought out PNM's gas
operations) and Goliath was tiny Los Alamos.
A full accounting of the politics around the utility acquisition could quite
possibly fill a book, as very nearly every aspect of Los Alamos life was
somehow a factor in the debate. It only adds color to the situation that the
federal officials involved, being AEC personnel, included well-known scientists
like Glenn Seaborg. Just the newspaper editorials written in support of either
position would make for a hefty binder. I will try not to get stuck on this
forever and jump to the conclusion: David did not fell Goliath, and Los Alamos
voters favored county control of utilities by 71%. Although the full transfer
process would take several years to complete, the decision was made and the
electrical and gas infrastructure were transferred to Los Alamos County, where
they remain today.
There were yet more utilities to consider. Telephone service in Los Alamos had
been provided by the AEC, with all service ultimately connected to the exchange
at the laboratory. This, too, needed to be transferred, and in 1964 the AEC put
out a request for bids. There was little precedent for government operation of
telephone systems, and it was viewed as legally difficult, so it was clear that
some private operator would step in. Two made an offer: Universal Telephone
and Mountain States Telephone and Telegraph.
Universal Telephone was born of Milwaukee, Wisconsin. Wisconsin was known in
the middle part of the 20th century for its particularly skeptical stance on
telephone monopolization under AT&T. Wisconsin was among the first states to
introduce pro-competition telephone regulation, namely the requirement that
competitive telephone carriers interconnect so that their customers could call
each other. Universal Telephone was thus among the most ambitious of AT&T's
competitors, and no doubt saw Los Alamos as an opportunity to capture a brand
new telephone market and demonstrate the capabilities of those outside the
Bell system.
Mountain States, though, was an AT&T company, and brought with it the full
force of AT&T's vast empire. The complex landscape of telephone regulation
immediately became the key point of this fight, with AT&T arguing that the
state telephone regulation agreements gave Mountain States the exclusive legal
authority to operate telephone service in Los Alamos county. Far from a
referendum debated in the newspapers, the future of Los Alamos telephony was
mostly debated by lawyers behind closed doors. The AEC's lawyers sided with
Mountain States, disqualifying Universal Telephone as a bidder. By the end of
1964 the writing was on the wall Mountain States would be the new telephone
company (besides, AT&T rarely lost a fight), although legal arguments would
continue into the next year. In early 1966, the deal was complete and Mountain
States began to staff its Los Alamos operation. They had a lot of work to do.
Even into the '60s the telephone exchange in Los Alamos supported only 5-digit
dialing. While 5-digit dialing is not unusual in some large private phone
systems (and indeed is still in use by LANL today), the use of 5-digit dialing
across a town was exceptional and meant that Los Alamos telephone users could
not directly dial outside of the town. In 1968, Mountain States began
construction on a new telephone building with a distinctive microwave tower for
direct connectivity to the broader AT&T network. The new switch installed
there, a 1ESS, offered for the first time touch-tone dialing, direct-dial long
distance, and even three-way calling. At the same time, numerous new lines were
installed to take advantage of the higher capacity of the new exchange.
Telephone service became far easier to order in Los Alamos and on Barranca Mesa
(not yet always considered part of the town), although White Rock would have to
wait almost another decade for improved service availability.
Electronic telephone switching wasn't the only new idea in Los Alamos. The
entire "H-class" county designation of Los Alamos was extremely confusing, and
more and more the nature of the county was considered a mistake going back to
its start in 1949. State legislators had tried to make Los Alamos a special
kind of county that had the authority to do certain things normally withheld to
municipal governments, but the piecemeal way they had done this created a lot
of problems and required some sort of state legislation to address one problem
or another nearly every year from 1949 to 1969. This included two cases of
significantly reworking the county's enabling legislation, which is part of why
it went from "7th category" to "class H" even though H isn't the 7th letter.
Los Alamos was created as a whole new kind of county in 1949, it was made yet
another whole new kind of county in 1955, major changes were made in 1963, and
now it was still a mess. Something had to change, and the people of Los Alamos
along with the state legislature were increasingly leaning towards the idea of
incorporating like a city---but as a county.
In 1963, the state legislature passed Constitutional Amendment #4. In 1964,
it was approved by state voters and took effect. This amendment, now article
X section 5 of the New Mexico Constitution, created yet another brand new
kind of county: the incorporated county.
The applicability of X § 5 was intentionally limited to Los Alamos, although
the realities of legislation discouraged writing it that way. Instead, it was
made applicable to any county with an area under 140 miles and a population
greater than 10,000 (this latter rule mostly just to agree with other rules on
incorporating municipalities). You can no doubt guess the number of counties
under 140 square miles in New Mexico, a state known for its vast counties. Even
tiny Bernallilo County, sized for its dense urban population, measures over
1,000 square miles. Despite its ostensibly general criteria, X § 5 applies
uniquely to Los Alamos. It's even written such that any new counties that
small would be excluded, only counties that existed at the time of its passage
are eligible.
The amendment allowed Los Alamos county to incorporate, just like a
municipality:
An incorporated county may exercise all powers and shall be subject to all
limitations granted to municipalities by Article 9, Section 12 of the
constitution of New Mexico and all powers granted to municipalities by
statute.
In other words, it would resolve the problem of Los Alamos County's confusing
city-county nature by making it, well, a real city-county. This concept is not
unique nationally and some states, such as Virginia, rely on it extensively
with "primary cities." San Francisco is a well known example of a city-county
in the West. Los Alamos has much the same status today, but due to the history
of the city being closely tied to AEC administration, the city-county title is
rarely used. The Los Alamos government today identifies itself only as Los
Alamos County (slogan: "Where Discoveries Are Made"), and its nature as a
municipal government is relegated to the legal details.
Los Alamos's administrative situation is actually oddly obscure today. Wikipedia
refers to Los Alamos proper only as a census-designated place and county seat,
making no mention that the county it is seat of is actually a municipality as
well. If I could mount one odd political campaign in Los Alamos it would be to
restyle the government as The City and County of Los Alamos for aesthetic
reasons, but then the strict limitations in X § 5 make it unlikely that the
idea will catch on more broadly in New Mexico.
But what about the actual incorporation process? An incorporated municipality,
under state law, requires a charter. The constitutional amendment provided
that such a charter could be developed by a charter committee and then approved
by a majority of the voters. This detail of the incorporation process would
become a very tricky one.
The first major charter effort gained momentum in 1966, when a charter
committee completed a draft county charter and began the process of gaining
electoral support. In doing so, they brought the debate over public utilities
back to life.
Incorporation would normalize the county's authority to franchise, operate, and
regulate utilities, but changed the legal situation enough that the county
would need to rebuild its administrative bureaucracy for utility oversight. The
1963 concern that utilities would become a "political football" (in the words of
one county commissioner at least) were back in the spotlight, as the new charter
proposed a utility advisory committee but gave it little actual authority and
mostly deferred utility matters to the elected county commission.
Utility operations might seem like a small detail of municipal government but,
as the original 1963 referendum shows, it was a very big detail in Los Alamos.
The problem of the 1966 charter and its utility advisory committee lead to many
newspaper articles and opinion columns, and a group calling itself the
"Citizens for Representative Government" formed a major opposition to the '66
charter. While their name makes it sound like a broader issue, the
representative government they sought was mostly the ability to put utility
rates to referendum, denied by the draft charter. Another, less decisive
problem was the charter's general increase in the powers of the county manager.
Several county positions which were previously elected would be appointed by
the manager, mostly positions related to finance and tax administration.
In a cunning political move, the Citizens for Representative Government seem to
have made themselves the principal source of information on the draft charter.
They mailed a copy of the proposed charter to every Los Alamos resident,
including their phone number for questions. A nearly full-page newspaper ad
featured the charter represented as arrows shot towards the hapless Los Alamos
public, captioned "There's still time - duck now! Vote no February 8th."
Major political points made in the papers included limited recall powers,
the possible appointment of judges by the county council, and the lack of a
requirement for competitive bidding on county contracts... but still focused
on utility management and utility rates as the single largest issue.
Whether through good political organizing or the genuine public fear of
politicization of town utilities, the '66 charter failed its referendum vote.
The general consensus, reflected in the papers, is that the charter was mostly
acceptable but there was widespread opposition to a few specific sections.
Given the situation, there was not much to do except to start over again, and
that they did. At the end of 1967, a new charter committee was formed. As an
example, they worked from the new City of Albuquerque charter, one of several
which rotated Albuquerque through various forms of city government in the mid
century.
The second charter committee had been asked to work quickly, and they delivered
on a tight timeline. In early 1968 a draft had been published and was the
subject of much debate once again, and as the end of that year approached it
had reached the form of a final draft. In many ways, this second charter was
more conservative, leaving much of the county government to function the same
way it had before. Like the City of Albuquerque at that time, it was based on a
manager-commission model without an elected executive.
This time around, proponents of the charter leaned heavily on its amendment
mechanism as a selling point. Being such a new municipality, and with an
unusual administrative situation, it seemed clear that Los Alamos government
would require flexibility. In an effort to address the specific complaints
about the '66 charter, this new charter also formed an independent utility
board with members appointed by the county council on a staggered basis... and
direct authority over utilities, alongside a professional utilities director
hired by the commission.
This new charter gained public support much more easily, apparently benefiting
greatly from the lessons learned during the last attempt. In September it was
approved by the county commission, and in December a referendum was held for
public approval. This time, it passed---with lower turnout than the last effort,
suggesting some mix of greater support with, well, less opposition. Still, not
making enemies is a kind of success, and at the beginning of 1969 Los Alamos
took its identity as New Mexico's only "incorporated county."
It took a lot of attempts for New Mexico to make the transition, perhaps five
depending on how you count, but the townsite eventually did convert from AEC
property to a municipality. We've even talked about the transition of the
school district and electrical, gas, and telephone service. But what of water
service? That story is complex enough I could go on about it for a while,
and I probably will, in a part III.
[1] This "public service" naming is extremely common in older, central US
electrical utilities, but should not be taken as implying public ownership.
Nearly all of the "public service companies" are private corporations with no
history of public ownership.
[2] It may be important here to understand that New Mexico is one of the states
that achieves most major legislation through constitutional amendments. The
New Mexico Secretary of State publishes a convenient, pocket-sized copy of
the constitution, which is 218 pages long.
This weekend, I found myself staying in Los Alamos for a volunteer role in
which I judge children on the quality of their software. Clearly this is not
the kind of opportunity I would turn down, but I also always take an excuse to
drive back up the hill. I only lived in Los Alamos briefly, but it left a big
impression. It's a unique place in many ways, born of a rather unusual history.
The role that Los Alamos as a place, Project Y and the Los Alamos Scientific
Laboratory as an institution, and more broadly the Manhattan Engineering Works
played in World War II is widely documented. The Manhattan Project-era history
of Los Alamos is actually surprisingly uninteresting to me, because few
physical remnants of it exist in publicly accessible areas. There has been a
concerted effort to offer more frequent public tours of places like Omega
Canyon as part of the Manhattan Project National Historic Park, but the reality
of DoE security requirements and complete paralysis of DoE outreach efforts
during COVID mean that these have not made much forward progress.
Unfortunately, LANL remains littered with historic buildings---demarcated by a
brown sign bearing an "H" placed by the laboratory historian---that no one
other than employees will ever see. And employees don't tend to make much note,
because when you work in the institution it seems rather unremarkable. The
nuclear weapons program poses a fascinating historiographical puzzle that way,
and I sometimes lament that the laser-focus of most nuclear history on the
pre-1945 period leaves even many nuclear history enthusiasts oddly unaware of
the evolution of DoE facilities in even the 1950s. While unfortunate, this
blindness to the Cold War era is quite practical: once the arms race kicks in,
the proportion of historical record that remains classified shoots steeply
upwards.
This is all sort of a preamble to explain that I am not going to discuss the
history of Los Alamos as it was founded, there are plenty of good sources on
that. Instead, I'm going to get into some of the fine details of how Los Alamos
became the place that it is today. I will also ponder as to whether or not
Omega Bridge is, was, or could be rigged to explode, so stay tuned for that
bit of urban legend.
There are aspects of the post-WWII story that ought to be fairly easy to
research, though: the areas of nuclear weapons installations that are now quite
public: formerly secret towns that are, today, just towns. From a formal
perspective these sites are Oak Ridge, Tennessee; Richland, Washington; and Los
Alamos, New Mexico. Los Alamos is, in my opinion, the most interesting, and
it's certainly the most the one I'm staying in right now, and so I want to talk
a bit today about part of the nuclear story that you don't often hear: The
transition of a secret military installation to a mostly (but not entirely)
normal town. A mid-sized one, by New Mexico standards.
Los Alamos was originally created as a military installation, most of it
hastily constructed of temporary buildings and reused facilities of the ranch
school that had formerly occupied the site. Not just housing but all municipal
services were managed by the military, and of course during WWII the fact that
Los Alamos existed at all was a secret (although not an especially well-kept
one in Santa Fe). Shortly after the war it became clear that this posed a
problem. From our perspective, many years later, WWII might seem to crossfade
seamlessly into the Cold War, but that was not the case at the time. The
conclusion of WWII was seen sometimes as the conclusion of war entirely.
In the years shortly after the Japanese surrender, the War Department was
tasked with a disarmament of incredible scale. The War Surplus Administration
was tasked with most of the liquidation, and repurposed several military
installations (including Sandia Army Base, which it shared tenuously with what
would become Sandia National Laboratories) as storage yards to hold the
numerous assets (including not just airplanes but entire buildings) that it was
expected to auction off. The clearance of military assets declared excess after
WWII took decades to run its course and lead to the phenomena of the military
surplus store, which have now largely faded away as our modern military
industrial complex (and particularly its inability to produce replacements for
anything) has eliminated the concept of military surplus.
The Atomic Energy Commission, formed almost immediately after the closure of
the war to assume responsibility for the former Manhattan Project, was not at
all immune. Especially the late '40s and '50s, when the Missile Gap had not yet
become a leading political issue, the AEC was expected to tighten its belt.
This meant finding a way to economize the enormous industrial complex it had
created, unwinding all of the costly special measures that had been justified
as wartime expediencies. One of these was the whole matter of having installed
the Corps of Engineers as a functionally socialist administrative authority
over entire towns.
In 1993, Los Alamos established a sister city relationship with Kremlyov, now
called Sarov, and formerly a series of number-suffixed codenames typical of
Soviet "closed cities." This unusual kinship of American military company town
and Russian military company town is presumably rather strained by today's
geopolitical events, but the sibling relationship of the cities is a profound
one. Sarov is, to put it in simple terms, the Los Alamos of Russia. Los Alamos
is still dominated (politically, culturally, and often physically) by a large
government laboratory that employs most of its residents. Sarov is much the
same. Sarov features rows of dull, government-constructed housing projects to
which its residents were assigned. Los Alamos is much the same, except that
today, those government housing complexes are sold and leased at rather
exorbitant prices.
Here is a critical difference: while Russia has made some efforts towards
normalizing its closed cities, the United States has eliminated them. Instead
of a government installation, Los Alamos is today a democratically governed
town that just happens to have an intensive dependence on the Department of
Energy. It all started in 1949.
Most of what is now Los Alamos County was obtained by the Army by either
transfer from other federal agencies (mostly the Forest Service, as Los Alamos
is firmly within the Santa Fe National Forest) or purchase from the private
holders. Construction of the laboratory began in late 1942. At this point in
time, Los Alamos remained a steep, forested site on the border of two counties.
The Army operated by the chain of command was not especially concerned with
administrative boundaries. The entire area was simply considered to be a
federal reservation---completely under the control of the federal government,
which was not an unusual arrangement for military sites at the time.
From the beginning, Los Alamos enjoyed some very limited form of municipal
government. A "town council," formed in 1943, consisted of seven members that
served in an advisory role only. They represented the municipal needs of
residents to the Army, the actual government of the installation.
Post-war, the federal enclave began to cause confusion [1]. Were Los Alamosans
residents of New Mexico? During the Manhattan Project secrecy they had all
semi-officially been considered residents of Santa Fe, but as Los Alamos came
to fame they started to use their Los Alamos addresses on paperwork.
Consequently, Los Alamosans were denied the right to vote, on account of their
not residing within the State of New Mexico. This matter was thought resolved
when the State Legislature specifically authorized Los Alamos residents to vote
in state elections in 1947, but that decision didn't hold up to review by the
New Mexico Supreme Court---a review prompted in part by a state court having
refused to hear a Los Alamos divorce proceeding due to questions of
jurisdiction, highlighting yet more problems with the federal situation.
Adding insult to injury, the state Attorney General ruled that Los Alamosans
would have to pay non-resident rates for hunting and fishing licenses. The
legal environment was quite unsatisfactory and lead to an escalating campaign
sometimes referred to as "Civil Rights for Los Alamos."
In an effort to resolve the conflict, US Rep. Fernandez of New Mexico
introduced HR 54 to the 81st congress. The bill was actually quite simple: it
would "retrocede" the jurisdiction over Los Alamos to the state of New Mexico,
conditional upon acceptance of jurisdiction by the state. The matter was seen
as quite urgent and so HR 54 was rushed through to pass before the New Mexico
legislature ended its own session. Almost simultaneously, State Senators Roach
(hailing from Hot Springs, which would later be known as Truth or Consequences)
and De Vargas (Española, Los Alamos's closest neighbor) introduced SB 110. This
bill authorized the governor to accept jurisdiction over the area, and quickly
after the passage of both pieces of legislation, Governor Mabry wrote to President
Truman indicating the state's acceptance.
The normalization of Los Alamos proceeded at a pace that is hard to imagine in
modern government. The next month, a duo of bills continuing the process
reached Mabry's desk. One created a new category of county, notable mostly for
the fact that its officials would receive no state salary. The second sliced
the Los Alamos Reservation out of Santa Fe and Sandoval Counties and
established Los Alamos County in the resulting hole. Hot on the heels of the
new county, the state legislature authorized the incorporation of the City of
Los Alamos. A slew of other bills addressed more minor matters ranging from
regulation of bus service between Española and Los Alamos to, well, the fishing
licenses.
One might wonder why Los Alamos was established as a county, instead of being
merely a city in Santa Fe or Sandoval county. The reason, unsurprisingly, was a
matter of politics: Los Alamos residents wanted representation at the state
level, and state senators and representatives were allocated by county. The
formation of a new county would guarantee positions in the state legislator,
the mere incorporation of a city would not. There had been an effort towards
incorporating a city starting in 1948, but the proposed city charter was set
aside in favor of the county. Ultimately, the project to create the county
determined the structure of both the county and city governments.
Over the course of April and May of 1949, the newborn County, and its School
District elected their first councils. While still largely a military
installation, Los Alamos now had a government other than the AEC and Corps of
Engineers. The fact that the AEC site manager is virtually always quoted in
newspaper articles about the elections reveals the ongoing importance of
federal largess. Indeed, while Los Alamos residents now elected their own
county government, the actual governance of the city remained largely
unchanged: the municipal government was the AEC, advised (but only advised)
by the town council. In some senses, the county was now the municipal
authority, but the AEC owned the entirety of the county, granting it superior
power in many senses. The path to normalization would prove rocky, not least
because of this city-county contradiction.
First, a legal issue relating to the state constitution prevented Los Alamos
residents having a vote for what were ostensibly their state senator and
representative in 1950. Even more, the creation of Los Alamos as a new type of
county produced more problems than anticipated. In the years after 1949,
numerous legal questions had to be resolved. The state legislator had to be
asked to grant Los Alamos County the authority to pass ordinances: the state
attorney general interpreted the 1949 act establishing it as providing only for
the enforcement of state law, perhaps because the AEC was the de facto
municipal authority. But as Los Alamos became less of a military installation,
it seemed more important for it to have a body of municipal law outside of AEC
policy.
In 1951, equipped by the state with the power of ordinances, Los Alamos County
began to institute traffic law. This is not to say that Los Alamosans had been
free to drive whatever speed they wanted, but town police had been enforcing
regulations that were really more AEC rules of personnel conduct than laws.
This was just one of the many small steps required to remove the AEC as the
local authority, and the legal challenges of 1949-1951 paved the way for Los
Alamos independence.
The operation of a town proved rather difficult, though, when considering that
the town was directly interspersed with AEC offices and research facilities.
First, the AEC had security concerns about what were now townsfolk, not
necessarily laboratory personnel, wandering about. Second, the Corps of
Engineers and AEC were rather explicitly looking to get out of the business
of recreational facilities, roads, and even churches. In order to separate the
operation of the laboratory from the operation of the town, they needed to be
physically separated. The saying goes that good fences make good neighbors, but
in Los Alamos it took a canyon.
I should explain, here, that the town of Los Alamos now consists of a series of
mesas. Today, these are Los Alamos Mesa, North Mesa, and Barranca Mesa,
tracking the town's expansion towards the north. Between the mesas are canyons,
so steep that parts of them remain dimly lit during the day. The canyons were
cut out of what was once a plateau by erosion, leaving the mesas behind. to
the west a mountain ridge provided harder ground, stopping the progress of the
expanding canyons. There, Diamond Drive skirts the ends of the canyons,
connecting the mesas together. The overall layout is a bit like a hand, not in
the sense of Michigan, but with separate fingers. The "palm," though, the west
part of the city, is steep and rugged---as much so as the canyons in some
places, limiting urban expansion.
Los Alamos Mesa, which we might call the eponymous mesa, is the oldest part of
the town. During WWII, the core of both housing and laboratory operations
shared the narrow protrusion, edging towards the cliffs as more space was
needed. South of Los Alamos Mesa is Omega Canyon, and its great depths provided
a level of isolation from the rest of the laboratory that made it a relatively
safe place for experimentation. Los Alamos's first reactors operated there. As
the lab expanded its nuclear and high explosive operations, both more
facilities and more isolation were needed, prompting expansion both down
canyons and to neighboring mesas. The lab expanded particularly to the south,
onto South Mesa, across Omega Canyon from Los Alamos Mesa. South Mesa, already
becoming a major part of the lab, was the perfect refuge for the reclusive
installation.
In 1953, the lab up and moved. It was not so much a move as a replacement.
Most of the original WWII structures on Los Alamos Mesa were already in poor or
questionable condition, having been thrown together as temporary facilities for
what was then a temporary project. Nearly all laboratory facilities on Los
Alamos Mesa were demolished, their users moving to newer buildings primarily on
a nearby part of South Mesa known as Technical Area 3, or TA-3 [2].
Facilitating this move, a handsome arched bridge over Omega Canyon, known as
Omega Bridge, was built in 1951 to provide an easy trip between the two mesas
that were otherwise separated by a particularly steep and difficult crossing.
From 1953 onward, Omega Bridge mostly (but not at all completely) separated the
lab from what came to be called the townsite.
Los Alamos was not exactly made normal, though. Even after the separation of
town and lab the townsite continued to be a secure area in the eyes of the AEC,
which operated a checkpoint (depicted in one of the most famous Los Alamos
photographs) on the "Hill Road" up the side of Los Alamos mesa where visitors
had to show identification and a plausible justification for their entry---a
"closed city," just like its Soviet counterparts. Oddly, the opening of Los
Alamos to the general public came with little fanfare or even discussion. In
1957, the AEC announced an administrative decision to discontinue the
checkpoint, allowing just anyone to drive up the hill. It solicited comments
from residents and received only three, two for and one against. The main
objection seemed to be that residents had become accustomed to the "gated
community" life and regretted having to start locking their doors. One
newspaper article snuck in a bit of a jab about how Los Alamosans would now
have to fear unexpected visits by their in-laws, a comment that seemed to
reflect the general view that opening Los Alamos was inevitable and not that
important of an occasion.
At the same time, Los Alamos began to grow. The post-WWII stand-down did not
last long, and as relations with the Soviet Union chilled into the Cold War,
the AEC suddenly became a top priority once again. One of the clearest symbols
of the post-war expansion of the lab is White Rock. Los Alamos county consists
mostly of the lab, the townsite, and an area south of the lab known as White
Rock. White Rock is sometimes considered a "resurrected ghost town." Originally
constructed as a temporary camp for construction workers, White Rock was
completely abandoned for most of a decade, during which all of the temporary
structures were demolished. In the 1960s, increased staffing at the lab created
a housing crunch in Los Alamos County, and a brand new White Rock was built
on the same site.
White Rock is an odd place---seldom has the term "bedroom community" fit a town
better. Los Alamos is small even today, but White Rock is much smaller, and
almost purely residential. It's curiously suburban, considering how far it is
from the nearest large urbanization. While it fits into small Los Alamos county
with the townsite and is just as close to the lab by some measurements, White
Rock is also pretty far away. The most direct road between White Rock and the
townsite passes directly through the lab. Today, it is restricted to badged
employees only, although that's a fairly recent situation. Even so, it has
always been the case that Pajarito Road between the two towns was somewhat
restricted and prone to closures. The next best route between the two means
driving all the way back to "the Y" of highways 502 and 4, basically leaving
Los Alamos county in order to drive back into it again. It's a ten mile trip
over some fairly difficult terrain.
The matter of White Rock's isolation provides a good example of the tensions
between public administration and dependence on the AEC. White Rock has never
been incorporated, but fell under the governance of Los Alamos County. While
the County was nominally independent throughout the '50s and '60s, it continued
to rely on the lab and the AEC to provide most basic services. Residents paid
utility bills to the AEC for power and water and, in case of emergency, called
the AEC fire department.
In the '60s, a house in White Rock burned to the ground. The response by the
AEC fire department was very slow, a situation attributed both to the lack of a
full staff at White Rock's small fire station and the fact that White Rock had
never been connected to the lab's central fire alarm system (presumably a
Gamewell fire telegraph network, common equipment at the time). The lack of a
fire telegraph was compounded by some sort of dispute between White Rock
residents and Mountain States Telephone and Telegraph. I have had a hard time
finding more details on this (and you know it's the kind of thing I badly want
to find details on), but the general situation seems to have been that there
were very few telephone lines in White Rock and Mountain States had required
exorbitant rates to install more.
This is a pretty typical matter of small-town politics, but here the politics
were not between the residents and the county but between the residents, the
county's barely teenage government, and the vast bureaucracy of a secretive
agency of the executive branch. Town hall meetings were held, but progress was
slow. There were many examples of disputes like this, often over rates, the
promptness of repairs, and AEC's cooperation with new housing construction.
The whole thing was very similar to another phenomenon of rural New Mexico
politics, the various small towns and ranch houses that had received their
running water (and sometimes electricity) from the railroad by consequence
of their close proximity to a station or water tower. In both cases, residents
seeking lower rates and modernized service found themselves fighting an uphill
battle against a huge organization that saw them as a low priority, if not an
outright expense.
These problems didn't exactly surprise anyone---the military has found itself
in similar situations many times, and in some ways this was a military
problem, Los Alamos having been willed into existence by General Leslie Groves.
Nuclear weapons, though, had been divorced from the military by the creation
of the AEC. Despite the military legacy, it was now a civilian operation, and
the federal government was not keen to see the AEC build a "force support"
and base operations organization that paralleled that of the military [3].
In 1955, congress passed the Atomic Energy Community Act. The Act ordered the
AEC to begin to sell off their housing stock, transfer commercial,
recreational, and religious property to outside companies and organizations,
and hand off the operation of utilities to local governments or private
providers. The goal of the AECA was to eliminate the AEC towns, encouraging
residents to purchase the housing they occupied (by giving them priority status
and a slight pricing discount) and incorporate governments or start
organizations to assume control of public services. The number of words the act
spends on the issue of government-owned churches highlights the extent to which
the AEC had completely operated AEC towns, but under the AECA the AEC was set
to wash its hands of the whole municipal government exercise within five years.
There was one catch: the AECA did not apply to Los Alamos. In its original
form, it applied only to Richland, WA (at the Hanford site) and Oak Ridge, TN
(at Site Y-12).
The trouble was this: not only residents of AEC towns but also the counties and
states that contained them feared the consequences of an AEC withdrawal. Not
only was the AEC responsible for just about all public operations, AEC towns
were mostly found in remote areas where the cost of these services was quite
high. This was true of Richland and Oak Ridge, but it was especially true of
Los Alamos, which remains difficult for utilities to reach today. The end of
AEC water and electrical service might be the end of water and electrical
service at all, the residents being unable to afford to continue them.
The AECA tried to address this problem on a few fronts. First, it created a
pool of funding from which the AEC would pay not only expenses related to the
transfer but also "local assistance funding" for ten years. The amount of
"local assistance funding" was to be determined by the AEC, but it was
specified that it should cover reasonable costs incurred by the newly
"self-sufficient" communities. Tellingly, the Act says that in determining the
payments the AEC should consider "the fiscal problems peculiar to the
governmental entity by reason of the construction at the community as a single
purpose national defense installation under emergency conditions." It was
understood that AEC towns were not exactly financially efficient, and in most
cases were left to resolve considerable infrastructure and housing problems
resulting from the Army's emphasis on speed over sustainability.
The AECA even included an interesting sort of guarantee of a future for these
towns. It stated that purchasers of housing from the AEC would be repaid from
AEC funds if, at any point in the following fifteen years, the AEC failed to
maintain certain levels of employment and population in those towns.
Still, for Los Alamos the problems were seen as too great. It took the Army to
build Los Alamos and it just about kept the army to keep it running. I will
likely write a follow-up post on the specific matter of water and electrical
supply in Los Alamos, but consider that drinking water for Los Alamos had to be
obtained from wells that were drilled 1,500 to 3,100 feet deep... starting
from the canyon floors. The water produced by those wells was then pumped
upwards of 1,000 feet uphill through rugged terrain. Just the cost of
the electricity to operate the system was considerable.
Los Alamos's exclusion from the AECA did not last long, though. Likely the
relative success of the program in Richland and Oak Ridge was encouraging,
and in 1962 congress amended the AECA to add Los Alamos to the list. The
enthusiasm was measured, though: most of the 5-year deadlines in the AECA
were extended in the case of Los Alamos, with the deadline for transfer
of utilities stretched to 1998.
The land transfer program was slow getting off the ground. It took until 1965
for the first sales of AEC-owned housing to begin, but that year many Los
Alamos residents became homeowners rather than renters. The process was not
entirely straightforward. Army housing built in Los Alamos consisted heavily of
duplexes. The AECA stated that the "senior" resident of a duplex, typically the
one who had lived there longer, had the priority right to purchase the whole
building. In practice, this involved a frustrating paperwork process of
obtaining an application from the AEC office and submitting it, then waiting
to hear the AEC's determination of priority purchaser status, and then putting
in another application to purchase and waiting yet longer.
Still, by early 166 several hundred housing units had been sold, with a little
more than ten additional sales being handled each day. With a total of over
1,500 units to sell the process would take most of the rest of the year, but
the pace was still impressive considering the complexities of real estate
transactions---especially with the federal government. In many cases the AEC
was financing the sales, offering a mortgage backed by the AEC itself to the
buyers, and the 15% discount against appraised value for the single-family
units made it a hard deal to turn down. Many of these duplexes are still
standing today, often with surprisingly ambitious additions.
The fervor of the housing sale process was accelerated by the housing crisis.
It is difficult to find housing in Los Alamos today, and this has been true
almost since Oppenheimer first selected the site. '60s reporting on the housing
sale process is invariably accompanied by coverage of the general housing
shortage and other projects intended to alleviate it, like a planned new
apartment complex in White Rock, the demolition of a number of four-plex
"Sundt" units to make room for more dense housing, and the planning of a new
apartment complex on the edge of Los Alamos Mesa. The concept of developing
the bottoms of some of the wider canyons as neighborhoods was already in play
at this point in time, and as we will see later this plan (and its challenges
and controversies) has been an ongoing part of housing politics.
Urban transformation brought its inevitable controversies. During the sale
process, residents of some Los Alamos homes (such as the "Denver Steel" homes
found near the high school and named for their origin and steel construction)
complained that the AEC was expecting similar prices for homes in their "more
congested" area. AEC management rebutted to the newspaper with a set of
statistics on the ratio of home sizes to lot areas, revealing that even in
cramped Los Alamos residents enjoyed lot sizes that would make a downtown
Albuquerque resident such as myself jealous [4].
The concern over increasing Los Alamos density was just one aspect of the urban
planning confusion. The AEC's administration of the town had erred on the side
of the pragmatic, and on account of the housing shortage the AEC continued its
policy of growth even as the sales process was ongoing. On the other hand, the
residents tended to favor preservation of the town's forested, green space
studded nature---a wonder of mountainous northern New Mexico. One particular
flashpoint was a large grassy lot near the Los Alamos Community Center (today
the Fuller Lodge Art Center). The AEC platted the lot for commercial use, while
the county ruled that the space should be reserved as a park. The controversy
went on for months, with such contributions as a letter from the New Mexico
Garden Clubs calling for preservation. The fact that the AEC owned the land
gave it a lot of authority over the matter, but by the next year the process of
selling the Community Center area, and the Community Center itself, to the
county and private interests had already begun. Ultimately the AEC tended to
pursue its original policies for as long as it owned a section of town, with
priorities shifting to those of the county once an area had been transferred.
The AECA had an express goal of making AEC towns self-sufficient, while still
being comfortable enough places to live that the AEC could readily attract
staff. While this provided some built-in motive for the AEC to support the
town, there remained inherent tensions between the interests of the bomb
builders and those of a small town in northern New Mexico. The AEC, after all,
was not exactly democratic. Despite the incorporation of the county, the people
of Los Alamos still felt that they had a limited ability to actually influence
policy. Besides, as the AEC began to sell off property, its surprisingly
informal, ad-hoc authority in the town began to subside. The transfer of the
schools to the ownership of the county school board in 1966 was taken as one
major sign of the AEC's withdrawal from civil life, but that withdrawal left
a rather significant hole: the city government.
The mid '60s made it clear that Los Alamos needed a city government, but given
the politics of the area there was surprisingly little appetite towards
incorporating Los Alamos as a city. It's always hard to divine the reasoning
for these political decisions when they were made so long ago, but numerous
newspaper articles and letters suggest one clear reason: Los Alamos county
consisted almost entirely of the town of Los Alamos by population, except for
White Rock. The residents of White Rock were already frustrated by the
perceived lack of consideration they received, and incorporating Los Alamos to
leave White Rock as "unincorporated Los Alamos County" seemed certain to worsen
the problem. Besides, the AEC transfer program had to give land to an entity
that existed, not one that might exist in the future, so essentially all public
land and resources were being transferred to Los Alamos County. In many ways,
such as the operation of parks and schools, the County was the municipal
government. The movement of Los Alamos politics shifted away from city
government entirely, and towards the amplification of the County's power.
Just a few years later, this would lead to one of the more curious events
in New Mexico's municipal law.
Let's save that for part II.
[1] Federal enclaves always do, and the concept of a "federal enclave" was
mostly eliminated by SCOTUS in 1953, Howard v. Commissioners, a case dealing
with another WWII war expedient federal installation. That one wasn't nuclear
in nature, but instead a Navy ordnance plant in Louisville. The City of
Louisville thought its employees ought to pay city taxes, the employees didn't
want to, and so legal history was made. They had to pay the taxes.
[2] The practice of organizing land into "Technical Areas" comes from the Corps
of Engineers and was historically seen at a number of military installations.
The term "Technical Area" was originally intended to distinguish areas used for
research and experimentation from cantonment areas, used mostly for personnel
services and housing. Today, the convention has mostly faded from military use
but is almost universal at nuclear weapons sites. Because renumbering TAs is
seen as potentially confusing or at least distasteful, the TA numbers often
reflect history more than geography, such as LANL being headquartered in TA-3.
Sandia is headquartered in TA-1, but has no TA-2 at all, etc.
[3] Not many decades later, the military would also try to get out of the
business of operating bases, significantly reducing the amount of military
housing and privatizing the operation of what remained. The current state of
most military bases suggests that the decision for the AEC to no longer provide
housing to employees was probably the right one, despite the challenges it has
created at Los Alamos today.
[4] My house doesn't actually even fit on my lot, a perennial cause of extra
paperwork and frustrated surveyors. Fortunately most of Los Alamos was platted
by the Corps of Engineers as part of the process of contracting housing
construction, so the surveying situation is relatively straightforward.
On the front page of HN today was an [article with a confusing headline],
"Farmers 'crippled' by satellite failure as GPS-guided tractors grind to a
halt." Of course the headline doesn't actually say this, but it does seem
to imply that there has been some kind of failure of GPS.
This is not quite as outlandish as it might sound. The GPS system, maintained
by the US Space Force, has occasionally suffered from serious funding shortages
and bureaucratic paralysis. In 2009, the GAO issued a concerning report. They
found that the program to launch a new generation of GPS satellites was so far
behind schedule that the system was in danger of falling below the required 24
satellites. This would result in partial outages of GPS in different parts of
the world, and could potentially take years to resolve. In response the Air
Force issued a set of strongly worded statements insisting that they had kept
GPS working for fifteen years and would consider to do so, and moreover made
some changes to accelerate the GPS-III program.
Fortunately, we are now largely out of the woods on this issue, as not just
GPS-III but subsequent designs are in service and the GPS constellation has
been restored to its full planned size---including on-orbit spares for use in
case of an unexpected failure. This is not to say that the administration of
GPS is all good news; the GAO continues to issue more or less annual reports on
how military programs to acquire more advanced GPS receivers (supporting the
newer "M-code" signal) are badly mismanaged. But at least the civilian aspect
of GPS should be quite reliable for years to come.
In any case, any significant failure of GPS would become major international
news. Another perennial topic of GAO reports is the failure of the US
government and infrastructure operators to develop any meaningful backup or
alternative for GPS. A GPS failure would cause huge swaths of transportation
and communications infrastructure to malfunction. While I remain a huge
advocate for the construction of the terrestrial PNT (position, navigation,
time) technology called eLORAN, it would cost money, and not in the F-35 kind
of way, so it's unlikely to happen.
What happened in Australia, the article tells us about halfway through, was
actually a problem with Inmarsat-41. Inmarsat is a bit of a historical oddity,
in the same category as ARINC (Aviation Radio Inc.). It was founded as the
International Maritime Satellite Organization, a non-profit entity sponsored by
the UN's International Maritime Organization to develop a satellite network for
emergency communications at sea. In 1998, though, Inmarsat was privatized,
becoming a British company. Despite its historic legacy, Inmarsat is today just
one of several major commercial satellite communications networks. It offers
two-way telephony and data service, and carries both first and third-party
broadcast services.
One of these services is SBAS, the Space-Based Augmentation System.
"Augmentation System" is a common, if not very specific, term for GPS
correction systems. The topic of GPS correction systems is sort of complicated,
and we're going to celebrate 04/20 by going into them in some depth. Speaking
of celebrating 04/20, this might be more of a ramble than a well-formed
narrative, but there's a lot of interesting ideas related to PNT correction.
First, let's start with the concept of GPS augmentation systems. GPS fixes are
subject to a number of sources of error. A high-quality GPS receiver with a
sufficiently long observation time can eliminate most of the receiver-based
positioning error, but there remain some pernicious errors which are hard to
eliminate. Two of the most prominent are orbital ephemera and atmospheric
effects.
First, orbital ephemera: to produce a GPS fix, receivers need to know the
locations of the GPS satellites in orbit. In order to provide this information,
ground stations observe the locations of GPS satellites and produce orbital
ephemera, sets of parameters that describe the satellite's elliptical paths in
relation to the earth. As the term "ephemera" suggests, these are "point in
time" measurements that describe the current path of the satellites. Details of
how space works like gravitational perturbations mean that satellite orbits are
prone to changes, and satellites sometimes navigate to correct their orbits.
This is why observations are used to determine the ephemera, and these
observations are carried out on a continuous basis.
The GPS operators at the Second and Nineteenth Space Operations Squadrons
regularly upload updated ephemera to the GPS satellites which transmit them for
receivers to use. For practical reasons the ephemera are transmitted at a very
low bitrate and can take some time to receive, which can contribute to "cold
start" times on traditional GPS receivers of upwards of ten minutes. Most
modern GPS receivers use some form of "assisted GPS" to reduce this time, with
the most common example being the widespread practice of smartphone GPS
receivers obtaining current ephemera from a web service instead of "waiting
for it to come around" on the low-rate GPS data feed.
The problem is that new ephemera are only produced hourly, and the orbits of
satellites varies on a minute level too rapidly for hourly observations to keep
up with. Additional latency in the upload process means that the ephemera
transmitted by GPS satellites can be several hours old. The observations are
somewhat limited in precision anyway, considering the incredible precision
modern GPS receivers are capable of. So, variation in the actual orbit of
satellites from the ephemera leads to ephemeris error in GPS fixes.
Ephemeris error isn't actually that big of a source of error, but it's an
interesting one so I wanted to talk about it anyway. Besides, it has
introduced us to the term "assisted GPS" or "AGPS," which is unrelated to
but often confused with augmentation systems. Assisted GPS just refers to
the use of an alternate data path, such as IP, to obtain the current GPS
network information that receivers need to make a first fix.
Second are atmospheric effects. GPS positioning relies on extremely accurate
time-of-flight measurements. Unfortunately, the atmospheric is a weirdly
complex thing, and numerous effects (such as "weather") result in radio
frequency radiation traveling by indirect, non-linear paths. This problem
is especially acute when dealing with satellites since, well, they're up in
space, and so the received signals have to travel through a whole lot of
atmosphere to get down here. This includes the ionosphere, which interacts
with RF in particularly strange ways and can add significant travel time.
There are a number of approaches to reducing these errors, but one of the most
common is differential GPS or DGPS. DGPS relies on a convenient property of
most sources of GPS error: satellite position, satellite clock drift, and
atmospheric conditions all tend to contribute a pretty similar error across a
region. This means that if you can measure the error of GPS fixes in one place,
the error at other nearby places is probably pretty similar. This is exactly
what DGPS does: a reference station with a precisely surveyed location
uses a high-quality GPS receiver to obtain very accurate fixes. The difference
between the GPS fix and known location is then distributed as a correction
signal that can be applied by other GPS receivers in the same area, canceling
much of the inaccuracy. DGPS is one of the most common types of augmentation
system, and is widely used by higher-quality GPS receivers.
Numerous augmentation systems exist, which makes them more complicated to talk
about. The big issue is how the correction signal is distributed. Historically,
one of the earliest major DGPS systems was the NDGPS or Nationwide DGPS. NDGPS
is operated by the US Coast Guard (having originally been developed for
maritime navigation) and uses a network of site across the US that transmit
correction signals at around 300kHz. NDGPS has a great historical detail: when
it was expanded from coastal stations only to a nationwide system, a great deal
of the new inland correction stations were installed on recently retired sites
of the Ground Wave Emergency Network (GWEN). GWEN was a short-lived survivable
radio system operated by the military for continuity of government and command
and control purposes, and much of the modern conspiracy theory around 5G can be
directly traced to historic conspiracy theories surrounding GWEN as a
government mind control system.
NDGPS is now being retired, though, as WAAS is viewed as a replacement. WAAS,
the Wide Area Augmentation System, is the FAA's version of the same concept,
intended for aviation use. Most WAAS reference stations are installed on the
roofs of air route traffic control centers, with some others added as needed to
fill gaps. WAAS differs from NDGPS in an important way: WAAS correction signals
are actually distributed by three different commercial communications
satellites. WAAS comes from space, just like GPS itself.
This is a good time to point out an odd bit of terminology: DGPS and
augmentation systems are essentially the same thing, but for largely historic
reasons "DGPS" usually refers to correction signals distributed by terrestrial
radio while "augmentation system" usually refers to correction signals
distributed by satellite.
You can probably see where this goes with Inmarsat. WAAS provides complete
coverage of North America, but not of elsewhere. Inmarsat operates equivalent
space-based augmentation signals as a contractor to several national
governments, including Australia, New Zealand, and likely soon the UK. And
that's what broke: some kind of problem with the satellite caused a
disruption of the SBAS feed for Australia.
While we're talking about GPS augmentation, we should also talk briefly about
RTK, Real-Time Kinematic correction. RTK, DGPS, and augmentation are sometimes
used almost synonymously, but once again there is a conventional difference in
the meanings. RTK most literally refers to the observation of the phase of the
the GPS signals. The phase can be used as additional timing information to
estimate the distance between the receiver and the satellite. Microwave GPS
signals have a short wavelength (about 19cm), and electronics for observing
phase difference can be made very precise, so this method can produce extremely
accurate fixes. The problem is that RTK only tells you where the receiver is
within a wavelength, or in other words it tells you where you are in a 19cm
window, but not which 19cm window you're in.
To resolve this mystery, RTK is almost always used with a reference station so
that the phase can be compared between a fixed point (with a location
determined through long, 48-hour+ observation) and the moving receiver used for
surveying in the field. Since RTK is mostly used by surveyors who are striving
for huge accuracy and aren't in a hurry, it's most often done with a portable
reference station installed nearby for the duration of the surveying project.
If you've ever run into a skywards antenna connected to a chained-down Jobox
with a solar panel, it's very likely an RTK reference station installed for
some highway construction project. While RTK tends to imply DGPS, it's a
separate technique, and produces even higher precision than DGPS or
augmentation. It's also possible to use RTK without a reference station at all,
by analyzing fixes from multiple locations later... a method referred to as
virtual reference station.
So how much does this stuff actually impact our lives? Well, for the most part
we tend to use smartphones for PNT in our daily lives, and smartphones have a
somewhat different bag of tricks that relies on the cellular network to assist
positioning. The thing is, smartphone GPS receivers really aren't very accurate
at all. They rely on network assistance not for precision but in order to
obtain any fix at all. Smartphones are constantly used in situations like urban
environments and even indoors where GPS reception is poor and subject to huge
multipath error. Cellular receivers bootstrap based on hints from the network
(using the known locations of cellular towers) to accelerate GPS fix, and to
produce a fix at all when they can't detect signals from a sufficient number of
GPS satellites.
And that's it on PNT for today, but maybe I'll come back to shill more for
eLORAN in the future.
One of the foundational goals of computer technology, at least as understood by
popular culture, is to automate away our jobs. When your job is (put simply) to
get computers to work correctly, there's sort of an irony to this view of
things. Still, just within the span of my career there have been technological
advancements that have both automated away parts of my job (Puppet might be an
example here, the prior art of CFEngine notwithstanding) and given me plenty
more to bill time for (Kubernetes).
With the rise of large language models (LLMs), there is a large camp of people
who think these ML applications are going to automate away larger portions of
more jobs. I find myself curious: should I fear for my employment?
The thing is, a large portion of my profession could be accurately described as
"Googling it," although lately I'm racking up a Kagi bill instead. Obviously
this is an important strategy for things like error messages where it's often
faster to see if someone has solved the same problem before than to figure it
out from first principles. But it's sort of true for routine tasks as well...
maybe my memory is just unusually poor, but there's a huge set of tasks where I
will repeatedly look up the same Digital Ocean guide because I just want to
copy-paste their config file snippets instead of having to look in docs to
remember the names of the right directives.
Of course, what makes me a "professional" is that I have opinions about the
correct ways certain things should be done, so I often ignore parts of these
guides or make changes to suit my preferences on important issues like Unix
domain sockets or localhost network sockets for communication with application
servers. In a way it's sort of silly to follow some guide on
best-linux-tutorials.biz and then just ignore the parts I disagree with but,
well, that's what I tend to do. So I wonder: would a better approach be an LLM?
In today's episode, I'm going to ask ChatGPT for guides for some increasingly
complex Linux sysadmin and DevOps tasks and then see whether or not I agree
with its output. I don't really intend this as that serious of a critique of
the use of ML in operations but hey, maybe we'll learn something useful!
The LEMP Stack
Back when I was a child first learning to hate software, the dominant stack for
websites was "LAMP." This stood for Linux, Apache, MySQL, PHP. Nowadays it
seems that at least a few people are saying "LEMP" for Linux, eNginx,
Mymariadb, PHP? I don't like this, aesthetically. In any case, there's a number
of things in PHP that I tend to deploy a lot, Dokuwiki being a prime example.
This means that I am repeatedly wanting to set up Nginx to proxy requests to
PHP-FPM, which is not difficult but does involve remembering a couple of config
directives and where php-fpm puts its socket by default. Most of the time I
don't actually want an RDBMS (personally I usually just use sqlite for
everything) so for a long time I've googled for some guide and copied their
snippets while ignoring the parts about MySQL/MariaDB. Let's see if ChatGPT
has something more to the point for this very common and simple task.
Prompt: "how do I set up nginx and php?"
To set up Nginx and PHP on your server, follow these steps:
Install Nginx on your server:
For Ubuntu/Debian:
sudo apt update
sudo apt install nginx
For CentOS/RHEL:
sudo yum update
sudo yum install nginx
The output starts off strong by offering snippets for both "Ubuntu/Debian" and
"CentOS/RHEL." These two cover the good majority of the Linux server landscape,
and while I could quibble with the label "CentOS/RHEL" rather than something
that doesn't invoke the mostly-dead CentOS project like "RHEL/Fedora," ChatGPT
is following the same convention most people do.
The instructions it provides are pretty reasonable, but there's something
interesting here that those of you familiar with the two dominant Linux package
manager families might detect. For Debian, it tells us to 'update' and then
'install.' for RHEL, it tells us to 'update' and then 'install.' These are
neatly parallel except that the 'update' subcommand of apt and yum do fairly
different things! On Debian it will retrieve the repo metadata and then install
nginx (upgrading software is a different subcommand), on RHEL it will upgrade
all of the managed software on the system and then install nginx (repo metadata
is retrieved automatically by yum when it exceeds a configured 'stale'
timeout).
I'm not sure how much we can make of this. It seems likely to me that we are
seeing ChatGPT's lack of understanding of the underlying material: it is
extremely common for people to 'update' and then 'install' on both platforms,
so each in isolation is pretty reasonable, but it's odd for it to put them in
parallel without noting that they will do different things. If the 'yum update'
were an intentional choice we should see 'apt upgrade,' but we don't.
Install PHP and the required PHP extensions:
For Ubuntu/Debian:
sudo apt install php-fpm php-mysql php-mbstring
For CentOS/RHEL:
sudo yum install php-fpm php-mysql php-mbstring
This is a little odder. It says "the required PHP extensions" which are
apparently mysql and mbstring. It's not clear where or why these are
required, which makes me think ChatGPT got this command from some guide on
deploying specific software. WordPress might be to blame here as it requires
mbstring, and the sheer number of LAMP/LEMP guides for the assumption that
we will need the mysql extension.
Configure Nginx to use PHP:
Create a new server block configuration file in /etc/nginx/sites-available directory with the following content:
server {
listen 80;
server_name example.com;
root /var/www/html;
index index.php index.html index.htm;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
}
Replace example.com with your domain name and php7.4 with your installed PHP version.
This is pretty much what you would find in any guide. One interesting thing is
the 'try_files' directive. There are about a million ways to set this up in
nginx and some people prefer different approaches, but what we see here is
pretty typical of php setup guides and should work fine. The catch is
explicitly passing the $query_string which sort of gets eaten otherwise when
nginx internally redirects to use index.php.
Test your Nginx configuration:
sudo nginx -t
Reload Nginx to apply the new configuration:
sudo systemctl reload nginx
Hey, nice work! A surprising number of people don't know about '-t' to validate
config files.
Create a PHP file to test if everything is working:
sudo nano /var/www/html/info.php
Add the following content:
<?php
phpinfo();
?>
Open your web browser and navigate to http://your-server-ip/info.php. You should see the PHP information page.
That's it! You now have Nginx and PHP installed and configured on your server.
So yeah, this isn't exactly part of the install process, but still very
normal for guides.
Overall I'd say this is about an 8/10. The only things I disliked were:
Telling the user to upgrade packages on RHEL... not an unusual thing in
guides to update first "for good measure" but I personally feel that it's not a
great practice to tell the user to 'yum update' without explaining that will
update anything.
Telling the user to install mysql and mbstring, which aren't really going to
do any harm, but if you're going to include a "common" set of PHP extensions
in the instructions it should be bigger than just these two. It's just odd.
'du' and 'df' disagree
This is actually an interview question I like to use for Linux-intensive
positions. It might seem a little in the weeds but to be honest I think I run
into this situation in real life probably at least once a month. The problem
looks like this: tools like 'df' tell you that a partition is 99% or 100% full,
and you might be having problems because of the full partition. But when you
use 'du' to investigate, it shows a total usage that is appreciably less than
'df' does. Why do these two-letter commands not get along? is dd somehow the
answer?
Prompt: On Linux, a partition is full but du doesn't show as much data as df does.
If you are seeing a discrepancy between the output of du and df on a Linux
system, where df reports that a partition is full but du does not show as
much data, it's possible that there are files that are being held open by
processes and therefore are not being deleted even though they have been
unlinked (deleted).
This actually isn't the answer that I expect but it's a reasonable answer. This
could very well be the cause, I just find that this situation is less common
than having a mount that "shadows" data on the volume at the next higher mount
point. For example, say you have a volume mounted at / and a volume mounted at
/home. If you have some kind of problem that results in /home failing to mount,
or changed the filesystem setup at some point, or even have weird boot-order
problems, you can sometimes end up with data in /home/ on the volume mounted at
/. This data counts against the usage of the volume at / but won't show up in
tools like 'du' since it's "shadowed" by /home/ now being a mountpoint to
another volume. Easy problem to have if you are rearranging the volumes/mounts
on a system that's already been in use.
I'll omit the rest of the output, but it makes sense. It tells us to use 'lsof'
to figure out what process is keeping the files open and then kill that
process. The guide does include the requisite paragraph that you need to
investigate to determine how this happened in the first place.
Overall I'm giving this a 5/10, because it gave a reasonable answer but omitted
the possibility of mount shadowing. Which of these two causes is more likely
depends on the system and workload, but I think a thorough answer should
include both possibilities.
Per-process network policy
One sort of common "advanced" Linux networking scenario is when you are using a
full-tunnel VPN and want to route all traffic through it, but you have to get
the VPN itself to connect to its endpoint without trying to go through itself.
There are a few ways to solve this problem, but one of the less common and (in
my opinion) more elegant approaches is to get the VPN service to use its own
special routing table. This same concept also makes it easy to set up firewall
policy around the VPN to prevent leakage. Now there are a lot of caveats to
this and I'm really just talking about userspace VPNs here, but that probably
makes it a good challenge for ChatGPT.
prompt: How do I configure per-process route tables on Linux
To configure per-process route tables on Linux, you can use the ip command and the setsockopt() system call.
This will create a new routing table with the name mytable and the ID 200.
It maybe seems a little weird that this is the way to create a routing table
but it's fine. The iproute2 stuff that facilitates multiple routing tables
without multiple headaches is pretty new and there isn't a lot of porcelain
around it... and I'm not sure there ever will be, this isn't a very common task
and while editing the file seems a little old-school compared to most of the
contemporary network tooling it works just fine. There's very little need to
expose this over a bus or anything.
Configure the routes for the new table:
sudo ip route add default via dev table mytable
This command will add a default route to the new table with the specified gateway and interface.
I will take this moment to make a couple of funny observations about the
mechanics of ChatGPT's output. First, every step in this guide is numbered
1. Some things here are probably copy-paste errors on my part (I'm reformatting
the output to look better in plaintext), but that isn't, this output has four
step ones. Another is the code blocks. I'm just using plaintext but ChatGPT
uses nice embedded code blocks with syntax highlighting. Syntax highlighting
means that it needs to know (or guess) the language it's highlighting and
the codeblocks show what language they're set to. It is... seemingly random?
The above shell command is 'csharp', but 'sql' seems to be a more common
guess for shell commands than even 'bash'. This just means that most of the
time the syntax highlighting doesn't really make sense.
I'm not sure where either of these are going wrong, but I get the feeling
(without doing any actual research) that ChatGPT is outputting markdown.
Markdown numbered lists can be fiddly and there are a few ways to end up with
all step 1 like this. When it comes to the code blocks, I'm not sure if chatGPT
is specifying the language or if the code block service is guessing it based on
contents. I tend to guess the former because otherwise the guesses are very
weird. I can't see any reason it would think one of these commands is csharp
and the other is sql, they both have the same set of characters in them.
And yeah, this instruction makes perfect sense. You need to add some routes to
the new route table.
Configure the process to use the new table:
sudo ip rule add from table mytable
This command will add a new rule to use the mytable routing table for traffic originating from the specified IP address.
:thinking_emoji:
There's some clear semantic confusion here. I think ChatGPT is just going
too readily to the many guides it's read that create a route table and then
apply that route table based on ip rules. This is a very common thing to do,
I would wager the most common way to use custom route tables, but it doesn't
help in our case. The intro of this step is also just wrong (this has nothing
to do with the process), but the sentence after correctly explains what it does.
Finally, start the process with the setsockopt() system call to bind it to the new table:
This code will bind the process to the eth0 interface and mark its packets with the mytable routing table ID.
This is, well, it's not exactly wrong but it's also not right.
If I fiddled with the prompt I'm sure I could get it to output something that
makes more sense, but I also feel like this is an unreasonable way to interpret
the prompt I did provide. To be clear, I expected that it would use cgroups and
iptables to apply the new custom route table to some existing application. This
seems like a way more common situation than to... write or modify an existing
application in C. On the upside, the code block correctly has the language as
'c'!
All that said, if we start from the assumption that we are writing a new
application this still doesn't work exactly. The option we see here, SO_MARK,
applies a 'mark' in the sense of netfilter to traffic on the socket. Marks are
a super useful feature that allows us to set an arbitrary tag on each packet
that we can then refer to in our other network policy. A very common use for
marks is to match the traffic up with an iptables rule that sets the routing
table for that traffic... but ChatGPT didn't tell us to set that rule, it had
us set the rule based on source port instead. I think maybe ChatGPT was
trying to cover this in step 1 number 3 but it didn't quite choose the right
rule.
Repeat steps 3 and 4 for each process that needs to use the new routing table.
There are no steps 3 and 4! There is only step 1!
Overall I'd give this a 3/10. Honestly I think ChatGPT did better than I
expected when considering that this is not a very common use-case, but
ChatGPT's output is worse than what you get if you google the same question.
Takeaways
This isn't intended to be a super carefully thought out research piece, but
like others I've seen reporting that suggests that ChatGPT is pretty good at
generating source code but has a tendency to leave subtle errors. I wondered
if the same thing might be true in the old-school system administration space,
but the feeling I walked away with is a little different: ChatGPT does make a
lot of errors, but they aren't particularly subtle. I suspect this has to do
with the different nature of the training material for these kind of questions,
which is more in the form of narrative guides and documentation that keep steps
simple but leave a lot of assumptions.
I pretty much covered every question I thought to ask in a short time spent
playing around. There was one question that I was just unable to get ChatGPT to
generate a good answer for... an AWS network architecture question involving
load balancing for both HTTP(S) and arbitrary TCP services on the same elastic
IP when you want layer 7 behavior. I could mark this as a 0/10 for ChatGPT but
it might be a case of more "prompt engineering" being required... it kept
producing output that was reasonable but failed to address one or more of my
requirements, so I kept making the requirements more explicit, and then it
would just miss a different set of requirements. It may very well be possible
to get ChatGPT to produce a correct solution but it was definitely getting to
be less useful than a search engine, even with how difficult the AWS
documentation can be to use for architecture questions.
On the whole, I felt that ChatGPT was performing more poorly than Google for
similar queries. It's possible to get ChatGPT to refine its output by adding
more detail (at least if you don't get too deep into AWS networking
capabilities), which is a big plus over a conventional search engine, but
honestly it still didn't feel to me like this was a savings of effort over
reading a few different articles and synthesizing.
One of the reasons this was on my mind is because I'm working with a client
right now who has an interesting habit of copying and pasting all the error
messages they get into ChatGPT, while still screen sharing. From this sort of
eavesdropping on AI I have not been very impressed with its output, which has
often been high-level to the point of uselessness. It sort of has the feel
of AWS documentation, actually... he would put a very specific error from,
let's say, Terraform into ChatGPT and it would answer with a few paragraphs
about how Terraform works at a high level. I'm sure this can be improved with
a more specific training corpus but I'm not sure what that corpus would be,
exactly, which continues to stymie my plans to just forward all the emails I
get from clients to ChatGPT and give them the response.
I mean, I think it would keep them going back and forth for a while, but they
might feel like it's a good value for money.