Previously on Deep Space Nine, we discussed (read: I complained about) the
short life and quick death of DLNA. On the way, I mentioned DLNA's dependence
on UPnP as an underlying autoconfiguration protocol. Let's talk a bit more
about UPnP, because it's 1) an interesting protocol, 2) widely misunderstood,
and 3) relates to an interesting broader story about Microsoft and Web
Services. It'll also be useful background knowledge for a future post about the
thing I originally intended the last post to be about, the Windows Media
Center-based home automation platform exhibited at Disney's Innoventions Dream
Home somewhat over a decade ago.
UPnP has sort of a bad reputation, and it's very common in online conversations
to see people repeating "disable UPnP" as stock advice. There is some reason
for this, although the concern is mostly misguided. It's still amusing, though,
as it relates to the one tiny corner of UPnP functionality which has ever
really had any success.
Before we dig into that, though, we need some history.
UPnP fits into a broader space called zero-configuration networking, which I
will be calling 0CN for short because the more common Zeroconf is, technically,
the name of a specific protocol stack.
0CN aimed to solve a big problem for early adopters of local area networks. You
can connect all your computers to a network, but then they need to be numbered
(addresses assigned) and then configured to know what addresses are capable of
offering which services. In early small networks this could be relatively
simple, but between the mass migration onto the relatively complex TCP/IP
stack, the ubiquity of "non-consumer" automatic address assignment via DHCP
[1], and the increasing number of devices on home networks caused by WiFi... it
was turning into a real pain.
It's just hard to sell a product to a consumer as "fun and easy to use" when
the setup instructions are going to involve some awkward arrangement to get the
device's IP address, then having to go enter that into other devices in order
to use the new service. But with a typical pre-0CN home network environment of
IP addressing assigned by DHCP (probably by ISC dhcpd running on a consumer
router), this was pretty much the only option for devices or software to
discover and connect to other network participants.
Some network developers had addressed these problems surprisingly thoroughly in
various pre-IP network stacks. For example, AppleTalk is often raised as a
prime example---AppleTalk is a pre-IP protocol based on Xerox's XNS that had
autoconfiguration and service discovery baked into it from the start. A number
of other early LAN protocols were similar. But the "everything over IP" trend
could be analogized to parking a heavy diesel truck in the driveway. It's
highly capable, which is attractive, but it was not designed for consumer use.
The industry loved TCP/IP as a powerful common platform, but typical home
computer users didn't want to learn to double-clutch. In this way, TCP/IP for
home networks sometimes felt like a regression compared to the various (often
XNS-based) ease-of-use-oriented precursors.
0CN, then, can be viewed as an effort to re-consumerize consumer networks, by
smoothing over the parts of TCP/IP that were not easy to use.
0CN systems thus tend to concern themselves first with discovery, that is, the
auto-detection of services available on a local network. Discovery tends to
further imply description, which is the ability of discovered devices to
describe what they are capable of and how to interact with them. Most computer
interconnects have some form of discovery and description, but 0CN differs in
that these capabilities are intended to be very high-level. What is
discovered is something like a media server, and it describes itself in
terms of protocols that can be used to retrieve media and their endpoints. This
is a much more end-use oriented form of discovery than we see in lower-level
discovery protocols such as ARP.
0CN systems may, or may not, extend discovery and description with a set of
standardized communication protocols to be used after discovery. For example,
0CN standards may specify the type of API that a device should present once
discovered. This may include standardized message bus protocols and other
application-level concerns. Later 0CN systems tended to be more prescriptive
about this kind of thing to ease implementation of "universal" clients, but
there are still plenty of 0CN standards around that leave all of it to
individual device and software vendors.
One of the weird things about 0CN is the relative lack of standardization.
It's not that there isn't a well-accepted standard for 0CN, it's that there's
like four of them. To be reliably discovered by a variety of operating systems,
devices typically need to implement multiple 0CN standards in parallel.
Here's a brief summary of 0CN protocols in common use today:
In the Windows world, NetBIOS specifies a basic discovery system for both
advertising capabilities and name resolution (of NetBIOS names) [2]. This
protocol is looking pretty crufty today but is still in reasonably common use
between Windows hosts offering file or printer shares. In part in response to
the limitations of the NetBIOS mechanism, Microsoft introduced WS-Discovery
(part of Microsoft's larger Web Service craze which we will discuss in the
future), which is a little more modern and a little more powerful. It is still
frequently used by network printers to advertise their capabilities. UPnP, as a
Microsoft protocol, is widely supported by Windows hosts and less widely by
devices such as printers and consumer NAS.
In the closer to UNIX world, discovery approaches derived from DNS are popular.
The most prominent is a combination of the mDNS distributed DNS service with
the DNS Service Discovery (DNS-SD) standard, which allows the distributed DNS
mechanism to be used to describe capabilities as well as presence and name
(basically by using a set of specially crafted SRV records). Apple's Bonjour
and the open source Avahi are both implementations of this mDNS-and-DNS-SD
combination. "Zeroconf," with a capital Z, has a tendency to refer to this
stack as well but there's a lot of inconsistency in how the term is used.
So this is the landscape in which UPnP exists---already somewhere between
solved and hopelessly fragmented. UPnP intended to compete, in part, by being
a more complete standard with higher level components. So, for example, UPnP
specifies the general structure of device APIs (XML SOAP as discovered from
the XML device description), an event system (basically a simple subscription
message bus), and a very lazy end-user standard that basically suggests that
all UPnP devices should offer a web interface.
More significantly, UPnP was expanded significantly into the A/V use-case. The
UPnP A/V spec includes a basic media architecture, including media servers,
renderers, and control points. UPnP specifies not just discovery but also
protocols between these devices. You are likely now wondering what the
difference between UPnP and DLNA is, and that can be a confusing point. DLNA
is directly based on UPnP, or perhaps it is more accurate to say that DLNA
incorporates UPnP. DLNA uses all of the UPnP protocols, including A/V, but
extends the UPnP specification with much more detailed standards for content
management. A simple way to think about it is that DLNA is the upper level while
UPnP is the lower level. This of course does not totally help with the fact
that DLNA was so closely associated with UPnP during its lifespan that it's
not uncommon for people to use the two terms interchangeably.
So, how does UPnP actually work? First, UPnP does not handle addressing but
instead incorporates either DHCP or RFC 3927 link-local addressing. So, UPnP
functionally starts at the discovery layer. For discovery, UPnP incorporates a
protocol that never quite made it on its own: Simple Service Discovery
Protocol, or SSDP. SSDP is actually as simple as the name suggests, but due to
Microsoft's incredible love of web services it uses an unusual transport. SSDP
runs on top of HTTPU, or HTTP over UDP, a protocol that basically consists of
taking a small HTTP payload and putting it in a single UDP packet.
A client wishing to discover services sends a simple HTTP request, in a UDP
packet, to the multicast address 239.255.255.250, port 1900. Any device that
has a service to offer replies with an HTTP response to the source IP and port
on the request. To help reduce traffic volumes, SSDP also allows devices to
gratuitously advertise their presence to the multicast address, and all clients
are permitted (and encouraged) to passively discover devices by monitoring
these extra announcements.
The next step is description. UPnP description is quite simple: the device
discovery information provides a URL. The client requests that URL to receive
an XML document that describes the device and gives a list of services it
provides. Each service is defined as a set of endpoints and commands that can
be issued to those endpoints.
For the most part, a UPnP client will now interact with the device by issuing
commands to the described endpoints. This is all done, following the general
trend you may have noticed, with XML SOAP over HTTP. That's basically the end
of the UPnP story, but UPnP does add one interesting additional feature: an
eventing or message system. The UPnP description of a service lists a set of
variables that describe the state of the service, and provide an event endpoint
that allows a client to subscribe for notifications whenever a particular
variable changes. This is all, once again, done with XML over HTTP.
So this is all interesting and helps to describe the underlying structure of
DLNA (which can be viewed as just a set of standardized, strongly specified
UPnP services). But it has oddly little to do with what we know UPnP for today.
What gives?
The service discovery and description functionality of UPnP is, for the most
part, boring and transparent. It is used in practice for things like
autoconfiguration of printers, but it doesn't usually do anything that notable.
What UPnP is broadly known for is NAT negotiation.
Many home devices that offer services might want to be internet-accessible, but
the fact that most home networks employ NAT makes it difficult to set that up.
UPnP intended to resolve this problem by throwing a NAT port mapping protocol
into the UPnP standard. Called Internet Gateway Device Protocol (IGDP), it
allows a UPnP client to discover a router, and then specifies a service the
router provides that allows a UPnP client to request that a given port be
opened from the internet back to that device. As a bonus, it also allows a UPnP
client to request the current external IP address from the router, so that it
knows its internet endpoint.
IGDP is a sibling to Port Mapping Protocol and the later Port Control Protocol
(PCP), but for a few reasons it is much more common. One of the reasons is a
business one: it's pretty much correct to say that UPnP IGDP is the Microsoft
solution, while PCP is the Apple solution. PCP is implemented and used by a
variety of Apple products, but UPnP support is more common in consumer routers.
In the eyes of many power users today, UPnP's service discovery function is so
little discussed that UPnP is more often used as a mistaken reference to IGDP
specifically (router vendors and innumerable support forums do not help to
alleviate the confusion here).
Because of security concerns around providing a low-effort way for software to
map incoming ports, it's common security advice to "disable UPnP," which in
practice means disabling IGDP... but since IGDP is the only meaningful service
provided by most consumer routers, that's often done by disabling the whole
UPnP implementation and it's labeled as such in the router's configuration
interface. I am actually somewhat skeptical of the security advantages of
disabling UPnP for this purpose. The concern is usually that malware on a
machine in the local network will use UPnP to map inbound ports, allowing
direct inbound connections to things usually not accessible from the internet.
That problem is real, but in practice malware running somewhere on the local
network has many options for facilitating external access and UPnP isn't even
really one of the easier ones.
There's actually a much better and less often discussed reason to be cautious
about UPnP: there is a long track record of embedded devices like routers and
IoT "things" having poorly implemented UPnP stacks that are vulnerable to
remote code execution. Several botnets have spread among IoT devices through
UPnP, but it had nothing to do with port mapping... instead, they took
advantage of defects in the actual UPnP implementations.
What's worse is that many IoT UPnP implementations turn out to have a defect
where they do not correctly differentiate requests coming from the LAN side vs.
the WAN side. This has the alarming result that it is possible to request and
receive port mappings from the WAN back to the WAN. All devices with this
defect are potential participants in reflected DDoS attacks, and indeed have
been widely abused that way.
The point is that I would agree that it's a good idea to disable UPnP, but not
because of what UPnP does, and not just on your router. Instead, it's a good
idea to be very skeptical of UPnP because of defective implementations in many
embedded devices, especially routers, but also all of your IoT nonsense.
Ultimately, 0CN almost feels obsolete. Most modern products have a near total
dependence on a cloud service, and simply use the cloud service as a broker
instead of performing service discovery. Where devices do need to be
discovered, it's usually pre-configuration for WiFi networks, and so it has to
be done over a side channel like WiFi Direct or Bluetooth. This has the
substantial downside that these seemingly local devices will stop working if
the internet connection is lost or the service unresponsive, but it's the 21st
century and we've all come to accept that our light switches are now dependent
on Amazon somehow.
More seriously, there is a general trend that more expensive, higher-quality
IoT products are more likely to perform service discovery and communicate on
the local network. This likely happens because their developers realize that
support for local communication will result in lower latency and better
reliability for many users. But ultimately reliance a cloud broker is easier,
so a lot of even high-end products ship a cloud intermediary as the only
way to communicate with them.
[1] DHCP might sound like it would be part of a 0CN environment, but 0CN is
usually used to describe easy-to-use, highly automatic, consumerized protocols
in contrast to configuration protocols like DHCP that were designed for large
networks and so are relatively complex and difficult to work with.
[2] Peer-to-peer discovery and name resolution protocols tend to have inherent
scalability problems due to their dependency on broadcasting. NetBIOS, having
originally been designed as a pretty complete and sophisticated network
application standard on top of pre-IP protocols, resolves this scaling problem
by supporting centralized name service for large networks. That centralized
name service is called Windows Internet Name Service or WINS, an acronym you
have probably run into before dealing with Windows network configuration. WINS
is essentially a pre-IP DNS, for the much more limited NetBIOS name standard.
It is fairly rare to encounter a WINS server today as Microsoft has shifted to
DNS for almost all purposes, even for establishment of NetBIOS connections
where the difference between "NetBIOS name" and "DNS name" can now be very
fuzzy. E.g., the UNC path "\\a.multipart.domain.name\a\share" has worked fine
for many years, even though the name is illegal for NetBIOS... the Windows
NetBIOS client has been fine with using DNS names pretty much since NT stopped
being called NT. NetBIOS names are also still supported but it's rare to
encounter a machine where the NetBIOS name is not the same as the local part of
the DNS name, making it not obvious that the distinction even exists.
Sonos offers a popular line of WiFi-based networked speakers that are
essentially a consumer distributed audio system. It's a (relatively) affordable
and easy to use spin on the network-based audio rendering systems already
common in large building background music/PA and, increasingly, entertainment
venue and theatrical sound reinforcement. The core of it is this: instead of
distributing audio from one amplifier position to multiple speakers over
expensive and lossy (in commercial contexts) or difficult to install (in
consumer contexts) speaker-level audio cables, audio is distributed over an IP
network to a small amplifier colocated with each speaker [1].
This isn't a new concept, although Sonos had to invest considerable effort in
getting it to function reliably (without noticeable synchronization issues)
over unreliable and highly variable consumer WiFi networks. From the
perspective of commercial audio, it's just a more consumer-friendly version of
Dante or Q-LAN. From the perspective of consumer audio, it's either
revolutionary, or the sad specter of two decades of effort in network-enabled
consumer AV systems. After all the work that was done, and all the technologies
that could have succeeded, Sonos is what we ended up with: a
feature-incomplete, walled-garden knockoff of DLNA.
Yes, I'm being unfair to Sonos. The biggest problem that Sonos figured out how
to solve, precise and reliable synchronization of audio renderers without
special network facilities like PTP, isn't one that DLNA attempted to address.
And it remains a hard problem today; even my brand-new bluetooth earbuds
regularly experience desynchronization problems when my phone's mediocre
Bluetooth stack gets behind (maybe my ailing phone is more to blame than
theoretical complexity).
But every time I really look at today's home media streaming and management
landscape, which is largely dominated by Sonos, Apple's AirPlay, and Google's
ChromeCast, it's very hard not to see it as a shadow of DLNA.
So what is DLNA, and why did it fail? In general, why is it that all of the
home network AV efforts of the late '90s through the '00s amounted to nothing,
and were replaced by thin, vendor-locked "cast" protocols?
The answer, of course, is Microsoft and Capitalism (rarely have there been two
more ominous bedfellows). But let's get there the long way, starting with the
evolving home media landscape of the late '90s.
The compact disc, or CD, came into widespread popularity in the late '80s and
represented a huge change in music distribution. Besides the low cost, small
size, and excellent fidelity of CDs, they were the first widespread consumer
audio recording format that was digital. For the first time it was, in
principle, possible to create an exact digital copy of CD. Once written to
another storage device, the CD could be handled as computer data [2].
"CD Ripping" actually did not become especially common until the early '00s.
In practice, "ripping" audio CDs from PCs is somewhat complex because of the
surprisingly archaic architecture of PATA CD drives. Early computers, in the
'90s, often weren't capable of fast enough I/O to read an audio CD at 1x speed
(meaning as fast as the audio bitrate, allowing real-time decoding). Even for
those that were, audio CDs amounted to hundreds of megabytes of data and hard
drives that large were very costly. As a result, PC CD drives played audio CDs
by behaving as plain old CD players and outputting analogue audio. Some of you
may remember installing an IDE CD drive and having to connect the three-wire
analogue audio output from the CD drive to the sound card. Some of you may even
remember the less common CD drives with discrete playback control buttons on
the front panel, allowing it to be used to play music without any media
software to control it.
These CD drives, when playing audio, behaved a lot like smartphones making
phone calls: the computer actually wasn't "in the loop" at all, all it did was
send the CD drive commands to play/pause/etc and the CD drive decoded the audio
and converted it to analog internally, sending it straight to the amplifiers in
the sound card. With this architecture, the way to "rip" a CD was actually to
tell the CD drive to start playback and then use the soundcard to record its
analogue output. This ran only at real-time speed, not all soundcards were
capable without an external jumper from the line out to line in, and the double
conversion reduced quality. It didn't take off, although Gnome's Sound Juicer
continued to use this method until around 2005.
The much better method, of actually reading the CD as data and decoding the
audio in software, took off in the '00s, mostly in the form of WinAmp 5.0 and
Windows Media Player in XP. It greatly accelerated a trend which already seemed
clear to industry: consumers would purchase their music on physical media
(downloading it was still infeasible on most consumer's internet connections),
but in the future they would immediately rip it, store it on a central device,
and then play it back in digital form using a variety of devices.
Microsoft leaned hard into this vision of the future, not just a little because
it strongly implied multiple licensed copies of Windows being involved in the
modern home stereo. A significant development effort that entailed the addition
of multiple new operating system features lead to Windows XP Media Center
Edition, or MCE, released in 2002. MCE was a really impressive piece of
software for the time. An MCE computer wasn't just a computer, it was a "home
media hub" capable of consuming media from multiple formats, storing it, and
then streaming it over the network to multiple devices. MCE introduced
consumers to significant expansions of the role of the "computer" in media.
For example, MCE supported playback and recording of television from cable and
multiple OEMs sold MCE desktops with preinstalled cable tuners. An XP MCE
machine could compete with Tivo, but with distributed playback which Tivo would
not introduce until later.
Microsoft leveraged their newfound place in the home theater, the Xbox, to
create an ecosystem around the platform. First-gen Xbox consoles with a
purchased upgrade and all Xbox 360 consoles could function as "media center
extenders," which entailed opening an RDP session to the MCE machine to present
the MCE interface on the Xbox. Extensions to RDP implemented to support this
feature, namely efficient streaming of HD video over RDP without re-encoding,
went on to drive the "network projector" functionality between Windows Vista
and various Ethernet-enabled projectors. This was arguably the primary
precursor to the modern ChromeCast as RDP introduced some (but not nearly all)
of the offloading to the streaming target that ChromeCast relies on. In fact
the Media Center Extender and Network Projectors are closely related both
technically and in that they have both faded into obscurity. Modern "network
projectors" typically rely on the simpler, AirPlay-like and Microsoft-backed
but open Miracast protocol [3].
Because Media Center Extender relied on RDP, it essentially required that the
Extender implement a good portion of the Windows graphics stack (remember that
RDP is basically a standardization of Citrix's early application streaming
work, which made a lot of assumptions about a Windows client connecting to
Citrix/Windows server environment and offloaded most of the drawing to the
client). This was no problem for the Xbox, which ran Windows, but the
improvements to RDP and open standards that make non-Windows RDP clients
practical today were not yet complete in the '00s and it was unreasonable to
expect conventional consumer A/V manufacturers to implement the Extender
functionality. Microsoft didn't make anything near a full line of home media
products, though, and so it needed some kind of Ecosystem.
Conveniently, Intel was invested in the exact same issue, having begun to
introduce a set of Intel-branded media streaming solutions (including WiDi, a
true unicorn to see in the wild, which allowed certain Intel WiFi adapters to
stream video to a television well before this was a common consumer
capability). Because media streaming required fast I/O and network adapters,
both Intel offerings, it had real potential to expand consumer interest in
Intel's then commercial products. Intel kicked off, and recruited Microsoft and
consumer A/V giant Sony into, a new organization called the Digital Living
Network Alliance. That's DLNA.
DLNA built on UPnP, the then-leading consumer network auto-discovery and
autoconfiguration protocol, to define a set of network standards for the new
integrated home media network. DLNA defined how a set of devices broadly
categorized as Media Servers, Media Players, and Media Renderers would discover
each other and exchange signaling in order to establish various types of
real-time media streams and perform remote control. As a broad overview, a
Media Server offered one or more types of stored (e.g. files on disk) or
real-time (e.g. cable tuner) media to the network using established protocols
like RTSP. A Media Player allowed a user to browse the contents of any
available Media Servers and control playback. A Media Renderer received media
from the Media Server and decoded it for playback.
The separation of Media Player and Media Renderer may seem a bit odd, and in
practice they were usually both features of the same product, but enabled
network-based remote control of the type we see today with Spotify Connect.
That is, you could launch a Media Player on your computer (say Windows Media
Player) and use it to play audio on the Media Renderer integrated into your
stereo receiver. Other Media Players could discover the renderer and control
its playback as well. This was all possible in Windows XP, although it was very
seldom used because of the paucity of Media Renderers.
The problem is not necessarily that DLNA failed to spawn an ecosystem. While
Microsoft added pretty complete DLNA support to Windows through MCE, Windows
Media Player, and Explorer (where it became intertwined with the Homegroup
peer-to-peer network sharing system), JRiver and TwonkyMedia were major
third-party commercial DLNA Media Servers. On the open source side, Xbox Media
Center (XBMC), now known as Kodi, had fairly strong DLNA support. Plex, still a
fairly popular home media system today, originated as a port and enhancement of
XBMC and went on to gain even more complete DLNA support. Even Windows Media
Center itself was extensible by third-parties, providing a public API to add
applications and features to the MCE interface.
Years ago, around 2008, I successfully interoperated TwonkyMedia, Plex, Windows
Media Player, and a Logitech hardware renderer all via DLNA. This has become
more difficult over time rather than less, as the market for DLNA-capable
hardware has thinned and DLNA software support has fallen out of maintenance.
One major challenge that DLNA faced was the low level of consumer readiness for
fully digital home media. It's an obvious problem that consumers weren't yet
used to the idea of "smart TVs" and other devices that would integrate DLNA
clients. DLNA clients were somewhat common in devices like HD-DVD and Blu-Ray
players (these already needed a pretty significant software stack so adding a
DLNA client wasn't much extra effort), but it tended to be a second-class
feature that was minimally marketed and usually also minimally implemented. I
remember a particular LG Blu-Ray player that had a mostly feature-complete DLNA
Media Player and Media Renderer but struggled to actually play anything because
somewhere between its chipset and 802.11g, network streaming performance was
too poor to keep up with 1080p content.
And 802.11g is part of the problem as well. Home networks were surprisingly bad
in the early to mid '00s. I suppose it shouldn't be that surprising, because
"home network" was largely a new idea in that time period that was displacing a
single computer directly connected to a modem. Broadband was taking over in
cities, but DOCSIS and DSL modems still provided a USB interface and it was not
uncommon for people to use it. Almost no one ran ethernet because of the high
cost of installing it concealed and the frustration and aesthetic impact of
running it along floorboards [4]. 802.11g, the dominant WiFi standard, was
nominally fast enough for all kinds of media streaming but in practice
congestion and range issues made 802.11g performance pretty terrible most of
the time. All of this is still mostly true today, but we've gone from nominal
WiFi speeds of 54mbps to well over a gigabit, which allows for a solid 20mbps
even after the very substantial performance reduction in most real
environments.
The biggest problem, though, is the troublesome concept of a "server" in the
home. Microsoft seemed to operate, at the time, on the assumption that a
desktop computer would serve as the Media Server. In fact, MCE specifically
introduced power management enhancements to Windows XP intended to allow the
MCE machine to stay in standby mode but wake when needed for media recording or
streaming (this never seems to have worked very well). Microsoft further
reinforced this concept with their marketing strategy for MCE, which was only
released to developers and OEMs and could not be purchased as a regular
standalone license. If you were going to use MCE, it was going to be on a
desktop computer sold to serve as a hybrid workstation and media server.
Unfortunately, around the same time period the household desktop started to
become a less common fixture. Lower prices and better performance from laptops
could free up a desk and add a lot of flexibility, and consumers were just
getting less excited about buying a mid-tower. And besides, the hard drive
capacities of the time made a normal desktop somewhat limited as a media
server. Remember, this was back in that awkward period of physical media where
it was common for desktops to have two optical drives because people wanted a
DVD player and a CD burner, and no one had figured out how to economically fit
both into one device yet. Hard drives were usually in the hundreds of GB only.
What was needed, it seemed, was some sort of home media server appliance that
was compact, affordable, and user-friendly enough that people would consider it
a typical network appliance like a router/AP combo. That's sort of what Apple
delivered in the AirPort Time Capsule (and then later kind of in the Apple TV,
it got confusing), and pretty much what we would call a consumer NAS today
(like a Drobo or QNAP or something). Back in the mid '00s neither of these
options were available on the market, and most people weren't going to watch
the episode of Screen Savers about connecting a JBOD to a desktop on the cheap.
So, in 2007, Microsoft served up a real wonder: Windows Home Server.
Windows Home Server was a stripped down (although surprisingly little) version
of Windows Server 2003 R2, intended to be easily set up and managed by a
consumer using a desktop client called the Windows Home Server Console. The
Console was a bit like the MMC in principle, but with fewer features and more
wizards. Although I'm having a hard time confirming this, I believe it actually
worked via RDP using the Virtual App technology, meaning that the actual
Console ran on the server and only the GUI drawing occurred on the client.
This probably eased implementation but had the result that you could only
really manage Windows Server from Windows, which did not help at all against
Apple's widely perceived media dominance.
Not a lot of Windows Home Server devices made it to the market. HP released
several, one of which I used to own. The larger HP Home Servers were pretty
similar to a modern consumer NAS, with multiple front-accessible drive bays.
Home Server functioned (perhaps first and foremost) as a DLNA server, but also
offered a number of other services like backup, brokering RDP connections from
the internet for remote desktop, and a web-based file browser to get your files
remotely---naturally tied into .NET Passport. Home Server was extensible and
third-party software could be installed from a small app store and managed from
the Console; both Plex and Twonky offered Home Server versions at the time.
Several commercial antivirus packages were available as well, as this was a bit
before Microsoft took antivirus and HIDS on as a first-party component, so
Norton and McAffee could still make a lot of money off of getting you to pay
for one more license. In fact the Home Server versions of these antivirus
products were interesting because they often functioned more like a miniature
enterprise antivirus/HIDS platform, centrally controlling and monitoring your
host-based security products from the Home Server.
Windows Server also introduced a significant new feature called Drive Extender,
which was a high-level file system feature that allowed the user to combine
multiple drives into one logical drive and use multiple drives for redundancy,
all in a way which was largely agnostic to the drives and interfaces in use.
This was presumably a response to the high cost of the hardware RAID
controllers that served the same purpose in most "real" Windows Server
machines, but it compared favorably with the software-first approach to storage
management that would shortly after spread through the POSIX world in the forms
of ZFS and Btrfs. Ironically Drive Extender was a source of a lot of
frustration and data loss as it turned out to buggy, which is perhaps why
Microsoft ignominiously killed the feature along with Windows Server. Years
later later it would reappear, seemingly in a much reworked form, under the
name Storage Spaces.
And what would you guess happened with Windows Home Server? That's right, it
failed to gain traction and slowly faded away. Windows Home Server 2011 did
make it to the light of day but proved to be the last version of the concept.
Ultimately I think it was a victim of Microsoft's usual failures: it addressed
a new use-case rather than an existing one, so it wasn't something that
consumers had much interest in to begin with. Microsoft massively failed at
creating a marketing campaign that would convince consumers otherwise. The cost
of Home Server devices was pretty high (for much the same reasons that consumer
NAS continue to be rather expensive today), and at the end of the day it just
kind of sucked. In a fashion very typical of Microsoft, it had a lot of
interesting and innovative features but the user experience and general level
of polish were surprisingly poor. The Console, I remember, was sometimes nearly
impossible to get to connect without rebooting the server. The Drive Extender
faults lead to a lot of instances of data loss which then lead to bad press.
It formed a part of Microsoft's generally confusing and poorly designed home
sharing experience, later Homegroups, and so got tangled up in all of the
uncertainty and poor usability of those features (for example to easily get
access to the server's storage via SMB). The whole thing was a failure to
launch.
And that sort of sums up the fate of DLNA as well: despite the best efforts of
its promoters, DLNA never really got to a position where it was attractive to
consumers. It was mostly intended to solve a problem that consumers didn't yet
have (access to their non-existent local media library and their non-existent
computer cable tuner [5]), and time has shown would never really have. The
ecosystem of DLNA products, outside of Windows, was never that large. Dedicated
media renderers never gained consumer adoption, and devices that threw in DLNA
as a value-add (like a lot of HD-DVD players) did a bad job and didn't promote
it. DLNA was complex and a lot of implementations didn't work all that well on
home networks, which was also true of related technologies (like SMB file
sharing) that were important parts of the whole home network ecosystem.
Moreover, DLNA was wiped out by two trends: the cloud, and proprietary casting
systems.
First, the cloud: the idea that consumers would have a large local library of
music, TV shows, and movies, has never really materialized. The number of
people who have a multi-terabyte local media collection is vanishingly small
and they are basically all prolific pirates, which makes the broader media
industry uninterested in keeping them happy. In fact much of the media industry
worked to actively discourage this kind of use pattern, because it is difficult
to monetize in a reliable way.
Instead, most consumers get their media from a cloud-based streaming service
like Spotify, which has no requirement for any particular features on the local
network. Since these types of services already need significant cloud support,
it becomes easier to implement features like casting and network remote control
within the product itself, without any need (or support) for open standards.
No one needs DLNA because they use Spotify, and Spotify has worked commercial
partnerships to get Spotify Cast support in their A/V devices.
Second, proprietary casting systems: one of the features but also, in
hindsight, defects of DLNA was its underlying assumption of a peer-to-peer
"matrix" system in which many devices interacted with many devices in a
flexible way. In practice, 90% of consumer use cases can be solved by a much
simpler system in which a media renderer operates under direct remote control
(and perhaps receiving media directly from) of a computer or phone. Miracast,
for example, operated in this fashion and while it never became that common it
was much more practical to integrate Miracast support into a device like a TV
than a DLNA renderer and player.
Moreover, casting features offer a compelling opportunity for vendor lock-in,
since it is natural to integrate them with operating systems and specific
applications. While Microsoft made some effort to promote the Miracast standard
it was lackluster at best, so the whole space was dominated by Apple (which had
a head start in the form of iTune's remote playback capabilities and massive
leverage by integrating the feature into iOS) and Google (which mostly won by
making the Chromecast extremely cheap, although leveraging the YouTube app was
also a major boon). Neither of these companies have much interest in
facilitating a multi-vendor ecosystem, and in the case of consumer A/V devices
where they have little strength they operated for closed partnerships with
device manufacturers over open standards. Along with commercial incentives the
scheme works: my TV supports the closed standards AirPlay and Chromecast, but
ironically not the open standard Miracast. The only device I have with
integrated Miracast support is a no-brand sub-$200 DLP projector, where it
works very well.
DLNA formally dissolved in 2017, although the writing was on the wall as early
as 2010 when Spotify began to transform the way music was consumed. Similar
capabilities are now found in various vendor-proprietary systems, but few of
these approach the original ambitions of DLNA. A huge industry preference for
cloud-intermediated platforms and increasing consolidation of home media onto
one of a small number of walled gardens makes any serious resurgence of a
DLNA-like project unlikely.
And Sonos? I don't want to be too mean to Sonos, the technology is pretty cool.
I just have a hard time dropping $500 on a speaker that handles the unsolved
problems of 2008 but not the solved ones. These newer distributed media systems
(also Yamaha Musicast, for example) are impressive but fail to provide the
flexible, multi-source matrix architecture that DLNA had once put within reach.
At the end of the day, who killed DLNA? In some ways DLNA was ahead of its
time, as it required better home networks than most people had and enabled a
media consumption pattern that was the future, not the present. Of course it
was ahead of a time that turned out not to exist, as the cloud streaming
services took over. In this way, technological progress (or more cynically the
twisted economics of the cloud) killed DLNA. Microsoft, and to a lesser extent
Intel, killed DLNA through poor marketing, few partnerships, and repeatedly
bungled products. Apple and Google killed DLNA by seeing a simpler and more
commercially advantageous solution and making it widespread (and we can't blame
them for this too much, as AirPlay and ChromeCast really are plainly easier to
implement well than DLNA ever was).
And, you know, capitalism killed DLNA, because as an open-standards distributed
system its profit-making potential was always limited. The consumer A/V
industry that could have flocked to DLNA because it offered wide
interoperability instead flocked to the proprietary standards because they
offered money. Heavyweights Apple and Google were playing for keeps.
Microsoft's unusually generous dedication to open systems ultimately left them
holding the bag, and to this day Windows lacks a coherent media streaming
ecosystem.
Also frankly all the DLNA products pretty much sucked but hey, I'm running on
nostalgia.
[1] This approach can be used as an alternative to, or in combination with,
high-voltage audio systems which we have discussed before.
[2] Due to the surprisingly analogue CD mastering process and the use of
generous error correction and error tolerance in CD playback, it is
surprisingly common for it to not actually be possible to create an exact copy.
But the general point still stands.
[3] Miracast is kinda-sorta a subset of WiFi, being standardized by the WiFi
alliance, and builds on previous HD-media-over-WiFi efforts like Intel WiDi.
These were tremendously unsuccessful but are important precursors of ChromeCast
and AirPlay. A lot of devices, including all Windows machines, support Miracast
but it's pretty rare to see it used in practice as TV manufacturers have not
been enthusiastic about it (which is to say Microsoft has not incentivized
integrated Miracast in the way that Apple has incentivized integrated AirPlay).
[4] This is still basically true today, although structured wiring has made
some inroads in new, especially multi-family, construction. Of course I
continue to meet people who live in a home or apartment with structured
ethernet and CATV who don't even realize it or don't understand how to use it.
Consumer interest in plugging cables into things remains low.
[5] In fact the whole DLNA story is tied up in the CableCard story, which I'll
probably write about in the future. The short version is that the whole idea of
using an arbitrary tuner for a cable subscription was unpopular with cable
carriers and hard to implement. Cable carriers preferred to provide their own
set-top boxes and DVRs, which consumers were mostly happy with. The concept of
connecting your cable, or even TV antenna, to a computer just never really
went anywhere.
I recently came across an article
about New Mexico's main PBS affiliate, KNME, starting a trial project to
"datacast" school materials to rural homes. One of the most interesting
takeaways, to me, is the fact that KNME's traditional broadcast radio network
of one 250kW primary transmitter and many low-power translators
is estimated to reach 98% of New Mexico homes, a significantly better
level of penetration than broadband internet... despite the telephone
industry's early history of strong rural penetration.
Even now in the 21st century, traditional broadcast radio is the most effective
way to deliver information to a large area at a low cost. While digital audio
has never really caught on in the US (I am oddly proud of my HD radio tuners)
like it has in Europe, digital television has been the norm for some time.
Over-the-air (OTA) digital television in the US is based on a standard called
ATSC [1]. ATSC OTA is ultimately just a 19.4 Mbps data stream consisting of a
series of MPEG transport stream (TS) packets. Because it's intended for use
with unreliable links like broadcast radio, MPEG TS is a essentially a
uni-directional network protocol with structural similarities to other
packetized radio protocols.
AV containers like MPEG TS are usually associated with a single program
consisting of a video stream and an audio stream, although it's not too unusual
to have multiple audio streams (e.g. for multiple languages) and many
containers support multiple video streams for the purpose of multiple angles
(this was a "killer feature" of the DVD that exactly no one ever used) [2].
MPEG TS takes this to the next level by permitting an arbitrary number of
streams, identified by a program table. This feature is best known for allowing
a single ATSC "channel" to actually carry multiple channels, leading to the
confusing world of OTA channel numbering. All of the programs carried on a
given ATSC channel must fit into the 19.4 Mbps, so lower-budget ATSC channels
tend to actually look worse as they're compressed more heavily to fit more
channels onto a shared transmitter.
Moreover, MPEG TS allows for streams carrying non-media metadata typically in
the form of "tables." While tables were originally intended to carry
program-related metadata for use by decoders, "private" table IDs exist that
allow you to shove basically anything you want into an MPEG stream. And we have
now looped back to IP-over-MPEG, something I have touched on before, which just
means cramming IP packets into an MPEG TS table alongside media.
And now perhaps you see how this "datacasting" works: the television station
adds a new stream to their MPEG TS transmission that sends a "private table."
The private table is actually decoded to whatever format the datacasting
receiver expects, which is very possibly UDP but might also be a purpose-built
lightweight transport protocol for files with metadata. It could also just be
like a series of ZIP files with a little framing bolted on, there are a lot of
options.
The client device uses an ATSC tuner to extract the special data table and then
decodes it into files, which it stores on an SD card. It has a local webserver
that allows users to browse the files it has accumulated at their leisure.
Since the total MPEG TS stream is nearly 20Mbps, there's a decent amount of
room to add in this kind of secondary use. KNME transmits five television
channels, but three are standard definition, so there may be a good 5Mbps
available for data.
This is all a pretty interesting use of existing broadcast television
infrastructure, and I appreciate that it echos a tradition of radio technology
being pioneered for New Mexico education (which, incidentally, is the NME in
the vanity call sign KNME). KANW, the first FM radio station in New Mexico, was
founded by Albuquerque Public Schools for education delivery. APS also helped
found KNME, which was not the first OTA television station in New Mexico but
was the first to upgrade to digital. APS continues to part-own both KANW and
KNME (in partnership with a community college and state university,
respectively), enjoying the FCC's generous approach to licensing educational
institutions.
A combination of the rural-access and educational mission of many
academically-owned public radio stations, along with the FCC's ongoing support
for educational broadcasting, has had the somewhat surprising result that
educational broadcasters have long been technical pioneers. Commercial
broadcasters are increasingly losing interest in OTA, especially for
television, creating the modern situation where a school district and a
university partner to reach 98% of the state while an affiliate of the
once-great American Broadcasting Company runs at a reduced power that barely
covers the city. But that's all a tangent.
What really drew my attention to this KNME project is the mere utterance of the
term "datacasting," a perennial business plan in telecommunications that has
seldom found long-term success. Datacasting, in general, refers to any
broadcast of data other than audio/video media.
Datacasting sometimes takes the form of a dedicated system for one-way delivery
to multiple points. One could argue that backhaul systems for media networks
(e.g. the WestLink satellite distribution system for public television shows,
incidentally operated by KNME out of Albuquerque) are datacasting systems,
since they transmit media that is not intended to be directly viewed. These
systems also increasingly use non-media transports like IP to allow for
delivery of syndicated shows at faster-than-real-time speeds. That might be
getting a little pedantic, though.
A clearer case of datacasting is SkyCache, later called Cidera and now defunct.
From 1997 to 2003, SkyCache operated a satellite network that delivered content
to internet service providers for caching, as a means of reducing their actual
bandwidth content. The university where I attended my undergraduate had used
SkyCache for a one-way NNTP feed in the late '90s, in order to relieve their
SONET internet uplink from the rather substantial bandwidth required by Usenet
at the time [3].
Under the Cidera name, SkyCache made an attempt to pivot towards delivery of
video to streaming media edge sites. This worked about as well as you might
imagine considering the rapid decrease in the cost of bandwidth in the early
2000s, and Cidera quickly found themselves obsoleted by the very internet
connections that they were trying to accelerate. Similar concepts seem to have
had a longer endurance in other countries, but I can't find anyone attempting
this business model today... it would come off as a bit ludicrous considering
that the cost of satellite transit is well-known to be higher than terrestrial
IP transit, which is really a testament to how cheap IP transit has become.
More common than dedicated datacasting systems were those that embedded
datacasting as a secondary use of traditional media broadcasts. I had a vague
recollection of having once owned a GPS navigation device that employed a
secondary data broadcast on an FM radio station to obtain live traffic
information. Thanks to a Twitter follower I just learned that this was most
likely DirectBand, which Microsoft operated vaguely as part of the sprawling
MSN product family [4]. DirectBand made use of some space above the stereo
component of an FM radio signal, and did a fairly high rate of 12 kbps with
error correction. Evidently it also supported applications like headlines,
stock quotes, and sports scores, which I never had the luck to see implemented.
The same Twitter follower pointed out that UK regulator Ofcom still has a
license out for exactly this kind of service. It's held by a company called
INRIX that seems to have once been a contender in the GPS navigation space but,
like basically all the legacy GPS companies crushed by Apple and Google, is now
attempting to make a pivot to enterprise mobility and GIS service. Some
research into this lead to reading this sentence:
The HA and the Network Information Systems Limited ("NISL") selected INRIX as the primary supplier of data services (data-as-a-service, or "DaaS").
So it's good to know that UK government contracting is as "aaS-obsessed" as my
own United States. There was a time we'd just call that, you know, a service
contract. Just a service as a service. Like normal.
Datacasting has a similar history in the television space. In fact, PBS
affiliates are particularly known for datacasting. Through the company National
Datacast, PBS offered for-hire datacasting over its portfolio of analog TV
affiliates. The defunct service MovieBeam took a page from Cidera's book by
using National Datacast to send out HD films that set-top boxes stored for
later viewing, it was not at all successful.
In fact, the PBS history goes back further than this to an interesting metadata
application. Since some time in the '90s, many PBS member stations transmitted
a timestamp encoded into their analog signal. It was intended to allow VCRs to
set their clocks automatically, but it doesn't seem to have been especially
commonly used and by the '00s the infrastructure had fallen into a state of
disrepair such that many PBS stations were actually sending incorrect
timestamps. Automatic VCR clock incorrecting, if you will.
Last in a tour of datacasting applications, I'll mention pagers once again.
Because pager transmitters simply broadcast short packets with an address
header, it's easy to turn any paging system into a sort of low-bandwidth
datacasting network. Although most of these applications have been replaced by
cellular data, it used to be pretty common for things transit station arrival
time displays to just be very large pagers. They received their info updates
over either a pager transmitter owned by the municipality or a commercial pager
network. Pager protocols are still sometimes used this way for applications
ranging from public information signage to triggering tornado sirens. You could
debate whether or not this truly constitutes datacasting, but a company called
Ambient Devices attempted to commercialize the network in a sort of proto-IoT
way.
They did not succeed, because the internet took over IoT as well.
[1] People sometimes confuse this with ClearQAM, but ClearQAM refers to
unencrypted digital television over coaxial cable - 256QAM encoding does not
perform well at long distances OTA, so OTA broadcasts use 8VSB instead.
ClearQAM is not especially common because modern cable television providers
rely on encryption, rather than physical disconnection, to enforce payment.
[2] There's a popular idea that the multi-angle feature was used primarily by
pornography. Smut on DVDs being even more of a historic artifact than DVDs in
general, I'm having a hard time finding a good source that multi-angle
pornographic DVDs were actually common. Certainly there were a few, but there
were also a few actual multi-angle DVDs, like Die Hard. Part of my skepticism
of this comes from the very popular "porn killed betamax" story that has never
actually stood up to scrutiny; I think the salaciousness of adult films and the
lack of well-established ratings agencies for them creates a lot of urban myths
about the role of adult entertainment in broader market trends.
[3] A few years earlier, computer center policy had forbidden use of usenet
during business hours as the NNTP server was also a workstation in the computer
lab and became unusably slow for the person sitting in front of it if there was
too much usenet activity. I am indebted to the late John Shipman in many
important ways, but also for having preserved decades of computer center
bulletins full of gems like this.
[4] DirectBand is not to be confused with RBDS, which is in common use in the
US to this day but is so slow as to only really be useful for delivery of the
title of the currently playing song... which not that many radio stations
actually implement correctly. RBDS is sort of infamously poorly implemented; in
theory it should allow car radios to set their clocks automatically but so few
radio stations have an RBDS time encoder that it's never actually worked for
me. A surprising number of FM stations here transmit RBDS and use it for...
their call letters, over and over. Presumably they installed the modulator and
then never connected anything to it.
One of the best-known brands in American burglar alarm systems is ADT. ADT
stands, of course, for American District Telegraph. As the name suggests, ADT
did not originate as a burglar alarm company. Instead, it started as a
telegraph operation, mostly providing stock quotes and other financial
information within cities. As happened with many early telegraph companies, ADT
became part of Western Union and later part of AT&T. The history of ADT as a
telegraph company is not all that interesting, and their telegraphy business
is not well remembered today.
The modern ADT story starts, the company history tells us, with someone
breaking into the home of one of the founders of one of ADT's constituent
companies. This was the origin of the business line we know ADT for today:
burglar alarms. While a number of companies have entered and exited the
burglar alarm market (including AT&T itself as a separate venture from ADT),
none have gained the market dominance of ADT... dominance so great that they
lost an antitrust case in the '60s.
Before we get to the history of burglar alarm signaling, though, we should
discuss the burglar alarms themselves.
The fundamental concept of a burglar alarm is fairly simple, and might better
be explained by analogy to a similar, simpler system. Many industrial machines
make use of a "safety string" or "safety circuit." This consists of a set of
limit switches or other devices connected in series on things that are safety
critical, like access doors. It's all arranged such that, in a "normal" state,
the safety string is closed (all of the switches are in their circuit-closed
state). If any unsafe condition occurs, such as an access door being opened, a
switch opens the circuit. This usually cuts power to the potentially dangerous
equipment.
A burglar alarm is simply the extension of this concept to multiple circuits.
A set of switches and sensors are installed throughout the protected area. Any
of these sensors indicating an "unsecured" state causes an alarm.
In practice, the wiring of burglar alarms is more complex than that of a simple
safety string. There are various reasons for this, but the most obvious is the
problem of supervision. Burglar alarms are designed to operate in a somewhat
hostile environment including the possibility of tampering, and in any case
insurance companies usually require that they have an adequate "self-test"
facility to ensure that the system is functioning correctly during arming.
This means that the alarm system needs to have a way to determine whether or
not the sensor circuits are intact---whether the sensors are still connected.
This is referred to as supervision.
A very common supervision method is the end-of-line resistor. A common burglar
alarm circuit arrangement is the normally closed circuit with 5.6k EOL resistor.
In this system, the sensors are each connected to the alarm in series and are
normally closed. When a protected door is opened, for example, the sensor opens
the circuit which is detected by the alarm controller. An enterprising burglar,
though, might realize that they can short the two alarm wires together and thus
prevent the circuit ever opening. An EOL resistor complicates this attack by
creating a known, fixed resistance at the far end of the sensor circuit.
Let's imagine that the alarm controller functions by measuring the resistance
of each alarm circuit (they basically do, but usually instead by putting a
fixed current on the circuit and then measuring the voltage drop). If the
resistance of the circuit is infinite, it is "open," which indicates an alarm.
If the resistance of the circuit matches the expected EOL resistor (say 5.6k
but the value just depends on what the alarm manufacturer chose), everything is
normal. If the resistance of the circuit is zero (or near zero), the circuit
has been shorted... and that can result in either the alarm sounding or
reporting of a "trouble" condition.
This isn't the only way to use an EOL resistor. Another approach that is very
common with fire alarms is a normally open EOL resistor circuit. In this
system, the normal state is the fixed resistance of the EOL resistor, but the
"initiating devices" such as smoke detectors are connected between the two
alarm wires and actually short them together (creating a zero resistance) to
cause the alarm to sound. In this case, an infinite resistance (open circuit)
indicates that the circuit has been broken somewhere. In practice the
difference between normally-open and normally-closed alarm circuits is more one
of convention than technical merit, as both have very similar reliability
characteristics. Largely for historical reasons, burglar alarms tend to be
normally closed and fire alarms tend to be normally open, but there are many
exceptions to both.
In early burglar alarms, this basic concept of measuring the resistance of an
alarm circuit to check for a "normal" or "secure" value was very clear. A
common vault alarm in the early 20th century had a large gauge showing the
resistance of the single circuit and a knob which was used to adjust a variable
resistor. To arm the alarm, the knob was turned (adjusting a reference
resistance) until the needle fell into the green band, and then the alarm was
switched on. The needle leaving the green band in either direction (indicating
an open or a short circuit) resulted in an alarm, often by a mechanism as
simple as the needle touching a peg (but perhaps also by things like
spring-balanced magnetic coils). Modern burglar alarms are mostly just an
evolution of this same design, although digital communications are becoming
more common.
A burglar alarm usually uses several of these circuits, which are labeled
"zones." Zones could correspond to physical areas of a protected building (and
sometimes do), but for practical reasons it's very common for zones to
correspond more to type of sensor than location. For example, it is very common
to have one zone for perimeter detection devices (e.g. window and door
closures) and one zone for presence detection devices (e.g. motion detectors)
[1]. This is done so that different types of sensors can be armed separately,
most often to enable a "perimeter" or "home" or "stay" mode in which the
sensors on entry and exit points are armed but the sensors on the interior
space are not.
Besides the sensors themselves, another important part of a burglar alarm is a
means of arming and disarming it. Due to the lack of practical digital
technology, early alarms took a very mechanical approach. A common arrangement
in commercial buildings was a two-keyswitch system. To disarm the alarm, a key
switch on the outside of the building had to be turned to the disarm position.
Then, within a certain time period (measured by anything from a mechanical
timer to a thermistor and heater), a second keyswitch located somewhere
in the building had to be turned to the disarm position.
This somewhat complex system is actually very clever. It's quite possible that
a skilled burglar will pick, dismantle, or otherwise defeat the outside
keyswitch. The interior keyswitch serves as a "what you know" second factor: a
burglar, not being familiar with the building's alarm system, would probably
not know where the second switch was and would run out of time while looking
for it. To improve this mechanism, the alarm panel and second keyswitch (it was
usually right on the alarm panel) was often put somewhere non-obvious like a
closet off of an office or behind a painting.
This use of two different key switches gets at a fundamental concept in the
design of burglar alarm systems. The set of access points (like doors and
windows) monitored by the alarm delineates a boundary between the secured space
and the unsecured space. Alarm equipment within the secured space gets a
certain degree of protection against physical tampering by virtue of its
location: it is difficult to a burglar to tamper with something if they can't
get to it without setting off the alarm. On the other hand, devices in the
unsecured space, such as a keyswitch placed outside, are much more vulnerable
to tampering. These devices need some kind of additional protection, or need
to have their capabilities limited. The same problem exists with door locks and
all sorts of other security systems [2].
In other cases a much simpler and more manual approach was taken. In some early
alarm systems, there was no way to disarm the alarm. Instead, the employee
opening for the morning would contact the monitoring point and exchange
codewords (or otherwise identify themselves), so that the alarm center operator
knew to disregard the upcoming alarm. This is actually still a fairly common
practice both in military and other high security installations (where the
extra manual checks are worth the resources) and in multi-tenant, infrequent
access locations like radio towers where it would become frustrating to issue
alarm codes to all of the different contract technicians. You will sometimes
see signs along the lines of "Call 123-456-7890 before entering," which is a
good indicator that this approach is still in use [3].
By the '70s the "alarm code" or "alarm PIN" was becoming the dominant approach,
with users expected to disarm the alarm by entering a numeric combination.
Modern alarms still mostly rely on this method, although it's getting more
common to be able to arm and disarm alarms via the internet or a remote keyfob.
In both the cases of sensors and arm/disarm mechanisms we see that there has
not been a great deal of technical progress over the last decades. In general
the alarm space is somewhat stagnant, but it depends on the market. Consumer
systems are changing very quickly, but in some ways for the worse, as newer
consumer alarms are often cheaper and easier to install but also less reliable
and easier to tamper with than older systems. No small number of the "ease of
use" improvements in consumer alarms are directly achieved by reducing the
security of the system, usually in a way that consumers don't clearly
understand (more about this later).
Commercial alarm systems, on the other hand, have changed much less. Partially
this is because the market is small and somewhat saturated, but partly it is
because of the relatively higher security and certification requirements of
commercial insurance companies. For businesses, insurance discounts are usually
a bigger factor in the decision to install an alarm system, and the insurance
companies are usually stricter about requiring that the alarm be certified
against a specific standard. The good news is that these standards mean that
commercial alarms are usually built on all of the best practices of the '90s.
The downside is that the standards do not change quickly and so commercial
alarms do not necessarily integrate modern cryptography or other methods of
enhancing security. All in all, though, commercial alarms tend to be both more
antiquated and more secure than modern consumer alarms.
This post has really just been a quick introduction to the topic of burglar
alarms, giving you the basic idea that they function by monitoring strings of
sensors and provide some authenticating method of arming and disarming the
alarm. In the future, we'll talk a lot more about burglar alarms: about
central monitoring systems (remember ADT?) and about the new landscape of
DIY consumer systems, at least. Probably some rambling about different types
of motion detectors as well.
[1] There are a GREAT variety of different types of perimeter and presence
sensors, and they are all very interesting, at least to me. You may be
detecting that technical security is an interest of mine. In the future I will
probably write some posts about different types of burglar alarm sensors and
the historical evolution they have gone through.
[2] While not as relevant today, this is one of the reasons that alarm keypads
are usually placed inside the secured space. In older alarm systems, the keypad
sometimes directly armed and disarmed the alarm via a simple electrical
mechanism, making it possible to "hotwire" the alarm from the keypad. Placing
the keypad inside of the secured space, such that anyone accessing it would
have set off a sensor and started the entry delay timer, makes this kind of
exploit more difficult by putting the burglar on the clock. In well-designed
modern alarms the keypad no longer has the capability to disarm the alarm
without the user entering a code (i.e. the keypad sends the code to the alarm
panel elsewhere to be checked), but even today we can't take this for granted
as some manufacturers have "economized" by putting the entire controller into
the keypad.
[3] This is particularly common in telecom facilities because telecom companies
have an old habit of implementing a "poor-man's burglar alarm." They would
simply connect a switch on the door of the equipment hut to the trouble circuit
used to report conditions like low backup batteries and failed amplifiers.
That way any unauthorized access would result in a trouble ticket for someone
to go out and inspect the equipment. Of course, authorized access just meant
calling the maintenance office first to tell them to ignore the upcoming
trouble alarm.
The history of electric lighting is long and interesting, and I don't expect
to cover it in much depth or breadth because it's somewhat off of my usual
topic and not something I have a lot of expertise in. But I do have a
fascination with the particular case of large-area outdoor lighting, e.g.
street lighting, and there are a few things to talk about around street
lighting that are really rather interesting.
As a very compressed history, the first electric street lights usually took the
form of "moon towers," tall masts with arc lamps at the top. Various types of
open and contained arc lamps were experimented with early on, but generally
fell out of favor as incandescent lamps became cheaper and lower maintenance.
The only remaining moon towers in the US are in Austin, Texas. They were
expensive and high-maintenance, so they were fairly quickly replaced with
low-height incandescent in most applications. Later, mercury vapor and metal
halide arc lamps would begin to replace incandescent street lighting, but let's
stop for a while in the era of the incandescent street light [1].
Let's say that you were installing a large number of incandescent lights, for
example to light a street. How would you wire them?
In our homes, here in the United States, we wire our lights in parallel and
apply 120v to them. This has convenient properties: parallel wiring means that
the lights all function independently. A fixed voltage means that the
brightness of a given light can be adjusted by using bulbs of different
resistances, which will result in a different power (say, 60 watts). That makes
a lot of sense, which is why it may be surprising to some to learn that
incandescent street lights were wire in series.
Series wiring had multiple advantages, but the biggest was conductor size: in a
parallel-wired system, wiring near the power source would need to carry a very
large current (the combined current of all of the bulbs), and lights further
from the power source would be dimmer due to voltage drop unless the conductors
were prohibitively large. In a series-wired system, the current is the same
(and much lower, approximately equivalent to a single bulb) at all points in
the wiring, and voltage drop is seen across the entire length, and thus
consistent across all of the bulbs.
These street lighting circuits are referred to as constant current circuits,
as opposed to the more conventional constant voltage. All of the bulbs were
designed to operate at a specific current, 6.6A was typical, and a specially
equipped power supply transformer adjusted the voltage on the circuit to
achieve that 6.6A. The voltage would be fairly large, typically something like
5kV, but that wasn't a problem because the streetlight wiring ran overhead with
power distribution wiring of similar and higher voltages.
In early street lighting systems, the constant current regulator was a complex
mechanical device using magnetic coils and levers. Today, constant-current
incandescent lighting circuits are still in use in some applications and the
constant current regulators have been replaced with more robust electronically
controlled systems. But the regulators aren't really that important, except
to understand that they simply apply whatever voltage is required to get 6.6A
to flow.
Of course, connecting lighting in series introduces a major problem that you
have probably realized. If any bulb burns out, the entire series circuit will
be broken. That's unacceptable for street lights, and a few different solutions
were invented but the most common was a cut-out disk. The cut-out disk was a
small fuse-like device, but operated somewhat in the opposite way of a fuse.
In fact, they're sometimes amusingly referred to as antifuses.
The cut-out disk is wired in parallel with the bulb. Should the bulb burn out,
the voltage across the two sides of the disk rises, which causes an arc that
"burns out" a film material separating two contacts. The contacts then touch,
and electricity flows freely through the cut-out disk, restoring the circuit.
When the bulb is replaced, the cut-out disk is replaced as well with a fresh
one.
"Stay-Lit" Christmas light strings employ a very similar method, but use a
thermistor instead of a cut-out disk so that it is not an additional consumable
part. But the concept is the same: a device in the bulb base begins to pass
current if exposed to the full 120v power supply, instead of the few volts
normally seen when the bulb is present.
Constant-current lighting circuits have an interesting safety property, which
this discussion of cut-out disks may have brought to your mind. A short circuit
in a constant-current lighting circuit is actually pretty safe, as the
regulator will reduce the voltage to near zero to maintain only 6.6A. But, when
no current flows, the constant current regulator will increase the voltage
applied to the entire circuit in an attempt to restore current flow. Modern
electronic regulators mitigate this somewhat by detecting this condition, but
it gives constant-current lighting circuits a particularly sharp edge. An open
circuit can be rather dangerous as the regulator will increase the output
voltage to the upper limit---potentially something like 10kV. Everything on the
lighting circuit needs to be rated for this maximum voltage, which leads to
more demanding requirements than typical 120v or 240v commercial lighting.
With that preamble on the concept of constant-current street lighting, let's
take a look at a particularly interesting bit of history that closely relates
to these technical details [2].
On July 9th, 1962, the United States conducted its 7th high-altitude nuclear
test. While an earlier series of high-altitude demonstrations had shown the
tactical potential of detonations far from the ground, significant advances had
been made in the science and instrumentation in the intervening years, and
greater resources had become available for geopolitical reasons (resumed Soviet
testing). The new test, named Starfish Prime, was thus expected to contribute
significant information on the effects of high-altitude detonations.
The approximately 1.5Mt detonation occurred at about 400 km altitude, well into
space. It was expected that a detonation at such a high altitude would create a
substantial electromagnetic pulse effect, since the "horizon" was far from the
detonation allowing wide coverage and less cancellation by ground reflections.
Prior to the Starfish Prime test, however, EMP had not typically been viewed as
a major consideration in the use of nuclear weapons... while the effect clearly
existed, the magnitude was not such that it was likely to cause substantial
disruption [3].
Starfish Prime seemed to suggest otherwise. Effects that are out of scope for
our purposes resulted in damage to several US military satellites. Moreover,
unanticipated effects resulted in higher EMP field strengths at the ground than
had originally been estimated. This effect extended over a very large area,
including the nearly 1,500km straight-line distance from the detonation near
Johnston Atoll to Honolulu.
Various effects observed in Honolulu (from which the detonation was visible in
the sky) were attributed to the explosion. It can be difficult to sort out
which of these reports are accurate, as something as dramatic as a nuclear
detonation tends to get tied up with all kinds of unrelated events in people's
memory. One thing that was particularly widely reported, though, was
streetlights in Honolulu going dark at the instant of the detonation.
Was this actually an example of EMP having a significant effect on a civilian
power distribution system, as has long been theorized but seldom validated?
The answer, it turns out, is yes and no. Honolulu, like many jurisdictions in
the early '60s, made extensive use of series-wired street lighting circuits.
Also like many municipalities, Honolulu had gone through growth and changes
that resulted in various ad-hoc and sometimes confusing adjustments to the
electrical distribution system. Part of this had involved changes in
distribution voltage on existing lines, which created problems with the safety
separation distances required between lines of different voltages attached to
the same pole. The result was somewhat odd arrangements of lines on poles which
complicated the installation of street lighting circuits.
On some Honolulu streets, it had become impractical to install series-wired
street lighting circuits at medium voltage (6kV being fairly typical) and still
have adequate safety separation from 240v split secondary wiring supplying
power to homes. Honolulu adopted a solution of mixing series-wired high-voltage
and parallel-wired low-voltage street lighting. On residential streets with
crowded poles, street lights ran on 500v so that their lines could be run
directly alongside the residential power supply.
At this point in time, the modern norm of photocell switches installed on each
light did not yet exist. Instead, streetlights were turned on and off by
mechanical timer switches (or occasionally manual switches) attached to the
constant current regulators. So, the 500v lighting used on residential streets
still needed to be connected to the series-wired system in order to run off of
the same controls. The solution, which simplified the cabling in other ways as
well, was to power the 500v constant-voltage lighting off of the 6.6A
constant-current system. This was implemented by connecting autotransformers to
the series-wired system that reduced the voltage to 500v. The constant-current
regulator meant that the autotransformer could cause an arbitrary voltage drop,
at 6.6A, as necessary to provide adequate power to the 500v lights connected to
it.
Essentially, the autotransformer acted as a particularly large 6.6A light on
the series circuit. This made it subject to the same series disadvantage: if
the autotransformer failed, it would cause the entire series circuit to shut
off. The solution was to equip the autotransformer with a cut-out disk just
like the lights, albeit one rated for a higher voltage.
It was determined that the street lights which failed in response to the
Starfish Prime EMP all seem to have been the low-voltage lights attached to
constant-current circuits. Perhaps you can see what happened.
Some of the low-voltage lighting circuits were positioned such that their
length was oriented perpendicular to the EMP field. The altitude and location
of the detonation was such that the EMP field reached Honolulu with horizontal
polarization nearly parallel to the ground. The combination of these factors
created a near-worst-case for induced voltage in certain low-voltage lighting
circuits. The induced voltage backfed the autotransformer, resulting in a
higher voltage on its constant-current side that fused the cut-out disk,
shorting the autotransformer and cutting off the power supply to the
low-voltage lighting circuit.
The solution was simply to replace the cut-out disk.
This should illustrate two things about the effects of nuclear EMP: First, it
is possible for nuclear EMP to have damaging effects on power infrastructure.
Second, it is not especially easy for nuclear EMP to have damaging effects on
power infrastructure. The Honolulu incident resulted from a specific
combination of factors, and specifically from the use of a "weak-link" design
element in the form of the cut-out disks, which performed as expected in
response to an unusual situation.
None of this is to say that EMP effects are insubstantial, but they are mostly
confined to digital and especially telecommunications systems due to their far
higher sensitivity to induced voltages. Power systems with long enough wiring
runs to pick up substantial induced voltages also tend to be designed for very
high voltages, making damage unlikely. The same cannot be said of telephone
lines.
By the '60s series-wired constant-current lighting systems were falling out of
favor. The future was in mercury-vapor lamps running at 240v and controlled by
local photocells, so that they could be powered directly off of the secondary
distribution lines. This is still basically the arrangement used for street
lighting today, but mercury vapor has given way to LED.
Constant-current lighting still has its place. While airfield marker lighting
is mostly being converted to LED, it's still generally powered by
constant-current loops. Many airfields still use 6.6A tungsten bulbs, which
were at least perceived to have reliability and visibility advantages until
relatively recently. Airfield constant-current regulators usually provide three
(occasionally more) output settings with different target currents, allowing
for a low, medium, and high intensity. While less common with today's better
marker light optics, you will still sometimes hear pilots ask the tower (or use
radio control) to turn the lights down.
When it comes to LEDs, most non-trivial LED lighting is actually
constant-current. The use of a small solid-state constant-current regulator
(usually called an LED driver in this context) improves efficiency and lifespan
by keeping LEDs at an optimal current as temperature and supply voltage
changes.
Despite staying reasonably close to their '70s state, streetlights have become
hubs for technology projects. They're convenient, widely-distributed power
sources that are either owned by the city or operated on contract for the city.
Many "smart city" technologies like environmental sensors and WiFi/municipal
LTE/smart meter radios are offered in packages intended to be clamped directly
onto streetlights and powered by tapping their cables. The fairly standardized
screw socket used for photocell switches on street lights has itself become a
form of power outlet, with some smart city devices screwing in to use it as a
power feed and incidentally also turning the light on and off.
In this way the line between "light controller" and "sensor package" can become
blurry. The City of Albuquerque now installs networked controllers on new
street lights that primarily allow for remote monitoring and management of the
light itself---but also support environmental data collection and traffic
monitoring via Bluetooth sniffing.
The humble street light today reminds me a bit of the cigarette lighter socket
in cars... barely changed in decades, and yet a major factor in enabling a huge
technology market.
Or at least that's an optimistic way to look at it. The pessimistic way is to
observe that we now live in a world where streetlights are being repurposed to
detect gunshots. Cheery.
Addendum: I am not personally a fan of aggressive street lighting. It tends to
create a substantial light pollution problem that is thought to contribute to
health problems in humans and environmental disturbances. Further, it's been
the common wisdom for some time now that street lighting is likely not actually
effective in reducing crime. That said, a recent 2019 study conducted in New
York City is the first randomized controlled trial of the impact of additional
street lighting on crime, and it actually did find a reduction in crime, and
not a small one. That stands in opposition to a history of studies that have
not found any crime reduction, but none of those studies were as rigorously
designed. Hopefully more research will be conducted on this question.
[1] Incandescent street lights are typically the ones fondly remembered as
"historic" street lamps anyway, with many "antique style" streetlamps installed
in historic districts today being loose replicas of various common incandescent
models, such as the GE or Novalux "acorn" glass globes with frilly metal
reflector. That is, assuming they are replicas of electric and not gas
fixtures, although makers of vaguely historic light fixtures often mix the two
anyway.
[2] Much of the information on this incident comes from the aptly titled "Did
High-Altitude EMP Cause the Hawaiian Street Light Incident?", SAND88-3341.
Charles Vittitoe, Sandia National Laboratories, 1989. As far as I can tell this
paper, prepared well after the fact, is one of the only detailed case reports
of a real EMP incident in the public literature. The fact that there was,
reportedly, very little rigorous investigation of the Starfish Prime test's EMP
effects in Hawaii for over twenty years after has interesting implications on
how seriously EMP effects on civilian infrastructure were taken at the time.
[3] I discuss this topic a bit more in my YouTube video on EMP simulation
facilities at Kirtland Air Force Base.