_____                   _                  _____            _____       _ 
  |     |___ _____ ___ _ _| |_ ___ ___ ___   |  _  |___ ___   | __  |___ _| |
  |   --| . |     | . | | |  _| -_|  _|_ -|  |     |  _| -_|  | __ -| .'| . |
  |_____|___|_|_|_|  _|___|_| |___|_| |___|  |__|__|_| |___|  |_____|__,|___|
  a newsletter by |_| j. b. crawford               home archive subscribe rss

>>> 2021-10-25 datacasting (PDF)

I recently came across an article about New Mexico's main PBS affiliate, KNME, starting a trial project to "datacast" school materials to rural homes. One of the most interesting takeaways, to me, is the fact that KNME's traditional broadcast radio network of one 250kW primary transmitter and many low-power translators is estimated to reach 98% of New Mexico homes, a significantly better level of penetration than broadband internet... despite the telephone industry's early history of strong rural penetration.

Even now in the 21st century, traditional broadcast radio is the most effective way to deliver information to a large area at a low cost. While digital audio has never really caught on in the US (I am oddly proud of my HD radio tuners) like it has in Europe, digital television has been the norm for some time. Over-the-air (OTA) digital television in the US is based on a standard called ATSC [1]. ATSC OTA is ultimately just a 19.4 Mbps data stream consisting of a series of MPEG transport stream (TS) packets. Because it's intended for use with unreliable links like broadcast radio, MPEG TS is a essentially a uni-directional network protocol with structural similarities to other packetized radio protocols.

AV containers like MPEG TS are usually associated with a single program consisting of a video stream and an audio stream, although it's not too unusual to have multiple audio streams (e.g. for multiple languages) and many containers support multiple video streams for the purpose of multiple angles (this was a "killer feature" of the DVD that exactly no one ever used) [2]. MPEG TS takes this to the next level by permitting an arbitrary number of streams, identified by a program table. This feature is best known for allowing a single ATSC "channel" to actually carry multiple channels, leading to the confusing world of OTA channel numbering. All of the programs carried on a given ATSC channel must fit into the 19.4 Mbps, so lower-budget ATSC channels tend to actually look worse as they're compressed more heavily to fit more channels onto a shared transmitter.

Moreover, MPEG TS allows for streams carrying non-media metadata typically in the form of "tables." While tables were originally intended to carry program-related metadata for use by decoders, "private" table IDs exist that allow you to shove basically anything you want into an MPEG stream. And we have now looped back to IP-over-MPEG, something I have touched on before, which just means cramming IP packets into an MPEG TS table alongside media.

And now perhaps you see how this "datacasting" works: the television station adds a new stream to their MPEG TS transmission that sends a "private table." The private table is actually decoded to whatever format the datacasting receiver expects, which is very possibly UDP but might also be a purpose-built lightweight transport protocol for files with metadata. It could also just be like a series of ZIP files with a little framing bolted on, there are a lot of options.

The client device uses an ATSC tuner to extract the special data table and then decodes it into files, which it stores on an SD card. It has a local webserver that allows users to browse the files it has accumulated at their leisure. Since the total MPEG TS stream is nearly 20Mbps, there's a decent amount of room to add in this kind of secondary use. KNME transmits five television channels, but three are standard definition, so there may be a good 5Mbps available for data.

This is all a pretty interesting use of existing broadcast television infrastructure, and I appreciate that it echos a tradition of radio technology being pioneered for New Mexico education (which, incidentally, is the NME in the vanity call sign KNME). KANW, the first FM radio station in New Mexico, was founded by Albuquerque Public Schools for education delivery. APS also helped found KNME, which was not the first OTA television station in New Mexico but was the first to upgrade to digital. APS continues to part-own both KANW and KNME (in partnership with a community college and state university, respectively), enjoying the FCC's generous approach to licensing educational institutions.

A combination of the rural-access and educational mission of many academically-owned public radio stations, along with the FCC's ongoing support for educational broadcasting, has had the somewhat surprising result that educational broadcasters have long been technical pioneers. Commercial broadcasters are increasingly losing interest in OTA, especially for television, creating the modern situation where a school district and a university partner to reach 98% of the state while an affiliate of the once-great American Broadcasting Company runs at a reduced power that barely covers the city. But that's all a tangent.

What really drew my attention to this KNME project is the mere utterance of the term "datacasting," a perennial business plan in telecommunications that has seldom found long-term success. Datacasting, in general, refers to any broadcast of data other than audio/video media.

Datacasting sometimes takes the form of a dedicated system for one-way delivery to multiple points. One could argue that backhaul systems for media networks (e.g. the WestLink satellite distribution system for public television shows, incidentally operated by KNME out of Albuquerque) are datacasting systems, since they transmit media that is not intended to be directly viewed. These systems also increasingly use non-media transports like IP to allow for delivery of syndicated shows at faster-than-real-time speeds. That might be getting a little pedantic, though.

A clearer case of datacasting is SkyCache, later called Cidera and now defunct. From 1997 to 2003, SkyCache operated a satellite network that delivered content to internet service providers for caching, as a means of reducing their actual bandwidth content. The university where I attended my undergraduate had used SkyCache for a one-way NNTP feed in the late '90s, in order to relieve their SONET internet uplink from the rather substantial bandwidth required by Usenet at the time [3].

Under the Cidera name, SkyCache made an attempt to pivot towards delivery of video to streaming media edge sites. This worked about as well as you might imagine considering the rapid decrease in the cost of bandwidth in the early 2000s, and Cidera quickly found themselves obsoleted by the very internet connections that they were trying to accelerate. Similar concepts seem to have had a longer endurance in other countries, but I can't find anyone attempting this business model today... it would come off as a bit ludicrous considering that the cost of satellite transit is well-known to be higher than terrestrial IP transit, which is really a testament to how cheap IP transit has become.

More common than dedicated datacasting systems were those that embedded datacasting as a secondary use of traditional media broadcasts. I had a vague recollection of having once owned a GPS navigation device that employed a secondary data broadcast on an FM radio station to obtain live traffic information. Thanks to a Twitter follower I just learned that this was most likely DirectBand, which Microsoft operated vaguely as part of the sprawling MSN product family [4]. DirectBand made use of some space above the stereo component of an FM radio signal, and did a fairly high rate of 12 kbps with error correction. Evidently it also supported applications like headlines, stock quotes, and sports scores, which I never had the luck to see implemented.

The same Twitter follower pointed out that UK regulator Ofcom still has a license out for exactly this kind of service. It's held by a company called INRIX that seems to have once been a contender in the GPS navigation space but, like basically all the legacy GPS companies crushed by Apple and Google, is now attempting to make a pivot to enterprise mobility and GIS service. Some research into this lead to reading this sentence:

The HA and the Network Information Systems Limited ("NISL") selected INRIX as the primary supplier of data services (data-as-a-service, or "DaaS").

So it's good to know that UK government contracting is as "aaS-obsessed" as my own United States. There was a time we'd just call that, you know, a service contract. Just a service as a service. Like normal.

Datacasting has a similar history in the television space. In fact, PBS affiliates are particularly known for datacasting. Through the company National Datacast, PBS offered for-hire datacasting over its portfolio of analog TV affiliates. The defunct service MovieBeam took a page from Cidera's book by using National Datacast to send out HD films that set-top boxes stored for later viewing, it was not at all successful.

In fact, the PBS history goes back further than this to an interesting metadata application. Since some time in the '90s, many PBS member stations transmitted a timestamp encoded into their analog signal. It was intended to allow VCRs to set their clocks automatically, but it doesn't seem to have been especially commonly used and by the '00s the infrastructure had fallen into a state of disrepair such that many PBS stations were actually sending incorrect timestamps. Automatic VCR clock incorrecting, if you will.

Last in a tour of datacasting applications, I'll mention pagers once again. Because pager transmitters simply broadcast short packets with an address header, it's easy to turn any paging system into a sort of low-bandwidth datacasting network. Although most of these applications have been replaced by cellular data, it used to be pretty common for things transit station arrival time displays to just be very large pagers. They received their info updates over either a pager transmitter owned by the municipality or a commercial pager network. Pager protocols are still sometimes used this way for applications ranging from public information signage to triggering tornado sirens. You could debate whether or not this truly constitutes datacasting, but a company called Ambient Devices attempted to commercialize the network in a sort of proto-IoT way.

They did not succeed, because the internet took over IoT as well.

[1] People sometimes confuse this with ClearQAM, but ClearQAM refers to unencrypted digital television over coaxial cable - 256QAM encoding does not perform well at long distances OTA, so OTA broadcasts use 8VSB instead. ClearQAM is not especially common because modern cable television providers rely on encryption, rather than physical disconnection, to enforce payment.

[2] There's a popular idea that the multi-angle feature was used primarily by pornography. Smut on DVDs being even more of a historic artifact than DVDs in general, I'm having a hard time finding a good source that multi-angle pornographic DVDs were actually common. Certainly there were a few, but there were also a few actual multi-angle DVDs, like Die Hard. Part of my skepticism of this comes from the very popular "porn killed betamax" story that has never actually stood up to scrutiny; I think the salaciousness of adult films and the lack of well-established ratings agencies for them creates a lot of urban myths about the role of adult entertainment in broader market trends.

[3] A few years earlier, computer center policy had forbidden use of usenet during business hours as the NNTP server was also a workstation in the computer lab and became unusably slow for the person sitting in front of it if there was too much usenet activity. I am indebted to the late John Shipman in many important ways, but also for having preserved decades of computer center bulletins full of gems like this.

[4] DirectBand is not to be confused with RBDS, which is in common use in the US to this day but is so slow as to only really be useful for delivery of the title of the currently playing song... which not that many radio stations actually implement correctly. RBDS is sort of infamously poorly implemented; in theory it should allow car radios to set their clocks automatically but so few radio stations have an RBDS time encoder that it's never actually worked for me. A surprising number of FM stations here transmit RBDS and use it for... their call letters, over and over. Presumably they installed the modulator and then never connected anything to it.