Fixes#1383
This solves an edge case where the target path of an atomic write is
a relative path without a directory part.
In this case, the temporary file was written to the system's tempdir
before, which may then fail the atomic rename as the tempdir often
is on a different device such as tmpfs.
These are security sensitive dependencies that haven't been updated
since 2024.
Thankfully, there do not appear to be any major incompatibilities that
prevent an update.
libxml2 now enforces that buffer size must only refer to the actual
content excluding the null terminator, which required replacing
`std::size` with `strlen` in two unit tests.
I originally tried to bump mbedtls to 4.0.0, but that seems to be
blocked by an upstream issue trying to write to the cmake source
directory, which cmake errors on.
3.6.5 has all CVE fixes included that come with 4.0.0 though, so from a
security perspective it is sufficient to update to this version with no
incompatibilities first.
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
All deprecations have been addressed in this PR and Protobuf has been
updated to 33.5 as older versions lead to build failures on Linux with
this abseil-cpp version.
Another of our currently vendored patches for nixpkgs / NixOS that has
been in use for about two years, hidden behind an option to allow both
variants to coexist.
The idea is that opting into `USE_SYSTEM_DEPENDENCIES` works without any
submodules, so when this option is enabled, we also expect the system to
already have Python with the `google.protobuf` module ready and
available.
This was done by git clang-format v18. Results are the same up to v22.
Newer versions were not tested.
Commands:
```
FIRST_SHA1=58a19df
git clang-format --style Chromium $FIRST_SHA1
git commit -a
```
From now on, we won't have to trip over ancient style issues from other
parts of a file we're working on.
"Probability factor of one to one. We have normality. I repeat, we have
normality. Anything you still can't cope with is therefore your own
problem." ~Douglas Adams
Add clang-format off/on markers inside arrays that contain inline
structural comments (NAL unit sections, codec config fields, PSI tables,
encryption metadata, etc.) so their layout survives a clang-format run.
Arrays with no internal comments are left unguarded so clang-format can
reflow them freely.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
In nixpkgs, we derive the version from the package definition's version
attribute.
As we only fetch the tarball instead of the full git repository, trying
to derive the version from git fails.
We have been using this patch in nixpkgs since 2024, so upstreaming it
would be appreciated!
This PR removes the CLOSED-CAPTIONS attribute (and a few others) from
the EXT-X-I-FRAME-STREAM-INF tag in the HLS master playlist.
According to the HLS RFC (Section 4.4.6.3), the CLOSED-CAPTIONS
attribute is not defined for EXT-X-I-FRAME-STREAM-INF tags. Including it
causes non-compliance and may result in playback issues on strictly
compliant players.
References
[RFC 8216 Section
4.4.6.3](https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-16#section-4.4.6.3)
Affected version : >= 2.5
---------
Co-authored-by: Xavier Laffargue <xavier.laffargue@radio-canada.ca>
Current deps require an old CMake version, support for which has been
removed from latest CMake in Ubuntu and Arch.
```
CMake Error at packager/third_party/c-ares/source/CMakeLists.txt:3 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 3.5 has been removed from CMake.
```
This upgrades:
- abseil-cpp, 20240116.0 => 20260107.1
- c-ares, v1.20.1 => v1.34.5
- json, v3.10.1-115-g954b10ad => v3.12.0
- libwebm, 1.0.0.28 => 1.0.0.32
With these changes, the build has been verified with CMake 3.31.11
(latest v3) and CMake 4.2.3 (latest v4).
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
I've finally generated test material and tests to make a PR to fix text
segment generation for MPEG-2 TS streams with sparse teletext data,
making a big improvement compared to the original teletext support in
#1344.
## Problem
When packaging MPEG-TS input containing DVB-Teletext subtitles for
DASH/HLS output, two fundamental issues arise:
1. **Sparse input handling** - Teletext streams only contain data when
subtitles are displayed. During gaps (which can span multiple segments),
no PES packets arrive, leaving the text chunker with no timing
information to drive segment generation. This results in missing
segments or segments with incorrect timing.
2. **Misaligned segment boundaries** - Even when segments are generated,
the text segment timestamps and boundaries differ from video/audio
segments. This causes `<SegmentTimeline>` mismatches in the MPD,
playback issues on some players, and sometimes fewer text segments than
video segments.
## Solution
This PR introduces two complementary mechanisms:
### 1. Heartbeat mechanism (sparse input handling)
The `Mp2tMediaParser` now sends periodic "heartbeat" signals to text
streams:
- Video PTS timestamps are forwarded to all text PIDs as
`MediaHeartBeat` samples
- `EsParserTeletext` emits `TextHeartBeat` samples when PES packets
arrive without displayable content
- `TextChunker` uses these heartbeats to drive segment generation even
during gaps in subtitle content
- Ongoing cues that span segment boundaries are properly split and
continued
A new `heartbeat_shift` stream descriptor parameter (default: 2 seconds
at 90kHz) controls the timing offset between video PTS and text segment
generation, compensating for pipeline processing delays.
### 2. SegmentCoordinator (segment boundary alignment)
A new N-to-N media handler (`SegmentCoordinator`) ensures text segments
align precisely with video:
- Passes all streams through unchanged
- Replicates `SegmentInfo` from video/audio `ChunkingHandler` to
registered teletext streams
- `TextChunker` in "coordinator mode" uses received `SegmentInfo` events
to determine segment boundaries instead of calculating from text
timestamps
This guarantees identical segment timelines across all adaptation sets.
## Testing
- **Integration tests** in `packager_test.cc`:
- `TeletextSegmentAlignmentTest.VideoAndTextSegmentsAligned` - Verifies
segment count, start times, and durations match between video and text
-
`TeletextSegmentAlignmentTest.VideoAndTextSegmentsAlignedWithWrapAround`
- Same verification with PTS timestamps near the 33-bit wrap-around
point (~26.5 hours)
- **Test files** (synthetic teletext with known cue timings at 1.0s,
3.5s, 13.0s):
- `test_teletext_live.ts` - Normal PTS range
- `test_teletext_live_wrap.ts` - PTS near wrap-around boundary
- **Unit tests** for `SegmentCoordinator` and updated `TextChunker`
tests
## Documentation
- Extended `docs/source/tutorials/text.rst` with DVB-Teletext section
covering:
- Page numbering (3-digit cc_index format)
- Heartbeat mechanism explanation
- Segment alignment behavior
- `--ts_ttx_heartbeat_shift` parameter tuning
- Troubleshooting guide
- Added teletext processing pipeline diagram to `docs/source/design.rst`
## Future work
The heartbeat and `SegmentCoordinator` mechanisms would likely benefit
**DVB-SUB (bitmap subtitles)** as well (Issue #1477) , which faces
similar challenges with sparse subtitle data in MPEG-TS input and
segment alignment. The infrastructure is now in place to extend this
support.
## Example usage
```bash
packager \
--segment_duration 6 \
--mpd_output manifest.mpd \
'in=input.ts,stream=video,init_segment=video/init.mp4,segment_template=video/$Number$.m4s' \
'in=input.ts,stream=audio,init_segment=audio/init.mp4,segment_template=audio/$Number$.m4s' \
'in=input.ts,stream=text,cc_index=888,lang=en,init_segment=text/init.mp4,segment_template=text/$Number$.m4s,dash_only=1'
```
Fixes#1428Fixes#1401Fixes#1355Fixes#1430
This change introduces support for signaling CEA-608 and CEA-708 closed
captions in DASH. (Equivalent of
https://github.com/shaka-project/shaka-packager/pull/1532)
It reuses the existing --closed_captions command-line flag. Note that
some options (name, autoselect, and default) are not used for DASH.
```
--closed_captions 'channel=CC1,lang=fra;channel=CC2,lang=eng;channel=SERVICE1,lang=eng'
```
For Dash this adds an Accessibility tag in each video AdaptationSet.
Example output :
```
<AdaptationSet id="1" contentType="video" ....>
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-608:2015" value="CC1=fra"/>
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-608:2015" value="CC2=eng"/>
<Accessibility schemeIdUri="urn:scte:dash:cc:cea-708:2015" value="1=lang:eng"/>
```
HLS changes :
* We force the language to 2 characters to be consistent with the
language of the audio files, for example.
* closed_captions is now in packaging_params
**Description**
This PR introduces support for subsample encryption for Dolby AC-4 audio
streams including muxing and packaging in Shaka Pakager.
**Motivation**
Subsample encryption is a mode of Common Encryption (CENC) that allows
selective encryption of media samples, leaving specific portions (such
as headers) in the clear. In most cases, subsample encryption is applied
to video streams (e.g., H.265/HEVC or AV1). It can also be used for
audio streams, even though many pipelines default to full-sample
encryption for audio. For Dolby AC-4, this is particularly useful
because the AC-4 frame header (ac4_toc) contains configuration data that
must remain accessible to parsers and playback systems even when the
rest of the frame is encrypted.
By implementing subsample encryption for AC-4:
- Parsers can validate and process encrypted streams without full
decryption.
- Playback systems can perform certain bitstream operations (e.g.,
stream identification, sync validation) efficiently.
- This aligns with best practices for encrypted media delivery and
improves performance with streaming platforms.
**Implementation Details**
- The implementation ensures that the ac4_toc() portion of each AC-4
frame remains unencrypted.
- The remaining payload is encrypted using the selected CENC protection
scheme (e.g., cbcs or cenc).
**Specification Reference**
This feature is not mentioned in any published specification. However,
it is discussed in detail in the specification, which has not yet been
publicly released but is expected to be published in the near future.
Once available, it will provide formal guidance on subsample encryption
for AC-4, including the handling of the ac4_toc() structure and
encryption boundaries.
**Notes**
- This implementation is compatible with the upcoming public release of
the AC-4 ISOBMFF specification.
- The feature has been tested on encryption/decryption, DASH, HLS, and
DRM. The generated content is conforms to expected encryption behavior.
- Additional documentation will be provided once the specification is
officially published.
---------
Co-authored-by: Xingzhao Yun <xyun@dolby.com>
Implement the MPEG DASH/HLS specifications to switch event/live streams
to VOD when the streams end. Use --event_to_vod_on_end_of_stream.
This change implements the DASH standard by:
1. Change MPD@type to static
2. Set MPD@mediaPresentationDuration to the media duration
3. Removing dynamic specific attributes in element, e.g.
availabilityStartTime, minimumUpdatePeriod, timeShiftBufferDepth etc.
4. Set SegmentTemplate@presentationTimeOffset in Representations
This change implements the HLS standard by:
1. Switch the PLAYLIST-TYPE to VOD at the conclusion of a LIVE stream
2. Append an EXT-X-ENDLIST tag at the end of the stream
Fixes#321
Before, we would add clear ranges to the start of the subsample,
therefore moving the encrypted start further into the video frame; now
we add clear ranges to the end by adding a new subsample (if needed).
This seems more correct. Both are allowed by the spec, but it would be
better to encrypt the beginning of the frame to give attackers less info
about the start of the frame. Having the clear data at the end of the
frame doesn't give attackers much info since most data depends on the
encrypted state.
This change introduces support for signaling CEA-608 and CEA-708 closed
captions in HLS.
New command-line flags, `--closed_captions`, have been added to allow
users to specify the caption tracks. The format for these flags is a
comma-separated list of semicolon-separated key-value pairs, e.g.,
`--closed_captions
"channel=CC1,name=French,lang=fra,default=no,autoselect=yes;channel=SERVICE1,name=English,lang=eng,default=yes,autoselect=yes"`.
For HLS, this change adds `#EXT-X-MEDIA` tags for each caption track
with a common `GROUP-ID` of "cc". The `EXT-X-STREAM-INF` tag for video
streams has been updated to reference this `GROUP-ID`.
Dash will be supported in another PR.
Fixes#986
Fix incorrect error handling in `LocalFile::Open()` when
`create_directories()` encounters an already-existing directory.
### Problem
The current code treats `create_directories()` returning `false` as a
failure:
```cpp
if (!std::filesystem::create_directories(parent_path, ec)) {
return false; // BUG: ec is not checked
}
```
However, according to the C++ standard,
[std::filesystem::create_directories()](https://en.cppreference.com/w/cpp/filesystem/create_directory.html):
- Returns true if a new directory was created
- Returns false with ec.clear() if the directory already exists (not an
error)
- Returns false with ec set if an actual error occurred
This causes failures in several scenarios:
1. Concurrent packaging: Multiple threads creating the same output
directory
2. External directory creation: Another process creates the directory
between is_directory() check and create_directories()
call
3. Retry after partial failure: Directory was created in a previous
failed attempt
### Example Failure (strace)
```bash
newfstatat("/tmp/out/dir") = -1 ENOENT ← Directory doesn't exist
mkdirat("/tmp/out/dir", 0777) = -1 EEXIST ← Created by another
process/thread
exit(0) ← Packager exits without writing output
```
The directory exists and is ready to use, but the code incorrectly
treats this as a failure.
### Solution
Check the error_code after create_directories() returns false. Only
return failure if ec indicates an actual error.
```cpp
if (!std::filesystem::create_directories(parent_path, ec)) {
if (ec) { // Only fail if there's an actual error
return false;
}
// ec is empty means directory already exists - continue normally
}
```
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
Change EXT-X-TARGETDURATION to have the value set to the highest
duration for each variant instead of the highest duration for all the
variants.
Fixes#1510
WebM color range metadata was not being converted to MP4 colr box
format, causing incorrect color range value and video may look washed
out. Add colr box generation for VP8/VP9 codecs in WebMVideoClient.
Fixes#1515
Inspired by @r0ro
https://github.com/shaka-project/shaka-packager/pull/840
Add a new --add_program_date_time HLS flag to enable or disable this
feature.
Introduces support for the EXT-X-PROGRAM-DATE-TIME tag in HLS media
playlists, aligning with the HLS RFC specifications.
Closes#365
This upgrades libpng from v1.6.37 to v1.6.50
This newer libpng doesn't assume the existence of fp.h on Mac (which
isn't present on newer ones), but the CMake options for libpng changed
and had to be adjusted somewhat.
This adds missing cstdint headers in every header where we reference a
uint*_t.
This also adds a cmake definition that works around a guard in newer
cmakes that is meant to guard against libraries that only work in older
ones. (`-DCMAKE_POLICY_VERSION_MINIMUM=3.5`)
The `SimpleHlsNotifier::NotifyEncryptionUpdate` method was modified :
When this method is called with the Common System ID, it now checks if
the stream's encryption method is CENC. If it is CENC, the notifier
skips adding the EXT-X-KEY tag with KEYFORMAT="identity", as CENC
content should be handled by the specific DRM system's key format (e.g.,
Widevine's urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed).
Closes#1439
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
Adding MV-HEVC support limited to stereo video, issue #1483.
Stereo video coded in MV-HEVC is becoming more widely available: Apple
Vision Pro supports stereo video playback in MV-HEVC and both the
headset and iPhone supports capturing stereo video using the format.
FFmpeg has also added support for MV-HEVC.
Note that this PR is only focusing on adding MV-HEVC support to .mp4 for
encryption/decryption support. Proper HLS and DASH support will still
need to be added.
This adds support for Docker-based testing on arm64 images and runners.
This also adds a multiarch build process so that future releases cover
both x64 and arm64.
This is tagged as "feat" so that this gets called out in the changelog
for the next release.
We no longer need a separate repo for CMake.
This fixes build failures on OpenSUSE 15.5. The extra repo for CMake is
no longer available. So it's lucky we no longer need it with the latest
OS.
When uploading a live stream, if a single segment upload fails, we shouldn't give up.
This adds an option to ignore HTTP output failures so that a single failed upload does not result in a hung stream. See https://github.com/shaka-project/shaka-streamer/issues/195 for details.
These errors were specifically coming up in Windows builds and in Linux release builds. This initializes those variables.
This also fixes errors in IAMF tests like "offset 2 out of bounds", which some compilers raise when you try to statically initialize a vector, then add to it later.
Upgrading curl fixes errors like "Failed sending data to the peer" flooding the log. This is described upstream in https://github.com/curl/curl/issues/10591 and fixed in curl 8.2.0. Here we upgrade to curl 8.9.1 (latest as of today).
This required updating the way we attach c-ares to libcurl and updating CURLOPT_PUT (deprecated) to CURLOPT_UPLOAD (compatible equivalent AFAICT).
This embeds the full python protobuf library into the output in pssh-box-protos, which is used by installation targets and Docker images. Without this, the installed pssh-box.py tool would depend on a separate installation of the python protobuf library.
Closes#1436
As per issue #1418, several Dash roles are missing from the
RoleFromString function, making them unusable in practice. This update
should enable them for use.
---------
Co-authored-by: John Laurin <john.laurin@svt.se>
include/file.h is breaking header installation for the shared library build. macros/classes.h must be included to the public headers.
Closes#1406
Co-authored-by: Zaki Ahmed <zaki.ahmed.perso@gmail.com>
Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
Support Dolby Vision profile 8.1, 8.2, 8.4, 10.1, 10.4 signaling in HLS
and DASH.
Adds new option `--use_dovi_supplemental_codecs` (off by default) to use
SUPPLEMENTAL-CODECS in HLS and `scte214:supplementalCodecs` and
`scte214:supplementalProfiles` for DASH.
To maintain compatibility with existing players the current behavior of
using two entries in the manifest remains the default. This will be
changed in a future version where `use_dovi_supplemental_codecs` will
become on by default.
Adds Dolby Vision compatible brands, 'db1p', 'db2g', 'db4g', 'db4h',
'dby1' based on https://mp4ra.org/#/brands
---------
Co-authored-by: Xingzhao Yun <xyun@dolby.com>
If color_space is VPX_COLOR_SPACE_SRGB, the specs says that color_range
should be 1 i.e. yuv_full_range = true.
However, yuv_full_range was initialized as false and wasn't set in the branch for color_space
is VPX_COLOR_SPACE_SRGB.
Fixes#990
---------
Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
Currently `media_info.media_file_url()` is not escaped when placed into
MPD for things like BaseURL. This for example breaks when trying to us a
file name that contains special characters like &. Since these are
supposed to be URLs let's URL encode them.
Fixes#1107
---------
Co-authored-by: Joey Parrish <joeyparrish@users.noreply.github.com>
After change to add forced command line ordering adaptation set IDs in
places were referenced by their sort index (the minimum representation
index they contained).
Instead always refer to adaptation sets by their own ID, and use the
index only as an optional sort key.
Fixes#1393
Set the start number in representation to the segment index that is sent by muxer.
With this enhancement, you can now specify the initial sequence number
to be used on the generated segments when calling the packager.
With the old implementation, it was always starting with "1".
---------
Co-authored-by: Cosmin Stejerean <cstejerean@meta.com>
