It has been very, very annoying lately to have PRs get stuck in several
loops of linter failures without finding out if the thing builds or
passes tests. From now on, lint in parallel with everything else.
Current deps require an old CMake version, support for which has been
removed from latest CMake in Ubuntu and Arch.
```
CMake Error at packager/third_party/c-ares/source/CMakeLists.txt:3 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 3.5 has been removed from CMake.
```
This upgrades:
- abseil-cpp, 20240116.0 => 20260107.1
- c-ares, v1.20.1 => v1.34.5
- json, v3.10.1-115-g954b10ad => v3.12.0
- libwebm, 1.0.0.28 => 1.0.0.32
With these changes, the build has been verified with CMake 3.31.11
(latest v3) and CMake 4.2.3 (latest v4).
---------
Co-authored-by: Joey Parrish <joeyparrish@google.com>
This handles trusted publishing for Shaka Packager to NPM without
tokens.
This also updates upload/download actions, and adds a filter on specific
artifacts downloaded to avoid conflicts that have arisen since the last
release.
Also updates the images used in the build matrix.
Issue shaka-project/shaka-player#9132
This upgrades libpng from v1.6.37 to v1.6.50
This newer libpng doesn't assume the existence of fp.h on Mac (which
isn't present on newer ones), but the CMake options for libpng changed
and had to be adjusted somewhat.
musl.cc is overwhelmed by GitHub traffic, and therefore has blocked all
of Microsoft. This switches to a mirror run on GitHub so that we can
continue to access the toolchains.
This adds support for Docker-based testing on arm64 images and runners.
This also adds a multiarch build process so that future releases cover
both x64 and arm64.
This is tagged as "feat" so that this gets called out in the changelog
for the next release.
Since the introduction of `vars` to GitHub Actions, I started using that
instead of the "environments" trick I used in Packager. However, it has
become clear now that the `vars` strategy has major drawbacks, such as
requiring the use of `pull_request_target`, which should only be used
for actions that do not execute PR-author-controlled code.
This updates the comments to clarify why this is used. This reusable
settings workflow will also be deployed now in other repos to
standardize on this "environments" mechanism, which is safer than
`vars`.
Because we used require() to read build-matrix.json, the file could be
replaced with build-matrix.json.js, allowing code injection into our CI
pipelines. This fixes this vulnerability by reading the JSON text with
the fs module, then explicitly parsing it, rather than relying on
require().
This also changes the location of the file, to match its location in
other projects.
Note that this workflow is not currently giving any elevated permissions
to users, so it is not currently possible to damage the repo through a
PR. But this might have been possible in the past, due to
organization-wide defaults for token permissions (recently fixed). No
evidence has been found of past exploit.
See also https://github.com/shaka-project/shaka-streamer/issues/216 and
https://github.com/shaka-project/static-ffmpeg-binaries/issues/57
The default .npmignore would disallow bin/, which contains binaries for
our NPM release. Every release before v3.0.3 was missing .npmignore, and
therefore had no binaries. These were unusable in NPM, and have been
marked as deprecated because of it.
Closes#1369
The release workflows did not run checkout with `fetch-tags: true`, so
the builds were unable to compute the correct release version number. I
audited all instances of `actions/checkout` to add `fetch-tags` where
needed and clean up unneeded options.
I also had to fix options to `docker/build-push-action`, which by
default ignores `actions/checkout` and tries to pull from git itself.
This led to the Docker build running in a context without the new tag.
Finally, to make verification easier and provide version info in the
build logs, this adds debugging info to the version-generation script
via stderr.
Closes#1366
Support for this came out in January, 2024. Explicit macos versions seem
to be necessary for now, until GitHub offers "latest" labels targeting
specific architectures.
This adds the option FULLY_STATIC to create fully-static executables.
To create portable, fully-static release executables on Linux, we need
to use musl instead of glibc. Static executables from glibc are not
portable.
The popular musl-gcc wrapper does not support C++, so instead we use a
full musl cross-compiler toolchain in the build workflow.
To build FULLY_STATIC, the user must point to the appropriate
cross-compiler, as we do in the workflow. On systems where musl is the
native libc (such as Alpine Linux), this is not necessary.
I have also read that musl's allocator is not very fast in
multi-threaded applications. So when FULLY_STATIC is enabled, we will
also enable mimalloc, a replacement allocator that is very fast.
I tested a very basic packaging command to compare speeds of dynamic
glibc, static musl, and static musl+mimalloc:
dynamic glibc:
runs: 2.527, 2.798, 2.703, 2.756, 2.959
avg = 2.749, std dev = 0.156s
static musl:
runs: 2.813, 2.920, 3.129, 3.003, 2.738
avg = 2.921s, std dev = 0.154s
static musl+mimalloc:
runs: 2.291, 2.034, 2.415, 2.303, 2.265
avg = 2.262s, std dev = 0.140s
The mimalloc build is 82% faster than musl default allocator, 77% faster
than glibc, and has more consistent runtime characteristics (lower
standard deviation).
According to a comment in
packager/third_party/abseil-cpp/source/absl/log/CMakeLists.txt, many
linkers will strip the contents of absl::log_flags because its symbols
symbols are only used in a global constructor, and that for now, clients
should link using
$<LINK_LIBRARY:WHOLE_ARCHIVE,absl::log_flags>.
Closes#1325
This brings some workflow improvements and fixes from the `cmake` branch
to `main`, as well as some unique fixes to keep gclient working, so that
we can continue to accept contributions in `main` until the `cmake`
merge is ready.
- Fix docs build in GitHub Actions (from `cmake` branch)
- Cancel workflow when a PR is updated (from `cmake` branch)
- Fix docker failures caused by running as root (from `cmake` branch)
- Work around exception in depot_tools on Windows
- Use Windows 2019 images in GitHub Actions for compatibility with gyp
- Remove Docker build on ArchLinux, which no longer supports python2 at
all
- (NOTE: The `cmake` branch is still building on ArchLinux. Docker
builds for Arch will be restored to the `main` branch when the `cmake`
branch is finally merged to `main`.)
Using the latest depot_tools no longer works. depot_tools also wants
to auto-update itself, which must now be disabled.
We also need to disable the copy of python (vpython) included in
depot_tools, since for some distros, it has dependencies on system
libraries that no longer exist.
Finally, we need to force some distros to use python 2, because our
build system is ancient and needs to be ripped out and replaced some
day soon.
This fixes build issues in our CI, our Dockerfiles, and in general on
certain platforms or distros.
Closes#1023
The official, static-linked linux builds were crashing in their use of
getaddrinfo, which libcurl was configured to use. Both getaddrinfo
and all of its alternatives available in glibc fail with static
linking.
We can fix this by configuring libcurl to use libc-ares on Linux
instead. This allows us to keep the benefits of a statically-linked
Linux binary.
Closes#996
Change-Id: Ib4a9eb939813fd165727788726459ef4adf3fc4d
It was suggested in code review for another project that we update the
runner labels for clarity. This brings Packager in line with that, so
that we are using the same labels across projects.
The runners have already been updated to register with the new label.
Change-Id: I30b22530225b5bd22b965ba98d276bcd74ade6cf
Now that we have multiple architectures, we should factor both OS and
architecture into the names of release binaries. This makes the names
more formulaic, as well as consistent with the static-ffmpeg-binaries
repository. Shaka Streamer will pull binaries from both this repo and
that one, so consistent names would be helpful.
The pssh-box release is actually OS and architecture independent, so
remove the suffix from that and only release one copy of it.
Change-Id: Ief3de49fae267c5267647a8dd4377023777ead37
This was causing failures on arm64, where the build action had an
arm-specific clause that was skipped due to the missing parameter.
Change-Id: I71b7fb15120855c444749dc2216b5f19f0561f6e
We never produced static release executables on Linux before, but the dynamic libraries they depended on were universal enough that nobody noticed. Now that we have released v2.5 and switched to GitHub Actions for CI builds, the Linux executables depend on libatomic, which is causing issues for some users.
Although we can't create fully-static executables on macOS or Windows, we can at least do so on Linux.
This adds a GYP variable static_link_binaries which can be set to request full-static binaries on Linux. This also exposes the Chromium build variable disable_fatal_linker_warnings, which is necessary when static linking on Linux due to static-link-related warnings generated by libcurl for its use of getaddrinfo. Finally, this enforces the definition of __UCLIBC__ with static linking on Linux, which is the only way to disable malloc hooks in Chromium base. Those hooks cause linker failures when linking statically on Linux.
A new check has been added to the release workflow to ensure that the builds we create are statically linked on Linux.
Closes#965
There is not a good reason to use a long-lived token attached to
shaka-bot. Instead, use a short-lived, automatic token generated by
GitHub Actions for the workflow run.
This brings our default build config more in line with what is
necessary for some platforms anyway: using the system-installed
toolchain and sysroot to build everything.
We will no longer fetch source or binaries for any specific build
tools, such as libc++, clang, gold, binutils, or valgrind.
The main part of this change is the changing of default gyp settings
in gyp_packager.py. For this, a bug in gyp_packager.py had to be
fixed, in which similar GYP_DEFINE key names (such as clang and
host_clang) would conflict, causing some defaults not to be installed
properly.
In order to enable clang=0 by default, some changes had to be made in
common.gypi:
- compiler macros added to fix a compatibility issue between
Chromium's base/mac/ folder and the actual OSX SDK
- replaced clang_warning_flags variables with standard cflags
settings, plus xcode_settings for OSX
- turned off warnings-as-errors for non-shaka code, rather than
allow-listing specific warning types, since we can't actually fix
those warnings on any platform
- disabled two specific warnings in shaka code, both of which are
caused by headers from our non-shaka dependencies
Also, one warning (missing "override" keyword) has been fixed in
vod_media_info_dump_muxer_listener.h.
Although these changes were done to make building simpler on a wider
array of platforms (arm64, for example), it seems to make the build a
bit faster, too. For me, at least, on my main Linux workstation:
- "gclient sync" now runs 20-30% faster
- "ninja -C out/Release" now runs 5-13% faster
The following environment variables are no longer required:
- DEPOT_TOOLS_WIN_TOOLCHAIN
- MACOSX_DEPLOYMENT_TARGET
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following GYP_DEFINES are no longer required for anyone:
- clang=0
- host_clang=0
- clang_xcode=1
- use_allocator=none
- use_experimental_allocator_shim=0
Documentation, Dockerfiles, and GitHub Actions workflows have been
updated to reflect this.
The following repos are no longer dependencies in gclient:
- binutils
- clang
- gold
- libc++
- libc++abi
- valgrind
The following gclient hooks have been removed:
- clang
- mac_toolchain
- sysroot
Change-Id: Ie94ccbeec722ab73c291cb7df897d20761a09a70
Internal CI systems and the new GitHub CI system were out of sync,
with the external system not doing any linting. Further, the internal
system was using an internal-only linter for Python.
This creates a script for Python linting based on the open-source
pylint tool, checks in the Google Style Guide's pylintrc file, creates
a custom action for linting and adds it to the existing workflows,
fixes pre-existing linter errors in Python scripts, and updates pylint
overrides.
b/190743862
Change-Id: Iff1f5d4690b32479af777ded0834c31c2161bd10
