• Drew's avatar
    Add atbin support · 1c22e037
    Drew authored
    atbin is a proposed binary interchange format for atbuild and the
    broader AT ecosystem.
    
    atbuild has a weakly standardized input format: `build.atpkg`.  But what
    does atbuild, um, actually build?  What's the output format?
    
    There is no weak standard here, or even a convention.  It may build an
    executable, a static library, or a dynamic one, or even a framework; it
    may emit swiftmodule and swiftdoc files, or not.  A modulemap may or may
    not be part of the build products and clients may or may not need it in
    their search paths.
    
    The uncertainty here complicates interoperability.  atpm should download
    binary libraries, but what actually lives in that tarball?  Some random
    dylibs we found in `.atllbuild/products`?
    
    How do we build libraries for "fat" archs (like iOS, with 4 sub-archs)?
    How would we write an `atinstall` tool that installs/updates
    atpm/atbuild (or makes homebrew packages out of them)?
    
    atbin proposes to answer these questions, providing a simple, portable,
    hackable binary interchange format for all platforms and all AT
    projects.
    
    An `atbin` is a folder that ends in `.atbin`.  It contains, at least, a
    `built.atpkg` file.
    
    `built.atpkg` is clojure syntax identical to the more familiar
    `build.atpkg`.  You can include tasks or w/e in there, although why
    would want to, I'm not totally sure (this is Anarchy Tools though,
    knock yourself out.)  However, the important bit is this:
    
    ```clojure
    (package
        :name "foo"
        :payload "libFoo.a"
        :platforms ["ios-x86_64" "ios-i386"]
        :type "static-library"
    )
    ```
    
    (Other fields could also be present, this is not a complete enumeration)
    
    This `.atbin` folder will then contain:
    
    * `libFoo.a`, a fat library for the indicated platforms
    * (optional) a `ios-x86_64.swiftmodule` and `ios-i386.swiftmodule` file
    * (optional) a `ios-x86_64.swiftdoc` and `ios-i386.swiftdoc` file
    * (optional) a `module.modulemap` file
    
    You can, of course, build an `.atbin` by hand from existing binaries you
    found lying around your disk.  And we may eventually ship an `atbin`
    packager for Xcode or SwiftPM projects.
    
    However more practically, we introduce a new tool, `packageatbin`, which
    packages an `atbin` payload from atllbuild.
    
    ```clojure
    :package {
       :tool "packageatbin"
    
       ;; Generate a mypayload.atbin
       :name "mypayload"
    
       ;; When used with the new --platform ios, will build a fat binary for all iOS platforms.
       ;; Alternatively specific platforms can be listed here
       :platforms ["all"]
    
       ;; The atllbuild task to package.
       ;; Special logic will re-run this task for each platform and merge the resulting output.
       :atllbuild-task "myatllbuildtask"
    }
    ```
    
    The obvious application is as an interchange format for prebuilt
    `atllbuild` dependencies.  Presently, `atllbuild` can link with the
    output of any dependent atllbuild task, but if a library wasn't produced
    by a dependent task as part of the current build (but was say produced
    on a different computer a month ago) there's no "obvious" way to link to
    it.  This PR does not actually include any of that tooling, but it would
    be a straightforward extension of this work.
    
    An second application is the building of fat files.  Currently, there is
    no mechanism to build a "fat" library or binary in atbuild, or even to
    specify that we want one.  Under this PR, we can do it.
    
    A third application is a distribution format for executables.  If an
    `.atbin` contains an `executable`, `atpm` (or hypothetical `atinstall`)
    could install/update/administrate that executable similar to `homebrew`
    or `apt`, and keep all your buildtools (or other random crap) up to
    date.  We would need to extend this with version fields and whatnot, but
    again, it's straightforward.
    
    An fourth application, and my real motivation, is as an intermediate
    binary representation.  An `atbin` can be "downcast" with another tool
    to a platform-native format like `deb`, `bottle`, or `Framework`.  This
    would allow us to cut debs, rpms, and framework releases with
    appropriate AT tools.
    
    One alternative is to adopt an existing "standard", like Framework, for
    this purpose.  Indeed, `atbuild` currently produces frameworks on OSX.
    
    There are some complexities of extending frameworks to this case.  For
    one, the Framework implementation is warty and involves a lot of
    symlinks and things like codesigning.  We don't currently maintain that
    code for Linux hosts, nor is the standard especially sensible for Linux,
    as it relies on plists and choices basically unpopular on that platform.
    
    For another, frameworks are not really built to house static library or
    executable payloads, which are important to atbuild.  There are air-
    quote "obvious" ways to extend to nontraditional payloads, but IMO this
    is more confusing than it is helpful.  An explicit step to "cast down"
    your atbin to a framework lets us check that your framework will
    actually make sense to the likes of Xcode.
    
    For a third, it's unclear what platform some random Framework is built
    for, and what architectures it supports.  You can find out by scripting
    out to platform-specific tools, but it's not portable.
    
    Another alternative is to support multiple payloads/libraries in a
    single atbin, "one atbin to rule them all".  However I don't see what we
    accomplish there that we don't accomplish with multiple atbins, except
    specification complexity.  So let's not do that, at least not initially.
    
    `packageatbin` is included in core primarily because it needs tight,
    source-level integration with atllbuild.  In addition to peeking at the
    atllbuild options it needs to run the atllbuild task several times in
    order to produce fat binaries, which means it has to work around the
    usual dependency pruning logic.  For that reason it can't be sensibly
    implemented via the current custom tool API.
    1c22e037
atllbuild.swift 22.8 KB