1. 11 May, 2016 3 commits
  2. 10 May, 2016 4 commits
    • Drew's avatar
      Support atbin compression · 3982ceb4
      Drew authored
      3982ceb4
    • Drew's avatar
      Add only-platforms · 658955ba
      Drew authored
      A task-level mechanism to skip tasks on some platforms
      658955ba
    • Drew's avatar
      Set environment variable $ATBUILD_PACKAGE_VERSION · 36f4cbef
      Drew authored
      36f4cbef
    • Drew's avatar
      Add executable-name option to atllbuild · 8af5d3c8
      Drew authored
      We add an executable-name option to atllbuild, allowing the use of "non-module-safe" names for executables.  This includes e.g. hyphens, which are a legal executable name but not a legal module-name.
      
      Resolve #27.  This resolution was chosen (over name/module-name) because the module-name is used in several places (such as Frameworks for example) and the executable case seems to be the odd one out at present.
      8af5d3c8
  3. 28 Apr, 2016 2 commits
    • Drew's avatar
      Change name to `compiled.atpkg` · 393fb733
      Drew authored
      393fb733
    • Drew's avatar
      Add atbin support · 1c22e037
      Drew authored
      atbin is a proposed binary interchange format for atbuild and the
      broader AT ecosystem.
      
      atbuild has a weakly standardized input format: `build.atpkg`.  But what
      does atbuild, um, actually build?  What's the output format?
      
      There is no weak standard here, or even a convention.  It may build an
      executable, a static library, or a dynamic one, or even a framework; it
      may emit swiftmodule and swiftdoc files, or not.  A modulemap may or may
      not be part of the build products and clients may or may not need it in
      their search paths.
      
      The uncertainty here complicates interoperability.  atpm should download
      binary libraries, but what actually lives in that tarball?  Some random
      dylibs we found in `.atllbuild/products`?
      
      How do we build libraries for "fat" archs (like iOS, with 4 sub-archs)?
      How would we write an `atinstall` tool that installs/updates
      atpm/atbuild (or makes homebrew packages out of them)?
      
      atbin proposes to answer these questions, providing a simple, portable,
      hackable binary interchange format for all platforms and all AT
      projects.
      
      An `atbin` is a folder that ends in `.atbin`.  It contains, at least, a
      `built.atpkg` file.
      
      `built.atpkg` is clojure syntax identical to the more familiar
      `build.atpkg`.  You can include tasks or w/e in there, although why
      would want to, I'm not totally sure (this is Anarchy Tools though,
      knock yourself out.)  However, the important bit is this:
      
      ```clojure
      (package
          :name "foo"
          :payload "libFoo.a"
          :platforms ["ios-x86_64" "ios-i386"]
          :type "static-library"
      )
      ```
      
      (Other fields could also be present, this is not a complete enumeration)
      
      This `.atbin` folder will then contain:
      
      * `libFoo.a`, a fat library for the indicated platforms
      * (optional) a `ios-x86_64.swiftmodule` and `ios-i386.swiftmodule` file
      * (optional) a `ios-x86_64.swiftdoc` and `ios-i386.swiftdoc` file
      * (optional) a `module.modulemap` file
      
      You can, of course, build an `.atbin` by hand from existing binaries you
      found lying around your disk.  And we may eventually ship an `atbin`
      packager for Xcode or SwiftPM projects.
      
      However more practically, we introduce a new tool, `packageatbin`, which
      packages an `atbin` payload from atllbuild.
      
      ```clojure
      :package {
         :tool "packageatbin"
      
         ;; Generate a mypayload.atbin
         :name "mypayload"
      
         ;; When used with the new --platform ios, will build a fat binary for all iOS platforms.
         ;; Alternatively specific platforms can be listed here
         :platforms ["all"]
      
         ;; The atllbuild task to package.
         ;; Special logic will re-run this task for each platform and merge the resulting output.
         :atllbuild-task "myatllbuildtask"
      }
      ```
      
      The obvious application is as an interchange format for prebuilt
      `atllbuild` dependencies.  Presently, `atllbuild` can link with the
      output of any dependent atllbuild task, but if a library wasn't produced
      by a dependent task as part of the current build (but was say produced
      on a different computer a month ago) there's no "obvious" way to link to
      it.  This PR does not actually include any of that tooling, but it would
      be a straightforward extension of this work.
      
      An second application is the building of fat files.  Currently, there is
      no mechanism to build a "fat" library or binary in atbuild, or even to
      specify that we want one.  Under this PR, we can do it.
      
      A third application is a distribution format for executables.  If an
      `.atbin` contains an `executable`, `atpm` (or hypothetical `atinstall`)
      could install/update/administrate that executable similar to `homebrew`
      or `apt`, and keep all your buildtools (or other random crap) up to
      date.  We would need to extend this with version fields and whatnot, but
      again, it's straightforward.
      
      An fourth application, and my real motivation, is as an intermediate
      binary representation.  An `atbin` can be "downcast" with another tool
      to a platform-native format like `deb`, `bottle`, or `Framework`.  This
      would allow us to cut debs, rpms, and framework releases with
      appropriate AT tools.
      
      One alternative is to adopt an existing "standard", like Framework, for
      this purpose.  Indeed, `atbuild` currently produces frameworks on OSX.
      
      There are some complexities of extending frameworks to this case.  For
      one, the Framework implementation is warty and involves a lot of
      symlinks and things like codesigning.  We don't currently maintain that
      code for Linux hosts, nor is the standard especially sensible for Linux,
      as it relies on plists and choices basically unpopular on that platform.
      
      For another, frameworks are not really built to house static library or
      executable payloads, which are important to atbuild.  There are air-
      quote "obvious" ways to extend to nontraditional payloads, but IMO this
      is more confusing than it is helpful.  An explicit step to "cast down"
      your atbin to a framework lets us check that your framework will
      actually make sense to the likes of Xcode.
      
      For a third, it's unclear what platform some random Framework is built
      for, and what architectures it supports.  You can find out by scripting
      out to platform-specific tools, but it's not portable.
      
      Another alternative is to support multiple payloads/libraries in a
      single atbin, "one atbin to rule them all".  However I don't see what we
      accomplish there that we don't accomplish with multiple atbins, except
      specification complexity.  So let's not do that, at least not initially.
      
      `packageatbin` is included in core primarily because it needs tight,
      source-level integration with atllbuild.  In addition to peeking at the
      atllbuild options it needs to run the atllbuild task several times in
      order to produce fat binaries, which means it has to work around the
      usual dependency pruning logic.  For that reason it can't be sensibly
      implemented via the current custom tool API.
      1c22e037
  4. 25 Apr, 2016 2 commits
    • Drew's avatar
      Fix usage · 05e02a16
      Drew authored
      We've broken this a couple of times, introduce test coverage so we stop
      breaking it.
      05e02a16
    • Drew's avatar
      Export ATBUILD_PLATFORM · db7b2bd2
      Drew authored
      ATBUILD_PLATFORM is now exported to custom tools and shell scripts.
      
      This allows external tools to know atbuild's target platform.
      db7b2bd2
  5. 24 Apr, 2016 2 commits
    • Drew's avatar
      Add iOS support · 656d6bb3
      Drew authored
      This commit adds support for static libraries, dynamic libraries, and
      executables compiled for iOS.
      
      FAQ:
      
      Q: How do I build them?
      
      Use the new `--platform` strings:
      
      * `ios-x86_64`
      * `ios-i386`
      * `ios-arm64`
      * `ios-armv7`
      
      Q: What if I want to build for more than one architecture?
      
      Coming Soon
      
      Q: What is an iOS "executable", anyway?
      
      No idea, but it works!
      
      Q: What is not yet supported?
      
      - [ ] XCTest
      - [ ] Deploying or running iOS build products
      - [ ] Frameworks
      - [ ] Code signing
      - [ ] Compiling for iOS on Linux.  Believe it or not, I think this
            is actually possible for some programs, but I have no use for it
      656d6bb3
    • Drew's avatar
      Support static linking · e56ece3e
      Drew authored
      To statically-link atbuild, use `--use-overlay static` when building
      e56ece3e
  6. 22 Apr, 2016 1 commit
  7. 21 Apr, 2016 1 commit
    • Drew's avatar
      Change to `.attool` · 5180ad67
      Drew authored
      @owensd convinced me that `plugin` is the wrong name.  We should call
      @them Custom Tools, with an extension `.attool`.  This emphasizes that
      @they are written for AT and provides something to google.
      5180ad67
  8. 19 Apr, 2016 1 commit
    • Drew's avatar
      Add plugin support · ecca2810
      Drew authored
      We're supposed to be building simple, hackable tools but atbuild is
      becoming more of a monolithic tool.  This patch aims to change that.
      
      There are many reasons some feature should not be included in core:
      
      1.  Core needs to be minimal so we can bootstrap it on new platforms;
          every new feature is a new burden
      
      2.  Core needs to be x-platform but many features do not
          (see: xcode-emit, packageframework)
      
      3.  Features may want different release frequency (see: xcode-emit) or
          don't want to coordinate with atbuild
      
      4.  Features may be "not part of AnarchyTools" (Caroline) but still
          commonly used together
      
      5.  I'm annoyed at upstream over the static linking issue, and thinking
          of ways to keep us from being that point of central failure that annoys
          somebody else someday
      
      Therefore, we introduce the world's simplest plugin system designed to
      move code out of core, or keep code out of core that doesn't need to be.
      
      xcode-emit and Caroline will consume this API.  packageframework is a
      good candidate for a plugin that might be moved out from core.
      
      Documentation to follow
      ecca2810
  9. 15 Apr, 2016 2 commits
    • Drew's avatar
      Add proper platform support · 9ecf5b64
      Drew authored
      Related to #36
      
      Presently, we tend to enable platform-specific config with an overlay.
      There are a variety of problems identified with this approach:
      
      1.  There is no convention for which overlay to use for platform-
      specific config.  This complicates the ecosystem.
      
      2.  In general, a program is always compiled "for some platform" but in
      practice a user may forget the necessary overlay.  `require-overlays`
      can catch this misconfig, but A) it has to be opted into in the atpkg,
      and B) there is no good way to default to the running platform, which is
      the sane behavior.
      
      3.  Currently, tools tend to conditionally-compile platform-specific
      code.  The effect of this is to complicate bootstrapping, as a Linux
      binary *cannot* perform some OSX behavior (and vice versa) because the
      necessary code is not present in the executable.
      
      4.  This is not scaleable to cross-compilation (iOS for example is
      Coming Soon but can't be supported in this architecture)
      
      To work around these problems, we introduce a `Platform` enum, to
      replace `PlatformPaths`.  `Platform` is a runtime technology, allowing
      for a Linux binary to reason about what the behavior would be on OSX
      etc.
      
      Internally, we have three "platforms":
      
      * `hostPlatform`, the platform on which `atbuild` is currently executing
      * `targetPlatform`, the platform for which we are compiling. By default
         this is the `hostPlatform`
      * `buildPlatform`, the platform where `swift-build-tool` will run.
         This is usually the `hostPlatform`, but if we are bootstrapping it
         is the `targetPlatform` instead.
      
      The user can override the `targetPlatform` by the use of `--platform
      foo`.  `linux` and `osx` are supported.  `mac` is supported as an alias
      of `osx`.
      
      The primary effect of a platform is to scope tool-specific behavior
      (e.g., target=OSX uses the OSX SDK, host=Linux uses a linux path for the
      toolchain, etc).
      
      In addition to the tool-specific behavior, we enable overlays for the
      target platform:
      
      * `atbuild.platform.linux` on Linux
      * `atbuild.platform.osx` and `atbuild.platform.mac` on OSX
      
      This allows packages to reliably perform per-platform configuration in
      the overlays.  Critically, some platform overlay is reliably active, so
      users in most cases will not have to `--use-overlay` to get proper
      platform behavior.
      
      DEPRECATION WARNING: We believe the `swiftc-path` key is no longer
      required, as the functionality used can be achieved either by
      `--toolchain` or `--platform`.  Therefore, I'm adding a warning to users
      that we intend to remove it and to try these features instead.
      
      We need to put out a release with these features (and the warning)
      before I'm happy to remove it.  In particular, we use it inside
      atbuild/atpkg, and removing it immediately would break bootstrapping, so
      let's give it a little time before we tear it out.  We should remove it
      from the documentation though.
      9ecf5b64
    • Drew's avatar
  10. 07 Apr, 2016 3 commits
    • Drew's avatar
      Toolchain support · 0c1ba2e6
      Drew authored
      This allows atbuild to use a different toolchain other than the one we
      use to develop atbuild (weekly snapshot).
      
      In particular, this allows you to use "released swift" "xcode swift" or
      any other kind of 2.2 Swift.
      
      Documentation PR to follow.
      
      Edited README to discuss atbuild options.
      
      Resolves #58 to my satisfaction.
      0c1ba2e6
    • Drew's avatar
      Implement packageFramework, a framework packaging tool · c449e5c3
      Drew authored
      This allows Mac frameworks to be built directly from atbuild.
      c449e5c3
    • Drew's avatar
      Allow variable expansion for shell tool · 409cc69b
      Drew authored
      This allows the execution of a shell tool with the "collected sources"
      of another task.
      
      The intent here is to allow the creation of arbitrary tasks that accept
      source files as arguments. xcode-emit is an obvious candidate, and
      would allow it to break its dependence on atpkg, which it only uses for
      this single feature.
      
      There are other obvious programs that we might want to call with "all
      sources" (e.g. preprocessors, etc.) and they may not want to take a
      dependency on atpkg either.
      
      See https://github.com/AnarchyTools/atpkg/pull/23
      409cc69b
  11. 28 Mar, 2016 1 commit
  12. 19 Mar, 2016 1 commit
  13. 04 Mar, 2016 1 commit
    • Drew's avatar
      Support whole-module-optimization · eadeb144
      Drew authored
      Due to SR-881, whole-module-optimization is more complicated than simply
      adding it to compile options.
      
      As a workaround, we compile twice.
      eadeb144
  14. 06 Feb, 2016 1 commit
    • Drew's avatar
      Mute warnings · b8374eb6
      Drew authored
      Update for atpkg:mute_warnings, which changes the package constructor
      
      We silence certain redundant warnings that are probably useless.
      b8374eb6
  15. 04 Feb, 2016 3 commits
    • Drew's avatar
      Ignore tool keys when warning from atllbuild · 6c86e64e
      Drew authored
      Previously, atllbuild would emit some spurious warnings when a tool key
      was used such as `overlays`.  We now ignore all tool keys.
      6c86e64e
    • Drew's avatar
      Fix linux build · fa2efa95
      Drew authored
      fa2efa95
    • Drew's avatar
      Umbrella headers v2 · 587894a9
      Drew authored
      We now include the synthesized module map in our build products.  To
      support this, a new `:modulemap "synthesized"` directive is available
      (and must be used if `umbrella-header` is used.)  We could potentially
      have other modulemap modes besides synthesized (explicit, for example).
      
      We use one modulemap privately as part of the build process (to store
      the umbrella header) and a different one publicly (which doesn't have
      the umbrella header).
      
      This is (surprisingly!) totally legal under the Clang module map
      specification.
      http://clang.llvm.org/docs/Modules.html#private-module-map-files
      
      > However, in some cases, the presence or absence of particular headers is used to distinguish between the “public” and “private” APIs of a particular library. For example, a library may contain the headers Foo.h and Foo_Private.h, providing public and private APIs, respectively. Additionally, Foo_Private.h may only be available on some versions of library, and absent in others. One cannot easily express this with a single module map file in the library:
      
      ```
      module Foo {
        header "Foo.h"
      
        explicit module Private {
          header "Foo_Private.h"
        }
      }
      ```
      
      > because the header Foo_Private.h won’t always be available. The module
      > map file could be customized based on whether Foo_Private.h is
      > available or not, but doing so requires custom build machinery.
      
      We are the custom build machinery of which the clattner foretold.
      
      Note that (some) Swift engineers claim this is dangerous, but it works,
      is compliant with the Clang specification, and the tests pass.
      
      I should probably also document one other thing that I found diagnosing
      why the previous approach didn't work.  A "swiftmodule" (which nobody
      seems to understand) actually (sometimes) functions as an overlay for a
      Clang module.  See e.g.
      https://github.com/apple/swift/blob/1b2288fa96e4d531956bc690e64616afc2fb3333/include/swift/Serialization/Validation.h#L41
      
      What happen is that if you compile with -import-underlying-module (which
      we do in the umbrella case) the swift module that gets built is a
      "overlay" that points to some underlying clang module, see here:
      https://twitter.com/drewcrawford/status/694995772148846592.  Then at
      import time somebody goes looking for that underlying module (e.g.
      modulemap) in addition to the .swiftmodule.
      
      However there is no reason (as far as Clang is concerned) why the
      modulemap it finds at runtime may not be totally different than the
      modulemap we used at compile time, so that's what we do.
      587894a9
  16. 02 Feb, 2016 4 commits
    • Drew's avatar
      Warn about invalid package keys · 3320fe4a
      Drew authored
      Merge with atpkg:warn_package
      3320fe4a
    • Drew's avatar
      Add `umbrella-header` to specify umbrella headers · 089ebe8b
      Drew authored
      This is essentially the same as a "bridging" header, except without any
      ObjC (straight C).  Upstream says it "can't be done"; let's see if Linux
      CI likes this implementation.  Take *that*, anonymous compiler engineers.
      
      My motivation for this patch is essentially as a workaround for SR-655.
      
      https://bugs.swift.org/browse/SR-655
      
      That is to say, I want to call some arbitrary C code, but due to that
      bug, I can't hide my C code behind a module / modulemap.
      
      Instead what I can do is just `#include <whatever>` in my umbrella
      header, and then the C imports are available to Swift.
      
      Internally, we emit a modulemap based on the umbrella header and then
      inject some compile flags (notably, `-import-underlying-module`) to make
      it load.  This is the design I reverse-engineered from Xcode.  That
      notwithstanding, it seems to be based entirely on public APIs and I
      guess we'll find out if it works on Linux.
      
      While my motivation is based on C imports, umbrella headers may be
      useful wherever Swift projects are sold.  There are multiple features
      that get unlocked here, including access to the C preprocessor.
      089ebe8b
    • David Owens II's avatar
      d46cecf6
    • Drew's avatar
      Make the "passed successfully" text green. · 0c61b897
      Drew authored
      Our swiftc frontend is typically colored, which draws attention to
      various (expected) compile failures in our test suite.
      
      To fix this, our test suite pass should report green and bold, so I stop
      debugging test failures that aren't failures at all.
      0c61b897
  17. 01 Feb, 2016 4 commits
    • Drew's avatar
      Add `publishProduct` atllbuild option · 0b8b3339
      Drew authored
      This allows a build.atpkg to export its products into a user-visible directory.
      
      I'm not totally sure this is the right design, but it does resolve #17.
      0b8b3339
    • Drew's avatar
      Allow `-f [file]` to specify a build file · f8e53f6e
      Drew authored
      This allows the building of files other than build.atpkg
      f8e53f6e
    • Drew's avatar
      Add atbuild "user paths". · b1fc7420
      Drew authored
      This patch adds a "user path" which is global to an entire buildchain
      and allows tasks to progressively build up contents that are somewhat
      arbitrary.
      
      In particular, this is useful for AnarchyDispatch.  If we have
      a/b/AnarchyDispatch, then AnarchyDispatch can install itself to the
      ATBUILD_USER_PATH and be visible to both a and b.  Then we can inject
      includeWithUser: ["AnarchyDispatch"] in an overlay and everything "just
      works".
      b1fc7420
    • Drew's avatar
      Update to new API to atpkg · 777f7d08
      Drew authored
      777f7d08
  18. 27 Jan, 2016 3 commits
  19. 19 Jan, 2016 1 commit