Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trans: Use LLVM's writeArchive to modify archives #26926

Merged
merged 1 commit into from Jul 10, 2015

Conversation

alexcrichton
Copy link
Member

We have previously always relied upon an external tool, ar, to modify archives
that the compiler produces (staticlibs, rlibs, etc). This approach, however, has
a number of downsides:

  • Spawning a process is relatively expensive for small compilations
  • Encoding arguments across process boundaries often incurs unnecessary overhead
    or lossiness. For example ar has a tough time dealing with files that have
    the same name in archives, and the compiler copies many files around to ensure
    they can be passed to ar in a reasonable fashion.
  • Most ar programs found do not have the ability to target arbitrary
    platforms, so this is an extra tool which needs to be found/specified when
    cross compiling.

The LLVM project has had a tool called llvm-ar for quite some time now, but it
wasn't available in the standard LLVM libraries (it was just a standalone
program). Recently, however, in LLVM 3.7, this functionality has been moved to a
library and is now accessible by consumers of LLVM via the writeArchive
function.

This commit migrates our archive bindings to no longer invoke ar by default
but instead make a library call to LLVM to do various operations. This solves
all of the downsides listed above:

  • Archive management is now much faster, for example creating a "hello world"
    staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also
    recently started requiring modification of rlibs, and linking a hello world
    dynamic library is now 2x faster.
  • The compiler is now one step closer to "hassle free" cross compilation because
    no external tool is needed for managing archives, LLVM does the right thing!

This commit does not remove support for calling a system ar utility currently.
We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward
(so the system LLVM can be used wherever possible), and in these cases we must
shell out to a system utility. All nightly builds of Rust, however, will stop
needing a system ar.

@alexcrichton
Copy link
Member Author

r? @brson
cc @rust-lang/tools

@rust-highfive
Copy link
Collaborator

r? @huonw

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive assigned brson and unassigned huonw Jul 9, 2015
@emberian
Copy link
Member

emberian commented Jul 9, 2015

cc @Kingsquee

@emberian
Copy link
Member

emberian commented Jul 9, 2015

I believe this should also remove the ability to specify an "ar" program in target specs?

@alexcrichton
Copy link
Member Author

I believe this should also remove the ability to specify an "ar" program in target specs?

Indeed! I haven't done so here as I suspect we'll want to leave them around for awhile to work with 3.5/3.6 (and perhaps bugs that come up in the LLVM implementation). We'll also probably want to just accept them as an option for forever and perhaps start emitting a warning one day saying "you don't need to do this any more".

@ghost
Copy link

ghost commented Jul 9, 2015

Beautiful!

@brson
Copy link
Contributor

brson commented Jul 9, 2015

@bors r+

@bors
Copy link
Contributor

bors commented Jul 9, 2015

📌 Commit 08847de has been approved by brson

@brson brson added the relnotes Marks issues that should be documented in the release notes of the next release. label Jul 9, 2015
@richo
Copy link
Contributor

richo commented Jul 9, 2015

Does this bring us one step closer to not needing a bunch of flags for seamless cross compiling? (has it been tested for cross compilation?)

@alexcrichton
Copy link
Member Author

@richo It should indeed, yes! I haven't tested this particular PR for cross compilation yet, but every time before I've used llvm-ar for cross compilation it works, so I suspect that this will also work.

We also may have to turn this off by default and put it behind a -Z flag if it becomes problematic, but that's what we have nightlies for :)

@alexcrichton
Copy link
Member Author

@bors: r-

Investigating some failures on OSX

@alexcrichton
Copy link
Member Author

re-r? @brson

I've now discovered that there are multiple formats for archives (or at least various tweaks on flavors). The known formats (at least by LLVM) are these, and our current revision of LLVM can only emit archives in the GNU format. Support has been added very recently for BSD formats (what I believe OSX uses), but it's not present in our LLVM version currently and we can't upgrade due to some assorted test failures.

I updated this to add an archive_format field to target specifications. Right now a value of "gnu" (which linux-like targets and MinGW all use) indicates that LLVM should be used, otherwise we fall back to the system ar tool. I'm hoping we can over time pull all targets into using LLVM so we don't have any lagging behind.

@brson
Copy link
Contributor

brson commented Jul 10, 2015

@bors r+

@bors
Copy link
Contributor

bors commented Jul 10, 2015

📌 Commit 1c5c84f has been approved by brson

@bors
Copy link
Contributor

bors commented Jul 10, 2015

⌛ Testing commit 1c5c84f with merge 7b855c0...

@bors
Copy link
Contributor

bors commented Jul 10, 2015

💔 Test failed - auto-mac-64-opt

We have previously always relied upon an external tool, `ar`, to modify archives
that the compiler produces (staticlibs, rlibs, etc). This approach, however, has
a number of downsides:

* Spawning a process is relatively expensive for small compilations
* Encoding arguments across process boundaries often incurs unnecessary overhead
  or lossiness. For example `ar` has a tough time dealing with files that have
  the same name in archives, and the compiler copies many files around to ensure
  they can be passed to `ar` in a reasonable fashion.
* Most `ar` programs found do **not** have the ability to target arbitrary
  platforms, so this is an extra tool which needs to be found/specified when
  cross compiling.

The LLVM project has had a tool called `llvm-ar` for quite some time now, but it
wasn't available in the standard LLVM libraries (it was just a standalone
program). Recently, however, in LLVM 3.7, this functionality has been moved to a
library and is now accessible by consumers of LLVM via the `writeArchive`
function.

This commit migrates our archive bindings to no longer invoke `ar` by default
but instead make a library call to LLVM to do various operations. This solves
all of the downsides listed above:

* Archive management is now much faster, for example creating a "hello world"
  staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also
  recently started requiring modification of rlibs, and linking a hello world
  dynamic library is now 2x faster.
* The compiler is now one step closer to "hassle free" cross compilation because
  no external tool is needed for managing archives, LLVM does the right thing!

This commit does not remove support for calling a system `ar` utility currently.
We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward
(so the system LLVM can be used wherever possible), and in these cases we must
shell out to a system utility. All nightly builds of Rust, however, will stop
needing a system `ar`.
@alexcrichton
Copy link
Member Author

@bors: r=brson 4a82427

@bors
Copy link
Contributor

bors commented Jul 10, 2015

⌛ Testing commit 4a82427 with merge 072d07c...

bors added a commit that referenced this pull request Jul 10, 2015
We have previously always relied upon an external tool, `ar`, to modify archives
that the compiler produces (staticlibs, rlibs, etc). This approach, however, has
a number of downsides:

* Spawning a process is relatively expensive for small compilations
* Encoding arguments across process boundaries often incurs unnecessary overhead
  or lossiness. For example `ar` has a tough time dealing with files that have
  the same name in archives, and the compiler copies many files around to ensure
  they can be passed to `ar` in a reasonable fashion.
* Most `ar` programs found do **not** have the ability to target arbitrary
  platforms, so this is an extra tool which needs to be found/specified when
  cross compiling.

The LLVM project has had a tool called `llvm-ar` for quite some time now, but it
wasn't available in the standard LLVM libraries (it was just a standalone
program). Recently, however, in LLVM 3.7, this functionality has been moved to a
library and is now accessible by consumers of LLVM via the `writeArchive`
function.

This commit migrates our archive bindings to no longer invoke `ar` by default
but instead make a library call to LLVM to do various operations. This solves
all of the downsides listed above:

* Archive management is now much faster, for example creating a "hello world"
  staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also
  recently started requiring modification of rlibs, and linking a hello world
  dynamic library is now 2x faster.
* The compiler is now one step closer to "hassle free" cross compilation because
  no external tool is needed for managing archives, LLVM does the right thing!

This commit does not remove support for calling a system `ar` utility currently.
We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward
(so the system LLVM can be used wherever possible), and in these cases we must
shell out to a system utility. All nightly builds of Rust, however, will stop
needing a system `ar`.
@bors bors merged commit 4a82427 into rust-lang:master Jul 10, 2015
@alexcrichton alexcrichton deleted the llvm-archive-writer branch July 13, 2015 20:41
kennytm added a commit to kennytm/rust that referenced this pull request Oct 8, 2017
…lexcrichton

Document that `-C ar=PATH` doesn't do anything

Are there any plans to use an external archiver in the future?
IIRC, it was used before, but its use was replaced with LLVM's built-in archive management machinery. I can't found a relevant PR though. EDIT: Found it - rust-lang#26926!

The `-C` option is stable so it still can't be removed right away even if there are no plans to use it (but maybe it can be deprecated?).
Target specifications have a field for archiver as well, which is unused too (these ones are unstable, so I guess it can be removed).

r? @alexcrichton
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
relnotes Marks issues that should be documented in the release notes of the next release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants