goinstall: an experiment in package installation

Goinstall already works with standalone packages that
don't depend on other packages. For example, I can run:

goinstall github.com/hoisie/web.go.git
goinstall github.com/jacobsa/igo.git/set
goinstall go-avltree.googlecode.com/svn/trunk
goinstall mimeparse.googlecode.com/svn/trunk/go
goinstall gomatrix.googlecode.com/hg
goinstall golang.googlecode.com/hg/src/pkg/scanner

And then import any of those using the same paths.
If those had referred to other import paths of the same
form, goinstall would have downloaded the dependencies,
recursively. So in your own package, you can put
a line like import "gomatrix.googlecode.com/hg" and
then just run

goinstall my/package

which will notice the import line and download and install
the remote package. The first time goinstall downloads
and installs a public package, it pings the Go dashboard
to update a list of what people are using.
http://godashboard.appspot.com/package.

This is a request for comments on the general approach.
Please comment here, on the mailing list, rather than in the
codereview site. It will be easier to have a discussion in an
actual discussion forum.

-- documentation from goinstall/doc.go

Goinstall is an experiment in automatic package installation.
It installs packages, possibly downloading them from public version
control systems, and it maintains a list of public Go packages at
http://godashboard.appspot.com/packages.

Usage:
goinstall [flags] importpath...

Flags and default settings:
-dashboard=true tally public packages on godashboard.appspot.com
-update=false update already-downloaded packages
-v=false verbose operation

Goinstall installs each of the packages identified by import path on
the command line. It installs a package's prerequisites before
trying to install the package itself.

The source code for a package with import path foo/bar is expected
to be in the directory $GOROOT/src/pkg/foo/bar/. If the import
path refers to a code hosting site, goinstall will download it if necessary.
The recognized code hosting sites are:

GitHub (Git)

import "github.com/user/project.git"
import "github.com/user/project.git/sub/directory"

Google Code Project Hosting (Mercurial, Subversion)

import "project.googlecode.com/hg"
import "project.googlecode.com/hg/sub/directory"

import "project.googlecode.com/svn/trunk"
import "project.googlecode.com/svn/trunk/sub/directory"

If the directory (e.g., $GOROOT/src/pkg/bitbucket.org/user/project)
already exists and contains an appropriate checkout, goinstall will not
attempt to fetch updates. The -update flag changes this behavior,
causing goinstall to update all remote packages encountered during
the installation.

When downloading or updating, goinstall first looks for a tag or branch
named "release". If there is one, it uses that version of the code.
Otherwise it uses the default version selected by the version control
system (typically HEAD for git, tip for Mercurial).

After a successful download and installation of a publicly accessible
remote package, goinstall reports the installation to godashboard.appspot.com,
which increments a count associated with the package and the time
of its most recent installation. This mechanism powers the package list
at http://godashboard.appspot.com/packages, allowing Go programmers
to learn about popular packages that might be worth looking at.
The -dashboard=false flag disables this reporting.

By default, goinstall prints output only when it encounters an error.
The -v flag causes goinstall to print information about packages
being considered and installed.

Goinstall does not attempt to be a replacement for make.
Instead, it invokes "make install" after locating the package sources.
For local packages without a Makefile and all remote packages,
goinstall creates and uses a temporary Makefile constructed from
the import path and the list of Go files in the package.

FAQ:

• Why not use full URLs instead of hard-coding hosting sites?

Goinstall is an experiment, and it is easier if the experiment does
not involve changes to the compilers and godoc. The paths used
above double as file system paths, while full URLs do not.
Also, the import path must also convey both the location of
the data and the choice of version control system, so a full URL
would not suffice.

Using this notation does not precluding changing to other
notation once we have more experience or in other contexts.
For example, if Go were to run inside a browser it would make
sense for import lines to be URLs of individual files rather than
version control repositories.

• Why isn't my favorite hosting site supported?

Feel free to add it and send us a CL.
See http://golang.org/doc/contribute.html

• How can I use goinstall with a package I commit changes to?

Instead of having goinstall do the checkout automatically,
check out your package yourself into the same location where
goinstall would put it. You can use that as your working copy
and push to the world as appropriate. For example, if you
were working on github.com/hoisie/web.go, you check it out
into $GOROOT/src/pkg/github.com/hoisie/web.go.git
and then edit the files in that directory, pushing back to github
as needed. Goinstall will use the checked-out git repository
instead of trying to download a fresh one.

• What information is sent to and logged by the dashboard?

The source code for both goinstall and the dashboard are
in the Go repository:

$GOROOT/src/cmd/goinstall/download.go
$GOROOT/misc/dashboard/godashboard/package.py

Goinstall sends the package import path.
The dashboard increments a counter and sets the "most recently installed" time.
There is no other reporting or logging. The data recorded by the dashboard
is publicly visible at http://godashboard.appspot.com/package.

-- end documentation

Again, comments welcome, but please post them here rather
than on the code review site.

Thanks.
Russ

For instance (and I like that you used my package as one of your
examples)
import "gomatrix.googlecode.com/hg"

The package name is matrix, which is not reflected in the import
statement. Now, I understand that, when goinstall downloads it, it
will examine what it downloads and know the correct package name for
use in the code.

But consider the case when someone asks how to use matrices in go, and
someone in the IRC channel says, "oh, just use <import statement> at
the top of your file", referring to the import atatement above.

Then this hapless coder has no immediate way to see what the name of
the package is. Maybe "gomatrix"? But no, it's "matrix", which you can
only know by examining the gomatrix source.

This problem doesn't appear so much normally, because it is customary
to name the directory containing a package after that package.

Of course, if this hapless coder is slightly less hapless, he or she
can name the import. This obviously isn't an insurmountable problem.
Just one that can cause some confusion at first for those coders who
have trouble looking stuff up on their own (for whatever reason).

But this seems like a really useful idea, and a very slick way to keep
code-versions up-to-date.

Which makes me think... is there a way to refer to a certain version
of, for example, a googlecode hg project? What if your code works with
an older version, and breaks with a newer? For instance, it is
possible that some clever compiler optimizations would make some of
the methods in gomatrix pointless, and they could be removed. But what
about projects that used those now-removed methods? Being able to
refer to a specific and immutable version would be sweet, and the
default code be the most up-to-date.

- John

> to update a list of what people are using.http://godashboard.appspot.com/package.

> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.

> athttp://godashboard.appspot.com/packages, allowing Go programmers

> Seehttp://golang.org/doc/contribute.html

> is publicly visible athttp://godashboard.appspot.com/package.

It's true; this is a problem. I think this is going to end up
being solved by common sense and convention. Maybe
gomatrix.googlecode.com/hg should call itself package gomatrix,
or maybe it should be in a subdirectory named matrix.
The subdirectory = package name rule is a good general
convention, but there are reasonable exceptions too.
For example, the mimeparse project has source code for
many different languages, and the Go code is in the go
subdirectory. It doesn't bother me that I write import
"mimeparse.googlecode.com/svn/trunk/go" to get package
mimeparse, and I'd much rather get package mimeparse
than package go. It's an open issue, but I think conventions
will go a long way.

Subversion encodes this info in the path, so you can always
import something other than trunk. In hg and git, there's no
direct support for specific versions but goinstall does prefer to
use the branch or tag named "release" over using git HEAD/hg tip.

In general the versioning problem is pretty tricky. If you must use
version 1 of a package and Michael must use version 2 and I want
to use both your packages, then I end up with both copies in my
binary. 6g can handle that; gccgo might need a little help.
Either way it's suboptimal to be linking every version in.
Goinstall nods at the versioning problem by preferring "release"
but it's by no means solved, and I don't know that there's
a magic bullet.

Russ

Sorry, typo. The correct URL is http://godashboard.appspot.com/package.

Russ

import "gospec"
import . "gospec/matchers"

And the gospec package in turn will depend on multiple internal
packages: "gospec/reporting", "gospec/foo" etc. All of those packages
are needed for the library to work.

- How are you handling inclusions in the Makefile (and the NOTEST
option), or would those packages not be recompiled during "all.bash"?

- bitbucket is mentioned in your email, but not explicitly as working.
is it supported?

thanks

downloaded Makefiles are ignored.
the package gets built using a standard-form makefile
generated from the list of files in the directory
(must only be one non-main package in the directory).

the packages are not recompiled by all.bash.

i intended to add it but haven't for lack of a test case.

russ

you'd make those

import "github.com/orfjackal/gospec.git"
import "github.com/ofjackal/gospec.git/matchers"

and then the gospec package would say, internally

import "github.com/orfjackal/gospec.git/reporting"
import "github.com/orfjackal/gospec.git/foo"

when goinstall downloaded gospec for the first import,
it would see the others and build those subpackages
first. when working on the package locally, you'd just
work in the $GOROOT/src/pkg/github.com/orfjackal/gospec.git
directory, which would be a standard git checkout.

russ

What about handling version dependencies? Let's say that I'm
developing a library and I have N projects using that library.

When working on the library itself, I edit the development version of
the library. But at the same time, when working on those N other
projects, they must use the release version of the library. And when
there are incompatible changes to the library, each of the N projects
must use a specific version of the library.

So there is need for being able to specify the version number of the
library. If the version number is part of the import statement, then
when a library is upgraded, one must change the version number in
possibly hundreds of files. This smells like http://en.wikipedia.org/wiki/Shotgun_surgery

And anyways I don't like being forced to put all the sources under
$GOROOT. Also having to install all packages there (as it's done
today) feels like a bad idea. I think that a long-term solution would
be a better build tool, where you can configure the dependencies and
versions in one file per project (similar to Maven et al).

So don't do that. I answered this earlier on the thread:
versioning is a tough problem and we're not trying to solve it today.

Do you have a proposal for how to specify the version number
that a particular file needs without maintaining that information?

Configuration = complexity.
Some may be warranted, but I don't want to just take what everyone
else does, because it's too much manual effort. In contrast, goinstall
is automatic if you're willing to live within the conventions.

I do think there will be a sort of unified go path eventually (today,
godoc, 6g, and 6l each have their own idea of what that means),
but we're not there yet. One step at a time.

Russ

First of all, as someone pointed out, there will probably be naming
conflicts. Github doesn't enforce any kind of unique naming, so you
could have packages overwrite each other. Neither does Google Code.

The second issue is dependency management. Right now most Go packages
are simple and only depend on the core library. However, this is
likely to change as libraries stabilize and become more useful. There
isn't any meta-data on Github or Google code about dependencies.

Why not model the system more closely to rubygems (http://
www.rubygems.org). Rubygems has a structured package format with a
small metadata file, so it can automatically handle versioning and
dependencies.

There could be some central repo of packages on http://godashboard.appspot.com/packages,
so global naming could be enforced. To submit a package to this site,
you would have to conform to the go package format. It would be more
strict, but I feel the system would be more extendable than just using
github or google code.

- Mike

> > possibly hundreds of files. This smells likehttp://en.wikipedia.org/wiki/Shotgun_surgery

But they do. I can't go to Github and create a package named
github.com/hoisie/web.go.git. That's what the package gets
installed and imported as. There might be packages with two
different import paths that both claim to be package web.
That's fine: Go explicitly allows that, and knows enough to keep
them separate.

But there is: the source files contain import statements.
Goinstall looks at them.

For example, goinstall github.com/jacobsa/igo.git/parse fails today,
because parse/parse.go tries to import "igo/set", which doesn't exist.
But if Aaron changed his program to use the goinstall conventions
(and I hope people will), it would import "github.com/jacobsa/igo.git/set",
and goinstall would gladly download and install that first.

Yes, we could set up a CPAN or a RubyGems or whatever,
but that requires a centralized authority giving out import paths
and is more mechanism. Goinstall is zero configuration
without any need for central coordination.

Russ

2. I'm concerned that package source trees would need to include a new
non-Go file whenever external packages are used: The "goinstall" script.

Would the build system be extended to automatically run such scripts?
Not ideal...

Instead, could goinstall be trivially internalized by the language, to
extend "import" to run "goinstall" when the package either is not
present, or is in the form of a goinstall path? (import error -> attempt
goinstall -> retry import) This would permit go programs and packages
to be distributed without an additional goinstall script file.

Another alternative would be to permit goinstall itself to recognize
goinstall commands in the comments at the top of a Go source file, and
execute them in sequence. Then the build system could call goinstall
either on every file, or do the parsing and call it only when goinstall
comments are present.

I'd like to see the default generated Makefile system not be broken by
goinstall dependencies.

-BobC

I'm not sure what you're asking.
How is needing goinstall different from needing 6g?
Goinstall is a single program to be installed with the main distribution.
Packages do not distribute their own copies of goinstall,
and there is no goinstall configuration file.

From the original post:

> I'd like to see the default generated Makefile system not be broken by
> goinstall dependencies.

We're not going to introduce dependencies on
external packages into the main tree, so I don't
think it will be.

Russ

Let's say you are deploying some server which depends on mysqlgo. You
want to make sure it's a stable interface, so ideally you'd want to
reference a specific version, like say 0.5.

Are you planning to support tags, like: "import github.com/mysqlgo/
mysqlgo.git#0.5" ?

Now let's say mysqlgo depends on goyaml. Would the author of mysqlgo
need to import specific version of goyaml for each of his release
tags?

- Mike

It seems to me that the release version of a package should only be dl'd once, after which it is stored on your local machine. Since only one version will be dl'd, changes in version obviously won't be an immediate problem for local builds.  Now, if there is a new release version of a package, you should be able to run goinstall again to get the new release, at which point you'd need to worry about changes in the API. But that isn't any different than what we have now, right? We'd still rely on documentation, change logs etc to guide us.

If goinstall were to run everytime I build, it would be a bad thing cuz my code could suddenly break after a new release, but I don't think it does work that way.

For _releasing_ source code for use with goinstall, we might want to reference a specific version number. But like others have said, this would require a complicated dependency relationship among various versions of various packages.

I think a better solutions is for published source code to remain living and changing, so that a change to a dependency would encourage updates to the dependents. So long as the code stays alive and and doesn't depend on rapidly changing packages, it shouldn't be a big problem. And if you happen to be the first person to notice that a package change breaks a dependent package, then you can report the bug or fix it, and in the meantime use hg, svn, etc to revert to a previous release.

In sum, we don't lose anything bc of goinstall, and we gain a lot of convenience when using stable packages.

Ryanne

- from my phone -

> > 2.  I'm concerned that package source tre...

I am not dismissing the problem. I just think it is difficult and
not any different for goinstall than it is for any other software
system.

Russ

You don't actually deploy anything with ZeroInstall; you just
"execute" a URL. ZeroInstall grabs and caches the package, recursively
following its dependencies. Dependencies denote versioning and
architecture as well. Two users on the same system can fetch different
versions of the same product by using slightly different URL's, and
ZeroInstall fetches any shared dependencies just once (That's the
theory, anyway; this wasn't implemented when I last looked.) The whole
system is based on strong hashing, so naming conflicts are basically
impossible.

Cheers,
Marcelo

P.S.: +1 vote for "goget"

> But there is: the source files contain import statements.

Wow, this is truly awesome. You've taken distributed version control
and extended it to code dependencies in a very intuitive way, very
neat.

--
Cheers,

Sverre Rabbelier

Thanks for the suggestion. I'll take a look at it.
Hash-based naming can solve the "keep things separate" part,
but Go can already do that itself without hashes.
The real issue, it seems, is whether you want a system that
leads to having ten different versions of the same library linked
into a single binary, because as soon as you can say
"I need this specific version", that's where you're heading.

The name goget is great. It's certainly catchier than goinstall.
I'm a little hesitant though, because the meaning is right
for my examples but not the day-to-day use I envision.
I've been writing examples in this thread like

goinstall mimeparse.googlecode.com/svn/trunk/go

to illustrate what the import paths look like, but I don't
expect that to be a common usage pattern. Instead,
I hope people will write their own packages in some local
directory my/pkg with import statements like
import "mimeparse.googlecode.com/svn/trunk/go"
and then run

goinstall my/pkg

which will grab the dependencies as needed.
This way avoids the repetition of having to write the same
string both in an import statement and on a command line.
For that example,

goget my/pkg

doesn't look quite right: the command's not getting my/pkg
at all, because I wrote my/pkg and it's already here.

Russ

In the source control directory, we could add a directory whose name
means something. One directory would be the name of the package (to
make it obvious how to refer to it in code). It can then be prefixed
with version information, or not in the case of edge.

Requires slightly more work by the developer, but only about 30
seconds per release. Pretty reasonable.

gomatrix.googlecode.com/hg/matrix //the "edge" version
gomatrix.googlecode.com/hg/release/matrix //latest stable version,
happens to be v0.6
gomatrix.googlecode.com/hg/v0.6/matrix //previous release that happens
to be release
gomatrix.googlecode.com/hg/v0.5/matrix //previous release that is now
out-of-date

For source controls that support version information in the url, we
can continue using that. But for gomatrix, which uses mercurial, we
can do the above.

Thoughts?

- John

Looks exciting!

Is there a reason why you don't do the build from within goinstall
itself, but instead generate a Makefile? I guess the advantage of
creating a Makefile and passing it off to make means that if the
included makefiles change, then you benefit from those changes for
"free" (since they are included). But on the other hand, you have a
dependency on make, and you have the complexity of both generating
correct input for make and running it properly.

I'm particularly curious about directions of the building back side
because I'm interested in making gotit template packages easy to
build. Teaching goinstall how to build .got files would be great, but
looks particularly complicated if it means both modifying the standard
Makefiles *and* modifying goinstall itself. The gotit approach to
templating (implemented most clearly in gotimports) is very close to
goinstall's approach to downloading: it recognizes special import
formats and builds the appropriate package for you. I think this is
compatible with goinstall, but using the two simultaneously would
obviously require some hacking. gotit isn't ready for prime time
(since noone but me has tried it), but I'd like to make it useful
enough that people *do* try it and try using packages built with it.

David

Someone performing shotgun surgery like that wouldn't be able to blame
goinstall, but instead the fragile dependencies they've been forced to
import.

I think this is great overall- it allows you to be as granular as you
need to be or as generic as you can be with your imports, including
versioning if/when the url-scheme + vcs allows it.

ZeroInstall operates at the dynamic binding level. At load time, it
binds environment variables such as LD_LIBRARY_PATH and PYTHONPATH so
that package A's executable can find package B's shared library. If B
gets an upgrade, the bindings change. If it's a static library you
want to "share", then the ideas in ZeroInstall may not suit unless you
can somehow bring them to bear on the build process.

I see. It's a subtle distinction. Then I will ask why install a
Makefile at all? Why not rely on the Makefile provided with the
package?

> Would making gotit compatible with the standard Makefile format solve
> this issue?

I'm not sure what you mean by that, unless you mean to modify the
"standard Makefile"?
--
David Roundy

The main reason is to force the target install path to
be the one that goinstall thinks it should be. This lets
packages written without knowing about goinstall
or without using a Makefile still be installed with goinstall.

Now that goinstall is known, I hope people will change
their makefiles to install under the goinstall target path.
If the Makefile has a reasonable TARG= then it is
probably okay to use the distributed Makefile.

Russ

I have been thinking about this. The information could be stored in a
special file included in the distribution, eg 'PACKAGE'. It would
contain the package name, description, website URL, logo URL, etc.

I'm also working on a rating system, so people can log into
godashboard and rate packages as good/bad. I'm not sure of the best
rating system though - it's a complex problem. Do you rate a
well-designed package badly if it's buggy? What if it's good but
totally unmaintained? Should individual ratings become less
influential on the overall rating as they age (so the more current
ratings have a greater influence)?

Andrew

goinstall -date='2010-03-15 01:02' package1 package2

This should ignore any commits after the specified date and time (GMT
presumably), so that anyone running goinstall with a -date argument
downloads exactly the same version of each file. (Presuming that the
remote source control systems are sane.)

- Brian

> to update a list of what people are using.http://godashboard.appspot.com/package.

> control systems, and it maintains a list of public Go packages athttp://godashboard.appspot.com/packages.

> athttp://godashboard.appspot.com/packages, allowing Go programmers

> Seehttp://golang.org/doc/contribute.html

> is publicly visible athttp://godashboard.appspot.com/package.

Versioning is a very hard problem, one that goinstall is not
attempt to solve right now. The command you gave may
be fine for one or two packages, but what if package1 needs
an older copy of package2 while package3 needs a newer one?
It gets complicated quickly.

Goinstall does have two decisions meant to help avoid
some versioning pain. The first is that by default goinstall
only downloads new packages; it does not update existing
packages unless you run it with -u. So if you're happy with
what you have, running goinstall to get one more package
won't break the ones you have. The second is that if a
package has a tag or branch named "release", then goinstall
will choose to use that one instead of tip or HEAD. So the
package author can check in as needed and occasionally
change the meaning of "release" when things are stable,
like we do with the Go releases.

These are clearly not a complete solution, but I don't know
what a complete solution would look like. I know that I don't
want to have N copies of every package just to get specific
versions for different dependencies.

Russ

Completely agreed and I'm not attempting to solve it.

Don't do that. For consistency, use the same -date parameter for every
package you download.

The idea is that if you do a clean install in a new directory using
goinstall, you're compiling at "head" of all the Go packages on the
Internet (where "head" is the latest released version of each
package). Normally that should build, but in case it doesn't, we can
temporarily go back in time to get a working build until "head" is
fixed.

This isn't so different from what Google does internally. It's not
intended to be a complete solution to the versioning problem but
should make the current situation more livable.

(That said, I suppose we should wait until the actual problem comes up.)

- Brian

1) Most of the time you just want the latest version
2) Most of the time, the imported package's imports will work with the
latest version
3) If you need to freeze you can just not goinstall
4) If you need older versions you can clone/fork pretty trivially and
just import the clone

I can live with this. I would rather have something under featured
than over featured like CORBA, Java, C++, and J2EE, or any number of
other committee designed abominations.

btw - goinstall updating http://godashboard.appspot.com/packages is a
clever idea. However I think the web site is broken; I'm getting a
blank page.

Andrew

I love the realtime/realworld reporting on packages at godashboard.
Excellent idea. At least until some idiot starts posting false info.

Suggestions for goinstall
---
1) add option for showing clones of installed packages -- helps you to
decide if you should be using a clone instead
2) add option for package searching, and showing clones -- maybe do a
text search of the readmes, or maybe even standardize on the readmes
(markdown anyone?), or standardize embedded package docs in the code
3) add option for showing when clones are pulled back into the main
trunk -- to help decide when you can abandon a bug fix clone and go
back to the main branch

What do you mean by a reasonable TARG=? It seems like it'd be bad to
mention the source in the TARG line, since it means that temporary
forks would have to change the Makefile. e.g. if I make a
github-hosted clone of web.go, I shouldn't have to modify the Makefile
in order for it to goinstall into a different directory.

To be honest, I'm still a little unclear as to how goinstall is
working. Where does it install the Makefile that it generates? When I
run

goinstall github.com/hoisie/web.go

The package is installed in $GOROOT/pkg/github.com/hoisie/web.go.a

But then if I do

cd $GOROOT/src/pkg/github.com/hoisie/web.go
make install

It gets installed as $GOROOT/pkg/web.a

I guess this is because I am now using the package's included
makefile. But where did the generated one go?

One appealing (to me) possibility would be to pass the installation
path to make as an environment variable. You could then check if the
existing Makefile mentions that variable, and if it does, you could
use it, otherwise try generating your own? (and if you keep the check
simple, then the Makefile could mention the variable name in a
comment, in case the build system is more complicated and doesn't
actually use the variable directly in the Makefile (which might itself
call something else that *does* check the install target).
--
David Roundy

That's an interesting argument, but it's only a single line change.
And right now goinstall does behave as you say:
you don't need to include a Makefile at all.

It went to make's standard input.

This fails when goinstall recurses to get dependencies.
There really needs to be a standard algorithm to translate
source location to install path.

Russ

It also won't install any package that I've written, so this question
does seem relevant to me. It looks like in order to allow goinstall
to work, I would need to give up any use of local private packages
(imported with "./foo/bar"). I don't want these packages to be
installed globally, and that seems to be the only option when using
the "standard" makefile (and hence goinstall).

goinstall seems to be going the direction of cabal (see
http://www.haskell.org/cabal/), which is to create an easy
installation tool, which embeds a simplistic build system. The result
(with cabal, but hopefully not with goinstall in the end) is that you
have a tool that works for the most simple of programs, but fails to
work properly for more complicated programs. (On other aspects of why
cabal is bad, you could talk with John Meacham, who is now employed at
Google...)

I don't quite follow. I didn't mean to suggest anything to the
contrary, just that by passing the install path as an environment
variable you could allow developers to create our own Makefiles that
would work with goinstall.

David

There's no such thing as a local private package.
You can pretend that, but 6l needs to find all the
object files when it links everything, so you'd have
to keep those object files around, both for yourself
and any clients of the package. It's much clearer
to install them in the standard place than have
things grubbing around in the build directories.
In general imports of ./anything are for special cases,
not for general use. They're not as private as you think.

We're certainly trying to build something that works for
complicated programs.

If you goinstall a package that has import lines that
refer to other packages, goinstall builds those first,
downloading them if necessary. The environment
variable could only possibly apply to the top level
of the recursion.

Russ

Okay, I hadn't realized this. You're right, that's ugly, and I'll
change my code to avoid doing that.

For applications, rather than packages, is there a problem with local
imports of the ./foo form? I don't like my program build process to
install anything until the build is complete, and would prefer that it
doesn't install anything but the application itself. I guess I could
also use -I and -L to avoid installing packages. Or, of course, I
could put everything in package main, but that doesn't work so nicely
if I want to use shared code for several programs.

I guess the next question is how to compile a multi-package package
(i.e. one that depends on a sub-package) without installing anything.
I guess the answer must be to use -I?

Good! (Then I hope I'm not annoying you by trying to get my
complicated programs to work...)

At the moment, I've modified all my code to use gotgo for their slice
handling. I store the generated packages in each repository, so users
shouldn't need to have gotgo itself installed, and the Makefile
(hopefully) works with this. But currently the generated packages
aren't always suitable for public consumption, since they may refer to
data types by qualified import names, which could clash with another
package's use.

Currently, gotgo's import syntax looks like:

import errorslice "gotgo/slice(os.Error)"

which will import a package for concatenating, importing, filtering,
etc on []os.Error. The ugliness shows up due to issues with package
naming. I don't want template expansion to require looking into the
contents of other imported packages (which potentially haven't yet
been built or installed, and might thus be hard to find), so I require
that "os" above is imported via:

import os "os"

so that I know that os.Error refers to the package imported as "os".
But at this point, I have no guarantee that you didn't write something
like

import os "io"

in which case we couldn't share this package with anyone else, because
the qualified import name isn't unique. So I guess maybe what I need
to do is use a syntax that directly uses import strings, which is much
uglier, and would look something like:

import fooslice "github.com/droundy/gotgo/slice(github.com/droundy/foo.Type)"

Does this sound like a reasonable sort of approach to you? Then the
build process would involve gotgo first looking through the imports
and creating any of those packages, and *then* goinstall looks through
and builds the remainder. And if gotgo were to be accepted into the
core, then goinstall could do both steps.

Hmmm. I'd have thought that the recursive goinstall would simply
redefine that environment variable, since it knows it's building a
different package.
--
David Roundy

It really sounds like you want to install the packages.
At some point we'll have to grapple with having multiple
places where packages can live, and then maybe you
won't be so opposed to installing them if they can go
in some private place?

Just install them.

I don't think it makes sense to change the design of imports or
installation to adapt to gotgo. gotgo is a fun experiment, and I hope
it leads to a nice system, but if so that system can't be import-based,
because:

* I have to write a different import line for every kind of vector I want.
* I can't use a vector of type T inside the package that defines type T,
because the imports would create a cycle.
* I can't use a vector of type t ever, because the imported package
cannot talk about my non-exported type t.

Those all seem like show stoppers to me, and they're fundamental to
any import-based approach to generics. Again, I hope that experience
with gotgo will help us learn more about the eventual solution, but I don't
think it's close enough to being that solution to adjust the import or
install mechanisms to accommodate it.

Russ

The problem is that then I can't easily have two versions of my code
on the same machine. If I compile the experimental one, then it
installs the experimental packages. Then if I compile the bugfix
code, it links with the experimental packages. The only way I see to
solve this is to put all my code into the main package, which doesn't
seem too fit well with the idea of minimal recompiles.

Yay, the first substantive feedback I've gotten on gotgo! :) Thanks!

I agree that I need to have a non-import way to use gotgo, and have
already implemented such a mechanism (not yet committed or pushed).
But it also seems important to include an import-based approach,
otherwise any generic library needs to be recompiled every time
anything that uses it is compiled, which brings us down to C++-level
compile time, so far as I can see.
--
David Roundy

I'm (very slowly) working on a build tool whose goal is to allow just
that: allow building and testing projects without requiring to install
them. Makefiles are just too cumbersome when you've gotten used to the
tools that for example the Java and Ruby communities have.

My biggest problem with Makefiles (which use "$(GOROOT)/src/Make.*")
was that in a multi-package project it's not possible to run tests
(with gotest) which use more than one of the current project's
packages, unless all of them are first installed. This in turn means
that during development, you are required to install broken code (code
which does not pass the tests) hundreds of times per day (when using
TDD), which in turn means that developing a library and using it in
other projects at the same time is very cumbersome, because those
other projects can not depend on a stable version of the library.

The second problem with Makefiles is that you need to manually
maintain a list of all the files which belong to the project, when the
build tool could find them automatically.

I have multiple installations (last release, tip, feature I'm
working on, local package(s)) and switch $PATH by sourcing a
script, e.g.:

$ . ./set_env

when I switch from one to another. set_env says something like:

$ cat set_env
export GOARCH=amd64
export GOOS=darwin
export GOROOT=/Users/giles/Projects/go-release/go
export GOBIN=/Users/giles/Projects/go-release/bin

. "$HOME/.path"
PATH="$GOBIN:$PATH"

I agree with you that this doesn't reduce the number of
compiles a lot, but with the speed of the 6g/8g compilers we
have to do /something/ to restore coffee drinking and mailing
list reading time so it's not too bad in practice. :-/

What is annoying of course is building a package with the
"wrong" environment variables set; for that I really have no
solution except to wrap 'make' in a function that checks $PWD
aginst $GOROOT, which I might yet do but haven't got around to
yet.

Keeping seperate trees around also has the advantage of
letting me use mercurial queues (or branches if I wanted).
The only trees that need the codereview extension in them are
ones I plan to submit code from.

Your milage may vary ....

Giles