This one is really huge... somewhat like the linux source code

Git itself has no problem with that repository size. And Gitlab seems to have at least one 18GB repository

As far as I know, and could be wrong, GitLab uses git-shell which calls git directly from pipeline, which let git self handles everything. Gogits/git implementation has a lot issues. Anyway, I need to get some repository as large as your to test.

I tested git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git and pushing that to gogs also failed.

Linux source was reported failed here before: #121

I think I found the problem, but it will probably take some time to fix it.

@nuss-justin you're blowing up the work! LOL

That's what I call 'boredom at work/home' :-)

Ok, it seems like hashing the sha1 hashes for the offsetValues map is really slow.
To work around that I think there are 2 solutions:

1. Don't read the whole file. I don't know if this would cause any big problems.
2. Store all infos about the repo somewhere it can be accessed faster (for example the database).

The 2. one would be really great I think, but it would complicate everything and take more disk space, because now commits, diffs, etc. would involve the database and we would have to rewrite some parts of the current code, that depends on the current structs for commits, etc. Also every commit would have to be inserted in the database.

On the other side, having everything inside the database would make it much faster. With indexes (and good database settings) access would be really fast.

Another way to do 2. would be to write another git module, that wraps the current gogits/git module and caches everything in the database, for faster accesses. (So it would first try to find something in the database and if it doesn't find anything it would fetch it from disk and then cache it in the database)

If you have other ideas, I'm open for everything :-)

Store all infos about the repo somewhere it can be accessed faster (for example the database).

Cache system is on the plan, but do not PR on this please until we release 0.5.

Be reassured, I don't plan to start something big like this without discussing it with you ^^
I'm working on gogs mostly when im bored at work and big thinks are always difficult to do, when i constantly have to switch between multiple projects.

Haha, you're good!

With new git-shell implementation, I've tested linux source code, takes 5s~12s to load page, would you mind to try dev code and see if have any problem regardless of load time?

Now I managed to push linux.git into gogs. But it took 3.5h! Most of the time gogs serv was calling git cat-file, maybe in modules/git/repo_commit.go

I no longer had any trouble with our in-house ~2GB git repo. Thanks!

Now I managed to push linux.git into gogs. But it took 3.5h! Most of the time gogs serv was calling git cat-file, maybe in modules/git/repo_commit.go

Is it happen when you push through SSH? Or HTTPS?

SSH

I think it would better to keep this issue open, to remind me. :)

Tested with current dev version (0.5.5.1014 Beta ).

Trying to push linux github.com/torvalds/linux to gogs.
Taking 4 hour still not finished after 6 hours failed.

Reading xorm.log, it is still working with hook_task and the log interval is one minute per log entry. (Not related)

Server is a Digitalocean 1 core / 1G droplet.

@tobyzxj Using HTTP or SSH?

@unknwon SSH!

I'd like to add another data point:

We maintain a 1.2 GB repository with less than 100 commits (manually tracking a vendor).

A first push to Gogs 0.6.1 via https works, but trying to view the repository in the web interface afterwards results in a HTTP 500:

[Macaron] Completed /vw-it/test 500 Internal Server Error in 3.783941ms
2015/06/12 14:35:42 [repo.go:281 func·008()] [E] GetTags: 

Only after we restart Gogs web, the error goes away. Subsequent (smaller) pushes seem to work without problem. How tags fit in this (GetTags from the log), I don't know, this particular repository doesn't have any.

Everyone on this thread please checkout develop branch, I've tested with mono project with over 100k commits.

Tested with https://github.com/torvalds/linux with SSH as well.

Close as fixed.

Is it required to use SSH to fix this problem? We are having the same issue pushing 1.4GB repo with 1255 commits over http.

@Longshot123

Is it required to use SSH to fix this problem?

Yes...