Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limit number of goroutines started when creating units #1149

Closed
cezarsa opened this issue Mar 25, 2015 · 1 comment
Closed

Limit number of goroutines started when creating units #1149

cezarsa opened this issue Mar 25, 2015 · 1 comment
Assignees
@fsouza
Labels
enhancement
Milestone
0.11.0

Comments

@cezarsa
Copy link
Member

cezarsa commented Mar 25, 2015

Every operation on applications that demand the (re-)creation of units (deploy, restart, start, env-set, env-unset) will start 1 goroutine for each unit.

This is a problem depending on the number of existing units in an application, in some tests we were able to starve the api process when deploying 300+ units due to the high number of threads (one for each goroutine due to being blocked on I/O).

A solution for this problem is having a maximum number of goroutines running in a pool and using channels to send workloads to them.
@andrewsmedina andrewsmedina added this to the 0.11.0 milestone Mar 25, 2015
@fsouza fsouza self-assigned this Mar 30, 2015
fsouza pushed a commit that referenced this issue Mar 31, 2015
provision/docker: limit the amount of goroutines used for containers
21055bb 
When using numbers like 1000 containers, we do have some issues with it.
We're now introducing a setting for customizing the maximum amount of
woekrs. This isn't a pure "worker" architecture, as some workers may
work harder than others, because we just split the work across then
equally.

Related to #1149. unit-add doesn't use this code, and I also need to
document it.
fsouza pushed a commit that referenced this issue Mar 31, 2015
docs/reference/config: document the docker:max-workers setting
c5dfbc5 
Related to #1149.
fsouza pushed a commit that referenced this issue Apr 1, 2015
provision/docker: use workers in unit-add
9a90e4c 
Related to #1149.
@fsouza
Copy link
Contributor

fsouza commented Apr 1, 2015

It's implemented and tested at master, we now have a new setting for defining the maximum amount of workers.

@fsouza fsouza closed this as completed Apr 1, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fsouza fsouza

Labels
enhancement
Projects
None yet
Milestone
0.11.0
Development

No branches or pull requests
3 participants
@cezarsa @andrewsmedina @fsouza