Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make "All Websites Dashboard" usable with 20,000+ websites #3124

Closed
diosmosis opened this issue Apr 24, 2012 · 8 comments
Closed

Make "All Websites Dashboard" usable with 20,000+ websites #3124

diosmosis opened this issue Apr 24, 2012 · 8 comments
Assignees
@diosmosis
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo. Critical Indicates the severity of an issue is very critical and the issue has a very high priority.
Milestone
Long term

Comments

@diosmosis
Copy link
Member

Currently the All Websites dashboard will work well when there are hundreds or even thousands of websites, but after a point it becomes unusable. It should, however, be able to scale to any number of websites.

There are currently many barriers to scaling this area, including:

  • Lack of server-side pagination means all site data must be queried at once.
  • The design of the Array Archive types will result in unnecessary object creation (in this case several objects per website).
  • There's no way to archive metrics for all websites, so summary & evolution data has to be calculated when queried.

These issues will be addressed in this ticket.
@mattab
Copy link
Member

mattab commented Aug 7, 2012

What is the status of this ticket, and how much work do you estimate is left to finish it?

@diosmosis
Copy link
Member Author

Still quite a bit of work left. I have to finish the refactoring of the Archive types, then commit the changes, piece by piece. One of the other two changes may not be as hard to implement. I'll send you a patch of what I've done so far so you can take a look (w/o having to review it).

@mattab
Copy link
Member

mattab commented Mar 11, 2013

Please contact us if you are interested in this fix!

@pebosi
Copy link
Contributor

pebosi commented May 24, 2013

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

@diosmosis
Copy link
Member Author

Replying to pebosi:

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

Hi @pebosi, the dashboard should be usable for ~1000 websites on any period except range, but ONLY if browser-initiated archiving is disabled (controlled by the first setting in the General Settings admin page). Is this disabled for you?

@diosmosis
Copy link
Member Author

Fixed by Archive.php refactor. When archiving is disabled and there are 20,000 sites each w/ visits, MultiSites.getAll takes 12-14s to return the result on my laptop. Memory usage is at 118 MB.

@anonymous-matomo-user
Copy link

In 3a3517e: Refs #3124, add benchmark for MultiSites.getAll and benchmark fixture that adds 20,000 sites to the DB then tracks one pageview per site.

@mattab
Copy link
Member

mattab commented Jul 6, 2013

Awesome Benaka, truly beautiful!

@diosmosis diosmosis added this to the 2.x - The Great Piwik 2.x Backlog milestone Jul 8, 2014
@diosmosis diosmosis self-assigned this Jul 8, 2014
@diosmosis diosmosis mentioned this issue Sep 3, 2012
Closed
This was referenced Oct 19, 2012
Closed
Open
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
Refs matomo-org#3124, add benchmark for MultiSites.getAll and benchma…
3a3517e 
…rk fixture that adds 20,000 sites to the DB then tracks one pageview per site.
@mattab mattab mentioned this issue Aug 3, 2014
Closed
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@diosmosis diosmosis

Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo. Critical Indicates the severity of an issue is very critical and the issue has a very high priority.
Projects
None yet
Milestone
Long term
Development

No branches or pull requests
4 participants
@diosmosis @pebosi @mattab @anonymous-matomo-user