SlideShare a Scribd company logo

Building Recommender Systems: Approaches and Challenges

•
•
B
blueace

The document discusses building a recommender system using collaborative filtering approaches. It describes collecting usage and rating data, calculating item-item and user-user similarities, making predictions for unknown values using k-nearest neighbors, and evaluating the system using measures like precision, recall and root mean squared error. Implementation details like programming languages, databases and cloud infrastructure are also summarized.

Technology
1 of 37
Download to read offline
Coen Stevens Lead Recommendation Engineer
How to build a recommender system? Wakoopa use case
Mission: Discover software & games
Software tracker Windows Mac Linux
Your profile
Updates
Software pages
Recommendations
Building a recommender system Approach and challenges
Data what do we have? Usage (implicit) Ratings (explicit) vs. • • Noisy Accurate • • Only positive feedback Positive and negative feedback • • Easy to collect Hard to collect
Data what do we use? • Active users (Tracker activity in the past month): ~9.000 • Actively used software items (in the past month): ~10.000 • We calculate recommendations for each OS together with Web applications separately
Recommender system methods Collaborative recommendations: The user will be recommended items that people with similar tastes and preferences liked (used) in the past • Item-based collaborative filtering • User-based collaborative filtering (we only use for calculating user similarities to find people like you) • Combining both methods
Item-Based Collaborative Filtering User software usage matrix Software items 220 90 180 22 280 12 42 80 Users 175 210 210 45 165 35 195 13 25 100 50 185 35 190 60 65 185
User software usage matrix [0, 1] Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Users 1 1 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1
How do we predict the probability that I would like to use GMail? Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 ? Users 1 1 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1
Calculate the similarities between Gmail and the other software items. Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Users 1 1 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1 Cosine Similarity(Firefox, Gmail)
Calculate the similarities between Gmail and the other software items. Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Users 1 1 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1 Cosine Similarity(Firefox, Gmail)
Calculate the similarities between Gmail and the other software items. Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Popularity correction, Users 1 1 0 1 0 1 0 we put less trust 1 0 1 1 1 1 0 in popular software 0 1 1 1 0 1 1 0 1 0 1 0 0 1 Cosine Similarity(Firefox, Gmail)
Item-item correlation matrix 1 0.1 0.6 0.1 0.1 0.1 0.7 0.2 1 0.8 0.5 0.8 0.1 0.9 0.1 0.6 1 0.5 0.7 0.2 0.3 0.2 0.6 0.4 1 0.8 0.2 0.3 0.5 0.4 0.4 0.4 1 0.1 0.2 0.5 0.5 0.3 0.5 0.3 1 0.3 0.2 0.6 0.3 0.8 0.7 0.7 1
Item-item correlation matrix Gmail similarities 0.6 1 0.1 0.6 0.1 0.1 0.1 0.7 0.8 0.2 1 0.8 0.5 0.8 0.1 0.9 0.4 0.1 0.6 1 0.5 0.7 0.2 0.3 0.4 0.2 0.6 0.4 1 0.8 0.2 0.3 0.3 0.5 0.4 0.4 0.4 1 0.1 0.2 0.3 0.5 0.5 0.3 0.5 0.3 1 0.3 0.2 0.6 0.3 0.8 0.7 0.7 1
K-nearest neighbor approach Gmail similarities • Performance vs quality 0.6 • We take only the ‘K’ most similar items (say 4) 0.8 • Space complexity: O(m + Kn) 0.4 • 0.4 Computational complexity: O(m + n²) 0.3 0.3
Calculate the predicted value for Gmail Gmail similarities User usage 1 0.6 1 0.8 1 0.4 0.4 1
Calculate the predicted value for Gmail Gmail similarities User usage 0.9 0.6 Usage correction, 0.8 0.8 more usage results in a higher score [0,1] 0.6 0.4 0.4 0.2
Calculate the predicted value for Gmail Gmail similarities User usage 0.9 0.6 0.8 0.8 0.6 0.4 0.4 0.2 (0.6 * 0.9) + (0.8 * 0.8) + (0.4 * 0.6) = 0.82 0.6 + 0.8 + 0.4 + 0.4
Calculate the predicted value for Gmail • User feedback Gmail similarities User usage • Contacts usage 0.9 0.6 • Commercial vs Free 0.8 0.8 0.6 0.4 0.4 0.2 (0.6 * 0.9) + (0.8 * 0.8) + (0.4 * 0.6) = 0.82 0.6 + 0.8 + 0.4 + 0.4
Calculate all unknown values and show the Top-N recommendations to each user Software items ? ? ? 1 1 1 1 ?1?? 1 1 1 ?1?1? Users 1 1 ?1111? 1 ?111?11 ?1?1??1
Explainability Why did I get this recommendation? • Overlap between the item’s (K) neighbors and your usage
User-Based Collaborative Filtering Finding people like you 1 1 0 1 0 1 0 1 1 1 0 1 0 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 Cosine Similarity(Coen, Menno) 0 1 1 1 0 1 1 0 1 0 1 0 0 1
Applying inverse user frequency log(n/ni): ni is the number of users that uses item i and n is the total number of users in the database 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0 0.8 0 0 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0.4 0.8 0.4 0 Cosine Similarity(Coen, Menno) 0 0.2 0.6 0.4 0 0.4 0.2 0 0.2 0 0.4 0 0 0.2 The fact that you both use Textmate tells you more than when you both use firefox
0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0 0.8 0 0 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0.4 0.8 0.4 0 Cosine Similarity(Coen, Menno) 0 0.2 0.6 0.4 0 0.4 0.2 0 0.2 0 0.4 0 0 0.2
User-user correlation matrix 1 0.8 0.6 0.5 0.7 0.2 0.8 1 0.4 0.7 0.5 0.5 0.6 0.4 1 0.4 0.9 0.1 0.5 0.8 0.4 1 0.6 0.4 0.8 0.5 0.9 0.6 1 0.2 0.2 0.5 0.1 0.4 0.2 1
Performance measure for success • Cross-validation: Train-Test split (80-20) • Precision and Recall: - precision = size(hit set) / size(total given recs) - recall = size(hit set) / size(test set) • Root mean squared error (RMSE)
Implementation • Ruby Enterprise Edition (garbage collection) • MySQL database • Built our own c-libraries • Amazon EC2: - Low cost - Flexibility - Ease of use • Open source
Future challenges • What is the best algorithm for Wakoopa? (or you) • Reducing space-time complexity (scalability): - Parallelization (Clojure) - Distributed computing (Hadoop)
1 evening, 3 speakers, 100 developers www.recked.org

Recommended

Boston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender SystemsBoston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender SystemsJames Kirk
 
Recommender system introduction
Recommender system introductionRecommender system introduction
Recommender system introductionLiang Xiang
 
Recommender system algorithm and architecture
Recommender system algorithm and architectureRecommender system algorithm and architecture
Recommender system algorithm and architectureLiang Xiang
 
Recent advances in deep recommender systems
Recent advances in deep recommender systemsRecent advances in deep recommender systems
Recent advances in deep recommender systemsNAVER Engineering
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsGirish Khanzode
 
Past, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry PerspectivePast, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry PerspectiveJustin Basilico
 
An introduction to Recommender Systems
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender SystemsDavid Zibriczky
 
Introduction to Recommendation Systems
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation SystemsTrieu Nguyen
 

Recommended

Boston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender SystemsBoston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender SystemsJames Kirk
 
Recommender system introduction
Recommender system introductionRecommender system introduction
Recommender system introductionLiang Xiang
 
Recommender system algorithm and architecture
Recommender system algorithm and architectureRecommender system algorithm and architecture
Recommender system algorithm and architectureLiang Xiang
 
Recent advances in deep recommender systems
Recent advances in deep recommender systemsRecent advances in deep recommender systems
Recent advances in deep recommender systemsNAVER Engineering
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsGirish Khanzode
 
Past, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry PerspectivePast, Present & Future of Recommender Systems: An Industry Perspective
Past, Present & Future of Recommender Systems: An Industry PerspectiveJustin Basilico
 
An introduction to Recommender Systems
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender SystemsDavid Zibriczky
 
Introduction to Recommendation Systems
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation SystemsTrieu Nguyen
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsFrancesco Casalegno
 
Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Ernesto Mislej
 
Recommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filteringRecommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filteringViet-Trung TRAN
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsCarlos Castillo (ChaTo)
 
Building a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engineBuilding a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engineNYC Predictive Analytics
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender SystemsJustin Basilico
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain
 
Recommender systems using collaborative filtering
Recommender systems using collaborative filteringRecommender systems using collaborative filtering
Recommender systems using collaborative filteringD Yogendra Rao
 
Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial Alexandros Karatzoglou
 
Matrix Factorization Techniques For Recommender Systems
Matrix Factorization Techniques For Recommender SystemsMatrix Factorization Techniques For Recommender Systems
Matrix Factorization Techniques For Recommender SystemsLei Guo
 
Tutorial: Context In Recommender Systems
Tutorial: Context In Recommender SystemsTutorial: Context In Recommender Systems
Tutorial: Context In Recommender SystemsYONG ZHENG
 
Recommendation System Explained
Recommendation System ExplainedRecommendation System Explained
Recommendation System ExplainedCrossing Minds
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at NetflixLinas Baltrunas
 
Recommender system
Recommender systemRecommender system
Recommender systemNilotpal Pramanik
 
Recommender systems
Recommender systemsRecommender systems
Recommender systemsTamer Rezk
 
Recommendation system
Recommendation systemRecommendation system
Recommendation systemAkshat Thakar
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveJustin Basilico
 
Recommendation Systems Basics
Recommendation Systems BasicsRecommendation Systems Basics
Recommendation Systems BasicsJarin Tasnim Khan
 
Recommendation system
Recommendation systemRecommendation system
Recommendation systemRishabh Mehta
 
Design of recommender systems
Design of recommender systemsDesign of recommender systems
Design of recommender systemsRashmi Sinha
 
How to build a Recommender System
How to build a Recommender SystemHow to build a Recommender System
How to build a Recommender SystemVõ Duy Tuấn
 

More Related Content

What's hot

Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Ernesto Mislej
 
Recommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filteringRecommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filteringViet-Trung TRAN
 
Building a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engineBuilding a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engineNYC Predictive Analytics
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender SystemsJustin Basilico
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain
 
Recommender systems using collaborative filtering
Recommender systems using collaborative filteringRecommender systems using collaborative filtering
Recommender systems using collaborative filteringD Yogendra Rao
 
Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial Alexandros Karatzoglou
 
Matrix Factorization Techniques For Recommender Systems
Matrix Factorization Techniques For Recommender SystemsMatrix Factorization Techniques For Recommender Systems
Matrix Factorization Techniques For Recommender SystemsLei Guo
 
Tutorial: Context In Recommender Systems
Tutorial: Context In Recommender SystemsTutorial: Context In Recommender Systems
Tutorial: Context In Recommender SystemsYONG ZHENG
 
Recommendation System Explained
Recommendation System ExplainedRecommendation System Explained
Recommendation System ExplainedCrossing Minds
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at NetflixLinas Baltrunas
 
Recommender systems
Recommender systemsRecommender systems
Recommender systemsTamer Rezk
 
Recommendation system
Recommendation systemRecommendation system
Recommendation systemAkshat Thakar
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveJustin Basilico
 
Recommendation Systems Basics
Recommendation Systems BasicsRecommendation Systems Basics
Recommendation Systems BasicsJarin Tasnim Khan
 
Recommendation system
Recommendation systemRecommendation system
Recommendation systemRishabh Mehta
 

What's hot (20)

Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011Recommender Systems! @ASAI 2011
Recommender Systems! @ASAI 2011
 
Recommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filteringRecommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filtering
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Building a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engineBuilding a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engine
 
Past, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
 
Recommender systems using collaborative filtering
Recommender systems using collaborative filteringRecommender systems using collaborative filtering
Recommender systems using collaborative filtering
 
Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial
 
Matrix Factorization Techniques For Recommender Systems
Matrix Factorization Techniques For Recommender SystemsMatrix Factorization Techniques For Recommender Systems
Matrix Factorization Techniques For Recommender Systems
 
Tutorial: Context In Recommender Systems
Tutorial: Context In Recommender SystemsTutorial: Context In Recommender Systems
Tutorial: Context In Recommender Systems
 
Recommendation System Explained
Recommendation System ExplainedRecommendation System Explained
Recommendation System Explained
 
Context Aware Recommendations at Netflix
Context Aware Recommendations at NetflixContext Aware Recommendations at Netflix
Context Aware Recommendations at Netflix
 
Recommender system
Recommender systemRecommender system
Recommender system
 
Recommender systems
Recommender systemsRecommender systems
Recommender systems
 
Recommendation system
Recommendation systemRecommendation system
Recommendation system
 
Recent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix PerspectiveRecent Trends in Personalization: A Netflix Perspective
Recent Trends in Personalization: A Netflix Perspective
 
Recommendation Systems Basics
Recommendation Systems BasicsRecommendation Systems Basics
Recommendation Systems Basics
 
Recommendation system
Recommendation systemRecommendation system
Recommendation system
 

Viewers also liked

Design of recommender systems
Design of recommender systemsDesign of recommender systems
Design of recommender systemsRashmi Sinha
 
How to build a Recommender System
How to build a Recommender SystemHow to build a Recommender System
How to build a Recommender SystemVõ Duy Tuấn
 
Recommendation system
Recommendation system Recommendation system
Recommendation system Vikrant Arya
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsT212
 
Collaborative Filtering Recommendation System
Collaborative Filtering Recommendation SystemCollaborative Filtering Recommendation System
Collaborative Filtering Recommendation SystemMilind Gokhale
 
Recommender Engines Seminar Paper
Recommender Engines Seminar PaperRecommender Engines Seminar Paper
Recommender Engines Seminar PaperThomas Hess
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsLior Rokach
 
How to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based FilteringHow to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based FilteringVõ Duy Tuấn
 
genetic algorithm based music recommender system
genetic algorithm based music recommender systemgenetic algorithm based music recommender system
genetic algorithm based music recommender systemneha pevekar
 
Buidling large scale recommendation engine
Buidling large scale recommendation engineBuidling large scale recommendation engine
Buidling large scale recommendation engineKeeyong Han
 
How to Build a Recommendation Engine on Spark
How to Build a Recommendation Engine on SparkHow to Build a Recommendation Engine on Spark
How to Build a Recommendation Engine on SparkCaserta
 
Wakoopa Recommendation Engine on AWS
Wakoopa Recommendation Engine on AWSWakoopa Recommendation Engine on AWS
Wakoopa Recommendation Engine on AWSMenno van der Sman
 
Impact of web 2.0 on evaluation and select solutions
Impact of web 2.0 on evaluation and select solutionsImpact of web 2.0 on evaluation and select solutions
Impact of web 2.0 on evaluation and select solutionssarvenaz arianfar
 
Interaction Design Patterns in Recommender Systems
Interaction Design Patterns in Recommender SystemsInteraction Design Patterns in Recommender Systems
Interaction Design Patterns in Recommender SystemsUniversity of Bergen
 
Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...
Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...
Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...Lucidworks
 
Recommender Systems in E-Commerce
Recommender Systems in E-CommerceRecommender Systems in E-Commerce
Recommender Systems in E-CommerceRoger Chen
 
H2O World - Intro to Data Science with Erin Ledell
H2O World - Intro to Data Science with Erin LedellH2O World - Intro to Data Science with Erin Ledell
H2O World - Intro to Data Science with Erin LedellSri Ambati
 
Recommender system
Recommender systemRecommender system
Recommender systemJie Jin
 

Viewers also liked (19)

Design of recommender systems
Design of recommender systemsDesign of recommender systems
Design of recommender systems
 
How to build a Recommender System
How to build a Recommender SystemHow to build a Recommender System
How to build a Recommender System
 
Recommendation system
Recommendation system Recommendation system
Recommendation system
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Collaborative Filtering Recommendation System
Collaborative Filtering Recommendation SystemCollaborative Filtering Recommendation System
Collaborative Filtering Recommendation System
 
Recommender Engines Seminar Paper
Recommender Engines Seminar PaperRecommender Engines Seminar Paper
Recommender Engines Seminar Paper
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
How to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based FilteringHow to Build Recommender System with Content based Filtering
How to Build Recommender System with Content based Filtering
 
genetic algorithm based music recommender system
genetic algorithm based music recommender systemgenetic algorithm based music recommender system
genetic algorithm based music recommender system
 
Buidling large scale recommendation engine
Buidling large scale recommendation engineBuidling large scale recommendation engine
Buidling large scale recommendation engine
 
How to Build a Recommendation Engine on Spark
How to Build a Recommendation Engine on SparkHow to Build a Recommendation Engine on Spark
How to Build a Recommendation Engine on Spark
 
Wakoopa Recommendation Engine on AWS
Wakoopa Recommendation Engine on AWSWakoopa Recommendation Engine on AWS
Wakoopa Recommendation Engine on AWS
 
Impact of web 2.0 on evaluation and select solutions
Impact of web 2.0 on evaluation and select solutionsImpact of web 2.0 on evaluation and select solutions
Impact of web 2.0 on evaluation and select solutions
 
Interaction Design Patterns in Recommender Systems
Interaction Design Patterns in Recommender SystemsInteraction Design Patterns in Recommender Systems
Interaction Design Patterns in Recommender Systems
 
Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...
Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...
Anyone Can Build A Recommendation Engine With Solr: Presented by Doug Turnbul...
 
Recommender Systems in E-Commerce
Recommender Systems in E-CommerceRecommender Systems in E-Commerce
Recommender Systems in E-Commerce
 
recommender_systems
recommender_systemsrecommender_systems
recommender_systems
 
H2O World - Intro to Data Science with Erin Ledell
H2O World - Intro to Data Science with Erin LedellH2O World - Intro to Data Science with Erin Ledell
H2O World - Intro to Data Science with Erin Ledell
 
Recommender system
Recommender systemRecommender system
Recommender system
 

Similar to Building Recommender Systems: Approaches and Challenges

DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)Wooga
 
Wakoopa Recommendations Engine on AWS
Wakoopa Recommendations Engine on AWSWakoopa Recommendations Engine on AWS
Wakoopa Recommendations Engine on AWSAmazon Web Services
 
Oslo Schibsted Performance Gathering
Oslo Schibsted Performance GatheringOslo Schibsted Performance Gathering
Oslo Schibsted Performance GatheringAlmudena Vivanco
 
Spil games konrad
Spil games konradSpil games konrad
Spil games konradBigDataExpo
 
Building native apps with web components
Building native apps with web componentsBuilding native apps with web components
Building native apps with web componentsDenis Radin
 
Keeping Swift Apps Small
Keeping Swift Apps SmallKeeping Swift Apps Small
Keeping Swift Apps SmallBruno Rocha
 
IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018
IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018
IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018Amazon Web Services
 
Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018
Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018
Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018Amazon Web Services
 
Our Favorite Admin Features in Cognos Analytics 11.1
Our Favorite Admin Features in Cognos Analytics 11.1Our Favorite Admin Features in Cognos Analytics 11.1
Our Favorite Admin Features in Cognos Analytics 11.1Senturus
 
PredictionIO - Building Applications That Predict User Behavior Through Big D...
PredictionIO - Building Applications That Predict User Behavior Through Big D...PredictionIO - Building Applications That Predict User Behavior Through Big D...
PredictionIO - Building Applications That Predict User Behavior Through Big D...predictionio
 
Introduction to GluonNLP
Introduction to GluonNLPIntroduction to GluonNLP
Introduction to GluonNLPApache MXNet
 
systemd and configuration management
systemd and configuration managementsystemd and configuration management
systemd and configuration managementJulien Pivotto
 
Presentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association EventPresentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association EventBen Cheng
 
Feature Based Opinion Mining from Amazon Reviews
Feature Based Opinion Mining from Amazon ReviewsFeature Based Opinion Mining from Amazon Reviews
Feature Based Opinion Mining from Amazon ReviewsRavi Kiran Holur Vijay
 
الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...
الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...
الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...Dr. Khaled Bakro
 
Static Software Watermarking
Static Software WatermarkingStatic Software Watermarking
Static Software WatermarkingJames Hamilton
 

Similar to Building Recommender Systems: Approaches and Challenges (20)

Open Social Tech Talk Beijing
Open Social Tech Talk BeijingOpen Social Tech Talk Beijing
Open Social Tech Talk Beijing
 
DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)DevOps goes Mobile (daho.am)
DevOps goes Mobile (daho.am)
 
Wakoopa Recommendations Engine on AWS
Wakoopa Recommendations Engine on AWSWakoopa Recommendations Engine on AWS
Wakoopa Recommendations Engine on AWS
 
Oslo Schibsted Performance Gathering
Oslo Schibsted Performance GatheringOslo Schibsted Performance Gathering
Oslo Schibsted Performance Gathering
 
Spil games konrad
Spil games konradSpil games konrad
Spil games konrad
 
Building native apps with web components
Building native apps with web componentsBuilding native apps with web components
Building native apps with web components
 
Keeping Swift Apps Small
Keeping Swift Apps SmallKeeping Swift Apps Small
Keeping Swift Apps Small
 
IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018
IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018
IoT Analytics Workshop (IOT314-R1) - AWS re:Invent 2018
 
Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018
Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018
Customer Showcase for AWS IoT Analytics (IOT219) - AWS re:Invent 2018
 
Our Favorite Admin Features in Cognos Analytics 11.1
Our Favorite Admin Features in Cognos Analytics 11.1Our Favorite Admin Features in Cognos Analytics 11.1
Our Favorite Admin Features in Cognos Analytics 11.1
 
PredictionIO - Building Applications That Predict User Behavior Through Big D...
PredictionIO - Building Applications That Predict User Behavior Through Big D...PredictionIO - Building Applications That Predict User Behavior Through Big D...
PredictionIO - Building Applications That Predict User Behavior Through Big D...
 
UCLA HACKU'11
UCLA HACKU'11UCLA HACKU'11
UCLA HACKU'11
 
Introduction to GluonNLP
Introduction to GluonNLPIntroduction to GluonNLP
Introduction to GluonNLP
 
Windows10TipsandTricksBooklet
Windows10TipsandTricksBookletWindows10TipsandTricksBooklet
Windows10TipsandTricksBooklet
 
systemd and configuration management
systemd and configuration managementsystemd and configuration management
systemd and configuration management
 
Presentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association EventPresentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association Event
 
Feature Based Opinion Mining from Amazon Reviews
Feature Based Opinion Mining from Amazon ReviewsFeature Based Opinion Mining from Amazon Reviews
Feature Based Opinion Mining from Amazon Reviews
 
IoT State of the Union
IoT State of the UnionIoT State of the Union
IoT State of the Union
 
الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...
الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...
الفصل الثالث البرمجيات التطبيقية - د. خالد بكرو Application Soft - Dr. Khale...
 
Static Software Watermarking
Static Software WatermarkingStatic Software Watermarking
Static Software Watermarking
 

More from blueace

Research & Tracking via een Social Network
Research & Tracking via een Social NetworkResearch & Tracking via een Social Network
Research & Tracking via een Social Networkblueace
 
Enhanced research via software & web tracking
Enhanced research via software & web trackingEnhanced research via software & web tracking
Enhanced research via software & web trackingblueace
 
(Dutch) CSN: social network succes
(Dutch) CSN: social network succes(Dutch) CSN: social network succes
(Dutch) CSN: social network succesblueace
 
Recommendations 101
Recommendations 101Recommendations 101
Recommendations 101blueace
 
(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0
(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0
(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0blueace
 
Roomware - The operating system for interactive spaces
Roomware - The operating system for interactive spacesRoomware - The operating system for interactive spaces
Roomware - The operating system for interactive spacesblueace
 
Wakoopa at The Next Web 2008
Wakoopa at The Next Web 2008Wakoopa at The Next Web 2008
Wakoopa at The Next Web 2008blueace
 
How we did RoR in Wakoopa
How we did RoR in WakoopaHow we did RoR in Wakoopa
How we did RoR in Wakoopablueace
 

More from blueace (8)

Research & Tracking via een Social Network
Research & Tracking via een Social NetworkResearch & Tracking via een Social Network
Research & Tracking via een Social Network
 
Enhanced research via software & web tracking
Enhanced research via software & web trackingEnhanced research via software & web tracking
Enhanced research via software & web tracking
 
(Dutch) CSN: social network succes
(Dutch) CSN: social network succes(Dutch) CSN: social network succes
(Dutch) CSN: social network succes
 
Recommendations 101
Recommendations 101Recommendations 101
Recommendations 101
 
(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0
(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0
(Dutch) Web 2.0 Succesfactoren @ Overheid 2.0
 
Roomware - The operating system for interactive spaces
Roomware - The operating system for interactive spacesRoomware - The operating system for interactive spaces
Roomware - The operating system for interactive spaces
 
Wakoopa at The Next Web 2008
Wakoopa at The Next Web 2008Wakoopa at The Next Web 2008
Wakoopa at The Next Web 2008
 
How we did RoR in Wakoopa
How we did RoR in WakoopaHow we did RoR in Wakoopa
How we did RoR in Wakoopa
 

Recently uploaded

Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 

Recently uploaded (20)

Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 

Building Recommender Systems: Approaches and Challenges

  • 1.
  • 3. How to build a recommender system? Wakoopa use case
  • 10. Building a recommender system Approach and challenges
  • 11. Data what do we have? Usage (implicit) Ratings (explicit) vs. • • Noisy Accurate • • Only positive feedback Positive and negative feedback • • Easy to collect Hard to collect
  • 12. Data what do we use? • Active users (Tracker activity in the past month): ~9.000 • Actively used software items (in the past month): ~10.000 • We calculate recommendations for each OS together with Web applications separately
  • 13. Recommender system methods Collaborative recommendations: The user will be recommended items that people with similar tastes and preferences liked (used) in the past • Item-based collaborative filtering • User-based collaborative filtering (we only use for calculating user similarities to find people like you) • Combining both methods
  • 14. Item-Based Collaborative Filtering User software usage matrix Software items 220 90 180 22 280 12 42 80 Users 175 210 210 45 165 35 195 13 25 100 50 185 35 190 60 65 185
  • 15. User software usage matrix [0, 1] Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Users 1 1 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1
  • 16. How do we predict the probability that I would like to use GMail? Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 ? Users 1 1 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1
  • 17. Calculate the similarities between Gmail and the other software items. Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Users 1 1 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1 Cosine Similarity(Firefox, Gmail)
  • 18. Calculate the similarities between Gmail and the other software items. Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Users 1 1 0 1 0 1 0 1 0 1 1 1 1 0 0 1 1 1 0 1 1 0 1 0 1 0 0 1 Cosine Similarity(Firefox, Gmail)
  • 19. Calculate the similarities between Gmail and the other software items. Software items 1 1 0 1 0 1 0 1 1 1 0 1 0 0 Popularity correction, Users 1 1 0 1 0 1 0 we put less trust 1 0 1 1 1 1 0 in popular software 0 1 1 1 0 1 1 0 1 0 1 0 0 1 Cosine Similarity(Firefox, Gmail)
  • 20. Item-item correlation matrix 1 0.1 0.6 0.1 0.1 0.1 0.7 0.2 1 0.8 0.5 0.8 0.1 0.9 0.1 0.6 1 0.5 0.7 0.2 0.3 0.2 0.6 0.4 1 0.8 0.2 0.3 0.5 0.4 0.4 0.4 1 0.1 0.2 0.5 0.5 0.3 0.5 0.3 1 0.3 0.2 0.6 0.3 0.8 0.7 0.7 1
  • 21. Item-item correlation matrix Gmail similarities 0.6 1 0.1 0.6 0.1 0.1 0.1 0.7 0.8 0.2 1 0.8 0.5 0.8 0.1 0.9 0.4 0.1 0.6 1 0.5 0.7 0.2 0.3 0.4 0.2 0.6 0.4 1 0.8 0.2 0.3 0.3 0.5 0.4 0.4 0.4 1 0.1 0.2 0.3 0.5 0.5 0.3 0.5 0.3 1 0.3 0.2 0.6 0.3 0.8 0.7 0.7 1
  • 22. K-nearest neighbor approach Gmail similarities • Performance vs quality 0.6 • We take only the ‘K’ most similar items (say 4) 0.8 • Space complexity: O(m + Kn) 0.4 • 0.4 Computational complexity: O(m + n²) 0.3 0.3
  • 23. Calculate the predicted value for Gmail Gmail similarities User usage 1 0.6 1 0.8 1 0.4 0.4 1
  • 24. Calculate the predicted value for Gmail Gmail similarities User usage 0.9 0.6 Usage correction, 0.8 0.8 more usage results in a higher score [0,1] 0.6 0.4 0.4 0.2
  • 25. Calculate the predicted value for Gmail Gmail similarities User usage 0.9 0.6 0.8 0.8 0.6 0.4 0.4 0.2 (0.6 * 0.9) + (0.8 * 0.8) + (0.4 * 0.6) = 0.82 0.6 + 0.8 + 0.4 + 0.4
  • 26. Calculate the predicted value for Gmail • User feedback Gmail similarities User usage • Contacts usage 0.9 0.6 • Commercial vs Free 0.8 0.8 0.6 0.4 0.4 0.2 (0.6 * 0.9) + (0.8 * 0.8) + (0.4 * 0.6) = 0.82 0.6 + 0.8 + 0.4 + 0.4
  • 27. Calculate all unknown values and show the Top-N recommendations to each user Software items ? ? ? 1 1 1 1 ?1?? 1 1 1 ?1?1? Users 1 1 ?1111? 1 ?111?11 ?1?1??1
  • 28. Explainability Why did I get this recommendation? • Overlap between the item’s (K) neighbors and your usage
  • 29. User-Based Collaborative Filtering Finding people like you 1 1 0 1 0 1 0 1 1 1 0 1 0 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 Cosine Similarity(Coen, Menno) 0 1 1 1 0 1 1 0 1 0 1 0 0 1
  • 30. Applying inverse user frequency log(n/ni): ni is the number of users that uses item i and n is the total number of users in the database 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0 0.8 0 0 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0.4 0.8 0.4 0 Cosine Similarity(Coen, Menno) 0 0.2 0.6 0.4 0 0.4 0.2 0 0.2 0 0.4 0 0 0.2 The fact that you both use Textmate tells you more than when you both use firefox
  • 31. 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0 0.8 0 0 0.1 0.2 0 0.4 0 0.4 0 0.1 0.2 0.6 0.4 0.8 0.4 0 Cosine Similarity(Coen, Menno) 0 0.2 0.6 0.4 0 0.4 0.2 0 0.2 0 0.4 0 0 0.2
  • 32. User-user correlation matrix 1 0.8 0.6 0.5 0.7 0.2 0.8 1 0.4 0.7 0.5 0.5 0.6 0.4 1 0.4 0.9 0.1 0.5 0.8 0.4 1 0.6 0.4 0.8 0.5 0.9 0.6 1 0.2 0.2 0.5 0.1 0.4 0.2 1
  • 33. Performance measure for success • Cross-validation: Train-Test split (80-20) • Precision and Recall: - precision = size(hit set) / size(total given recs) - recall = size(hit set) / size(test set) • Root mean squared error (RMSE)
  • 34. Implementation • Ruby Enterprise Edition (garbage collection) • MySQL database • Built our own c-libraries • Amazon EC2: - Low cost - Flexibility - Ease of use • Open source
  • 35. Future challenges • What is the best algorithm for Wakoopa? (or you) • Reducing space-time complexity (scalability): - Parallelization (Clojure) - Distributed computing (Hadoop)
  • 36.
  • 37. 1 evening, 3 speakers, 100 developers www.recked.org