Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5490

KMeans costs can be incorrect if tasks need to be rerun

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 1.3.0
    • None
    • MLlib

    Description

      KMeans uses accumulators to compute the cost of a clustering at each iteration.

      Each time a ShuffleMapTask completes, it increments the accumulators at the driver. If a task runs twice because of failures, the accumulators get incremented twice.

      KMeans uses accumulators in ShuffleMapTasks. This means that a task's cost can end up being double-counted.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sandyr Sandy Ryza
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: