Uploaded image for project: 'Sqoop (Retired)'
  1. Sqoop (Retired)
  2. SQOOP-1341

Sqoop Export Upsert for MySQL lacks batch support

    XMLWordPrintableJSON

Details

    Description

      MySQL export upserts are limited to one row per statement. This wastes bandwidth and makes the exports unusably slow.

      I wrote a patch to support multiple rows per statement at (https://github.com/cloudera/sqoop/pull/22/files). Now the records.per.statement parameter actually works, and I get fast exports (60k rows/sec) with 1000 records per statement, 10 statements per transactions, and 5 mappers.

      Attachments

        1. SQOOP-1341.patch
          2 kB
          Andy Skelton

        Activity

          People

            skeltoac Andy Skelton
            skeltoac Andy Skelton
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified