@EikeDehling can you enable debug logging for indices.cluster and grep the logs for the output of this line? I want to see what the difference is between the two sources . A gist will be great.

logger.debug("[{}] parsed mapping [{}], and got different sources\noriginal:\n{}\nparsed:\n{}", index, mappingType, mappingSource, mapperService.documentMapper(mappingType).mappingSource());

@bleskes Thanks for the quick reponse!

Gist here: https://gist.github.com/EikeDehling/129aa3f8213ad8552f49

The difference in mapping appears to be in nested elements, apparently they are not ordered alphbetically? The difference in serialisation is under posting.properties.body.fields.text.fielddata , there entries there are ordered differently in the original/parsed version.

This gist is a bit easier to read:

https://gist.github.com/EikeDehling/fc1289cc443b7acdc3f4

The issue is under the key posting.properties.body.fields._text_.fielddata : Ordering is different for the original/parsed mapping.

@EikeDehling thx for that. It's accurate. The problem lies in the way the field data settings are rendered:

https://github.com/elastic/elasticsearch/blob/master/src/main/java/org/elasticsearch/index/mapper/core/AbstractFieldMapper.java#L756

builder.field("fielddata", (Map) fieldDataType.getSettings().getAsMap());

The order of the keys in that map is arbitrary (practically). It may be different between master and nodes causing this endless loop.

To work around this, you can set indices.cluster.send_refresh_mapping to false (requires node restart). This will disable the sending of mapping refresh instructions. You must remember to remove this settings before you upgrade to the next ES version, which will have a fix for this.

@EikeDehling do you run on Java8 by any chance? (wondering to better understand how frequently this can happen)

We're running Java7, 1.7.0_45

Thanks for the tip about settings.

I also found that line of code indeed, i'll try and make a patch/test.

Cool. If you wait an hour or two, I’ll probably make a PR with a fix.

On 01 Apr 2015, at 10:25, EikeDehling notifications@github.com wrote:

We're running Java7, 1.7.0_45

Thanks for the tip about settings.

I also found that line of code indeed, i'll try and make a patch/test.

—
Reply to this email directly or view it on GitHub.

This is my initial patch+unit test, happy to compare to what you're producing. Sorry, i'm not that handy with github/PR's yet.

https://gist.github.com/EikeDehling/2e34a78a54de646b71ca

Any chance there will also be a 1.4 release with a fix?

@bleskes You said that the indices.cluster.send_refresh_mapping requires node restart, what would the effect be if you only have a few (say half of) the nodes which have this setting set to false?

Then the other half might still send refresh mapping to the master. You need it on all data nodes. But you can do a rolling restart, one by one.

On 01 Apr 2015, at 11:24, wkoot notifications@github.com wrote:

@bleskes You said that the indices.cluster.send_refresh_mapping requires node restart, what would the effect be if you only have a few (say half of) the nodes which have this setting set to false?

—
Reply to this email directly or view it on GitHub.

I am trying this fix in our staging environment, i'll let you know if that fixes the issue.

https://github.com/EikeDehling/elasticsearch/commit/cc79d71bbc4d55cb12a50df2acc67ca6ba4ac5dc

@EikeDehling looks good can you make a PR ? see https://www.elastic.co/contributing-to-elasticsearch . Also it would be great if you simplify the test and add random keys (but we can iterate on the PR).

I made a PR, and afterwards signed the contributor license, i hope that's ok.

I randomized the test and simplified a bit, happy to hear suggestions for improvements.