Details
Major Description
Something is holding onto KahaDB scheduler log files. We have reports of up to 400GB of scheduler log files. I have tried to isolate the issue and create a minimal example (attached). In the troubleshooting I have done, the scheduler GC process is running, it's just deciding not to GC files that it should be. I have also found behavior inconsistent on the log files it does remove.
The ran the attached example/test on 5.10.0, 5.11.1, 5.12.0 and 5.13.0. The test schedules 20 messages that are large enough to cause 4 log files to be created. It then consumes all 20 messages. When on 5.10.0, it behaves like I would expect, files 1-3 are GC'd and the 4th (the current log file) is left. On all other versions I've tried it always leaves the first 2 files, and sometimes will GC the 3rd.
Below is a snippet from the log of the scheduler process and why it's deciding not to GC these files:
2016-02-01 16:32:33,327 DEBUG [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Job Scheduler Store Checkpoint started. 2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Last update: 4:29447367, full gc candidates set: [1, 2, 3, 4] 2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] gc candidates after reference check: [1, 2, 3] 2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A remove in log 1 has an add still in existance. 2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A remove in log 2 has an add still in existance. 2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A remove in log 3 has an add still in existance. 2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] gc candidates after removals check: [] 2016-02-01 16:32:33,339 DEBUG [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Job Scheduler Store Checkpoint complete. 2016-02-01 16:32:33,727 INFO [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] JobSchedulerStore: test-mq/localhost/scheduler stopped.
This issue was originally reported in the Open Source PuppetDB project, ticket here.