Skip to content
This repository has been archived by the owner on Feb 24, 2020. It is now read-only.

rkt: implement initial image gc for the tree store. #1487

Merged
merged 1 commit into from Sep 28, 2015

Conversation

sgotti
Copy link
Contributor

@sgotti sgotti commented Sep 27, 2015

This is on top of #1240. The last patch is the interesting one.

rkt: implement initial image gc for the tree store.

This patch adds an initial image gc command.

Image gc can be thought in multiple ways:

  • Remove treeStores not referenced by any non GCed pod
  • Remove images not used for some time (rkt: implement garbage collection for the store #1205)
  • Cleanup the store from other garbage:
    • Entries removed from the aciinfo tables but failed to remove for any reason from the diskv stores during rkt image rm
    • Cleanup store's treestorelocks directory
    • Cleanup store's tmpdir directory

This patch implements the removal of treeStores not referenced by any non GCed pod.
A "preparelock" is added to avoid races between getting the list of referenced treeStoreIDs and new pods/treeStores being created/referenced in the meantime.

@sgotti
Copy link
Contributor Author

sgotti commented Sep 27, 2015

I'm not sure if all the above needed different kind of store gc (or some of these can be thought as an fsck?) should belong to the same rkt image gc or different commands/subcommands will be better. Thoughts?

Edit: For this reason I waited to add a systemd service or change the current one to also call rkt image gc.

@sgotti sgotti changed the title Rkt image gc treestore rkt: implement initial image gc for the tree store. Sep 27, 2015
@sgotti sgotti force-pushed the rkt_image_gc_treestore branch 2 times, most recently from 763f1d7 to f9a6ca7 Compare September 27, 2015 13:55
@iaguis
Copy link
Member

iaguis commented Sep 28, 2015

Image gc can be thought in multiple ways:

  • Remove treeStores not referenced by any non GCed pod

Addressed by this PR.

This can be done in a follow-up PR.

  • Cleanup the store from other garbage:
    • Entries removed from the aciinfo tables but failed to remove for any reason from the diskv stores during rkt image rm
    • Cleanup store's treestorelocks directory
    • Cleanup store's tmpdir directory

This seems like some sort of fsck or cleanup task, different from gc. I'm not too worried about it since the amount of space consumed by this data should not be high.

Edit: For this reason I waited to add a systemd service or change the current one to also call rkt image gc.

I'm hesitant whether to run rkt image gc in rkt-gc.service. If we do that, any subsequent start of an image after a gc will take longer because of the rendering.

I think rkt-gc.service should just do pod GC and there should be a rkt-image-gc.service that deals with old images and also removes their treeStores.

return fmt.Errorf("error removing the tree store: %v", err)
}
return nil
}

func (ds Store) GetTreeStoreIDs() ([]string, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs?

@sgotti
Copy link
Contributor Author

sgotti commented Sep 28, 2015

This seems like some sort of fsck or cleanup task, different from gc. I'm not too worried about it since the amount of space consumed by this data should not be high.

Agree. Removed them from the commit message.

I think rkt-gc.service should just do pod GC and there should be a rkt-image-gc.service that deals with old images and also removes their treeStores.

Agree

@iaguis
Copy link
Member

iaguis commented Sep 28, 2015

Please rebase

}

if err := gcTreeStore(s); err != nil {
stderr("rkt: failed to remove unreferenced treeStores: %v", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes we print treeStore(s) and sometimes treestore(s). I don't have a preference but we should be consistent :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made all the printed one lowercase (except ID).

My bad but I'm also not sure if it's better to say (given that a treestore is the output of a rendered image and every treestore has an ID) treestoreIDs / treestoresIDs/ treestore IDs / treestores IDs / treestore's IDs / treeStores' IDs or others... Any suggestion?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the way it is now is OK: treestoreID(s) when functions for getting the IDs fail and treestore(s) when we refer to the directory (e.g. removing a treestore).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iaguis ok! Thanks!

@iaguis
Copy link
Member

iaguis commented Sep 28, 2015

A nit but LGTM

This patch adds an initial image gc command.
Image gc can be thought in multiple ways:

* Remove treeStores not referenced by any non GCed pod
* Remove images not used for some time

This patchs implements the removal of treeStores not referenced by any non GCed pod.

A "preparelock" is added to avoid races between getting the list of referenced
treeStoreIDs and new pods/treeStores being created/referenced in the meantime.
@iaguis
Copy link
Member

iaguis commented Sep 28, 2015

Thanks @sgotti!

iaguis added a commit that referenced this pull request Sep 28, 2015
rkt: implement initial image gc for the tree store.
@iaguis iaguis merged commit 68710d6 into rkt:master Sep 28, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants