Event all the things! #18888

calavera · 2015-12-23T22:06:16Z

This is an attempt to make our events more rich and extensible.

In order to do that, we stop using JSONMessage as a struct to deliver events. That struct is an abomination for all kind of purposes, and the less we use it the better (imo). I created a specific purpose structure that includes new fields:

Type: it's the type of resource the event is triggered for, container, image, volume and network.
Action: it's the name of the event itself, create, destroy, mount, connect and so on.
Actor: it's information about the resource, ID, Name and Attributes. Attributes is an open map to send extra metadata, like the labels for a container event.

IMPORTANT: This change is backwards compatible at the API level, but not at the CLI level.

The new event structure preserves old fields that we've been using until now for container and image events. People consuming the API won't see any difference in those events, but will start to see events without information in those fields, because they come from storage and network operations.

I've included a test to guarantee that we keep that backwards compatible information: https://github.com/docker/docker/compare/master...calavera:event_types?expand=1#diff-1e68c88f5c9fa605c0f84199cd04feb0R38

The CLI output has been completely modified and it's standard for all types of events, the syntax is as follow:

timestamp actor_type action actor_id (<optional>actor_attributes)

The attributes are listed in KEY=VALUE format to be consistent with other places where we show this kind of values, like labels.

Some examples:

Networking and container events

Network events that operate on containers (connect and disconnect) include the container id as part of the attributes:

2015-12-23T21:38:24.705709133Z network create 8b111217944ba0ba844a65b13efcd57dc494932ee2527577758f939315ba2c5b (name=test-event-network-local, type=bridge)
2015-12-23T21:38:25.075955890Z container create b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (name=test-network-container, image=busybox)
2015-12-23T21:38:25.119625123Z network connect 8b111217944ba0ba844a65b13efcd57dc494932ee2527577758f939315ba2c5b (name=test-event-network-local, container=b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e, type=bridge)
2015-12-23T21:38:25.121051956Z container start b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (name=test-network-container, image=busybox)

Storage and container events:

Storage events that operate on containers include the container id as part of the attributes:

2015-12-23T21:05:28.136212689Z volume create test-event-volume-local (driver=local)
2015-12-23T21:05:28.383462717Z volume mount test-event-volume-local (read/write=true, container=562fe10671e9273da25eed36cdce26159085ac7ee6707105fd534866340a5025, destination=/foo, driver=local, propagation=rprivate)
2015-12-23T21:05:28.650314265Z volume unmount test-event-volume-local (container=562fe10671e9273da25eed36cdce26159085ac7ee6707105fd534866340a5025, driver=local)
2015-12-23T21:05:28.716218405Z volume destroy test-event-volume-local (driver=local)

Future improvements out of the scope here

Add a format flag with a go template.

TODO

Modify the events documentation.

/cc @coolljt0725, @dnephin

thaJeztah · 2015-12-23T22:18:09Z

Nice! Looking at the command line output, should we put the type of event earlier on? the type of event is kinda hurried now. E.g.

2015-12-23T21:38:24.705709133Z network create 8b111217944ba0ba844a65b13efcd57dc494932ee2527577758f939315ba2c5b (test-event-network-local) (type: bridge)
2015-12-23T21:38:25.075955890Z container create b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (test-network-container) (image: busybox)
2015-12-23T21:38:25.119625123Z network connect 8b111217944ba0ba844a65b13efcd57dc494932ee2527577758f939315ba2c5b (test-event-network-local) (container: b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e, type: bridge)
2015-12-23T21:38:25.121051956Z container start b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (test-network-container)  (image: busybox)

Or

2015-12-23T21:38:24.705709133Z create network 8b111217944ba0ba844a65b13efcd57dc494932ee2527577758f939315ba2c5b (test-event-network-local) (type: bridge)
2015-12-23T21:38:25.075955890Z create container  b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (test-network-container) (image: busybox)
2015-12-23T21:38:25.119625123Z connect network 8b111217944ba0ba844a65b13efcd57dc494932ee2527577758f939315ba2c5b (test-event-network-local) (container: b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e, type: bridge)
2015-12-23T21:38:25.121051956Z start container b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (test-network-container)  (image: busybox)

We could even consider a --format flag in the future

calavera · 2015-12-23T22:21:04Z

Obligatory regular expression meme (do not look at events_utils.go):

@thaJeztah I'm okay with moving the action at the beginning, let's wait to hear from other people to see if we agree in a format and I only need to change it once 😸

thaJeztah · 2015-12-23T22:23:10Z

@calavera haha, absolutely, just my first impression looking at it. great work otherwise, I like it!

thaJeztah · 2015-12-23T22:23:34Z

Oh, and
/cc @TomasTomecek @samsabed

dnephin · 2015-12-23T22:43:15Z

daemon/daemon.go

+		ID:   imageID.String(),
+		Name: newTag.String(),
+	}
+	daemon.EventsService.Log("tag", eventtypes.ImageEventType, actor)


I believe all the eventtypes.ImageEventType events should include the image labels in Actor.Attributes. Previously this was being injected by daemon.GetLabels()

fixed, thanks!

I think they're all done except for this one place?

😔 fixing

actually there are a couple more places in pull/push

dnephin · 2015-12-23T22:45:09Z

Nice! I think this is awesome. We'll be able to make use of the new event data in the soon-to-be-released docker-compose events command.

calavera · 2015-12-23T23:58:12Z

@thaJeztah one thing that I don't like about your suggested syntax is that it becomes harder to parse, because both name and attributes are optional at the end and are between parenthesis. But I'm open to suggestions.

calavera · 2015-12-24T00:12:00Z

But I'm open to suggestions.

maybe we should add the name to the attributes. That would leave us only with one optional field, which I think it's better:

timestamp type action id (<optional>attributes)

Example:

2015-12-23T21:38:25.075955890Z container create b4be644031a3d90b400f88ab3d4bdf4dc23adb250e696b6328b85441abe2c54e (name: test-network-container, image: busybox)

TomasTomecek · 2015-12-27T17:27:17Z

This is really impressive. Very nice work! A couple questions.

Will there be a precise documentation of what are the possible values (states) to expect? Regarding the new changes to layering, image IDs 'n stuff, would it make sense to also include digest? How about untagging an image, will I get the tag name? And last thing: would be super cool to unite events status and inspect status (inspect status contains text starting with capital letter and human-readable time which is a nightmare to parse).

thaJeztah · 2015-12-27T20:40:32Z

@calavera oh, I like the suggestion of adding name to attributes. In addition, we can still add --format in a future PR. My primary concern was that the "type" of event got a bit lost in the other information, putting it more to the start of the line makes that easier to read (visually).

calavera · 2015-12-28T16:02:28Z

Regarding the new changes to layering, image IDs 'n stuff, would it make sense to also include digest? How about untagging an image, will I get the tag name?

@TomasTomecek yes, with the new attributes, we can include more relevant information about the events, like that one. This is not a one time only change, we can improve the information we add to each event in the future.

And last thing: would be super cool to unite events status and inspect status (inspect status contains text starting with capital letter and human-readable time which is a nightmare to parse).

I'm not completely sure that's related to this change. I think we should discuss it further, but I understand your issues.

calavera · 2015-12-28T20:20:39Z

@thaJeztah I've changed the log format to put the name in the attributes.

I've also added a new feature to be able to filter by event type with --filter type=eventType, like --filter type=container or --filter type=network.

Can I have some 👍 / 👎 to all this before moving it to code review?

@TomasTomecek documentation is coming.

Signed-off-by: David Calavera <david.calavera@gmail.com>

calavera · 2016-01-04T19:59:02Z

good point @moxiegirl. The API still works with old fields, but I agree that's worth noticing in the deprecation document. I added it already.

thaJeztah · 2016-01-04T20:02:59Z

docs LGTM

@moxiegirl the API is backwards compatible, but CLI output changed, I think we should mention that in the release notes

moxiegirl · 2016-01-04T20:41:33Z

@thaJeztah If by release notes you mean the changelog. Then we are good. :-)

thaJeztah · 2016-01-04T21:03:07Z

potatoes potatos, yup that's what I meant, haha

calavera · 2016-01-04T21:06:46Z

Event all the things!

TomasTomecek · 2016-01-05T12:42:59Z

@calavera very nice! can't wait to play with this

thaJeztah · 2016-01-05T12:46:30Z

@TomasTomecek for playing; https://master.dockerproject.org is your friend :-) And please do! We can never have enough testers :)

GordonTheTurtle added the status/0-triage label Dec 23, 2015

calavera added status/1-design-review and removed status/0-triage labels Dec 23, 2015

calavera force-pushed the event_types branch 2 times, most recently from f3061c0 to f4b543b Compare December 23, 2015 22:12

calavera force-pushed the event_types branch 2 times, most recently from 3cef129 to 411e141 Compare December 23, 2015 22:30

dnephin reviewed Dec 23, 2015
View reviewed changes

calavera force-pushed the event_types branch 4 times, most recently from f063836 to ce954d8 Compare December 23, 2015 23:54

calavera force-pushed the event_types branch from ce954d8 to e9054ee Compare December 24, 2015 00:49

calavera force-pushed the event_types branch 3 times, most recently from 9eb3945 to 93be08b Compare December 28, 2015 20:18

calavera force-pushed the event_types branch 2 times, most recently from 7027b1b to 5afee52 Compare December 28, 2015 20:42

Add filter by event type and documentation.

851fe00

Signed-off-by: David Calavera <david.calavera@gmail.com>

calavera force-pushed the event_types branch from 9cc7046 to 851fe00 Compare January 4, 2016 19:58

thaJeztah added status/4-merge and removed status/3-docs-review labels Jan 4, 2016

calavera added a commit that referenced this pull request Jan 4, 2016

Merge pull request #18888 from calavera/event_types

723be0a

Event all the things!

calavera merged commit 723be0a into moby:master Jan 4, 2016

calavera deleted the event_types branch January 4, 2016 21:07

thaJeztah added this to the 1.10 milestone Jan 4, 2016

bjaglin mentioned this pull request Jan 4, 2016

Post-start hooks not always fired michaelsauter/crane#238

Closed

TomasTomecek mentioned this pull request Jan 5, 2016

use updated events API TomasTomecek/sen#58

Closed

vdemeester mentioned this pull request Jan 5, 2016

Proposal: Add label support to update command #18958

Closed

4 tasks

thaJeztah mentioned this pull request Jan 7, 2016

Generate hosts file with all IPs and hostnames for all containers nginx-proxy/docker-gen#149

Open

This was referenced Jan 7, 2016

Upgrade events to use the new API fields docker/compose#2618

Closed

docker-compose events docker/compose#2392

Merged

thaJeztah mentioned this pull request Jan 16, 2016

Remove nested BTRFS subvolumes on volume removal automatically #18624

Closed

bboreham mentioned this pull request Jan 20, 2016

add/remove DNS entries on network connect/disconnect weaveworks/weave#1914

Open

albers mentioned this pull request Jan 21, 2016

Support new events in bash completion #19541

Merged

thaJeztah mentioned this pull request Jan 31, 2016

network events #19862

Closed

thaJeztah mentioned this pull request Mar 4, 2016

Add LastUsedAsContainerTime access time attribute for Images #20925

Closed

thaJeztah mentioned this pull request Apr 26, 2016

Unrecognized input header: 99 #21902

Closed

LK4D4 mentioned this pull request Sep 16, 2016

OOM event of exec is the same as OOM event of container main process #18433

Closed

yongtang mentioned this pull request Mar 2, 2018

Migrate events tests to api tests #36444

Merged

thaJeztah mentioned this pull request Aug 26, 2023

Remove uses of deprecated "Status", "ID", and "From" fields for events #46336

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Event all the things! #18888

Event all the things! #18888

calavera commented Dec 23, 2015 •

edited by thaJeztah

thaJeztah commented Dec 23, 2015

calavera commented Dec 23, 2015

thaJeztah commented Dec 23, 2015

thaJeztah commented Dec 23, 2015

dnephin Dec 23, 2015

calavera Dec 23, 2015

dnephin Dec 23, 2015

calavera Dec 23, 2015

calavera Dec 23, 2015

dnephin commented Dec 23, 2015

calavera commented Dec 23, 2015

calavera commented Dec 24, 2015

TomasTomecek commented Dec 27, 2015

thaJeztah commented Dec 27, 2015

calavera commented Dec 28, 2015

calavera commented Dec 28, 2015

calavera commented Jan 4, 2016

thaJeztah commented Jan 4, 2016

moxiegirl commented Jan 4, 2016

thaJeztah commented Jan 4, 2016

calavera commented Jan 4, 2016

TomasTomecek commented Jan 5, 2016

thaJeztah commented Jan 5, 2016

Event all the things! #18888

Event all the things! #18888

Conversation

calavera commented Dec 23, 2015 • edited by thaJeztah

Some examples:

Networking and container events

Storage and container events:

Future improvements out of the scope here

TODO

thaJeztah commented Dec 23, 2015

calavera commented Dec 23, 2015

thaJeztah commented Dec 23, 2015

thaJeztah commented Dec 23, 2015

dnephin Dec 23, 2015

Choose a reason for hiding this comment

calavera Dec 23, 2015

Choose a reason for hiding this comment

dnephin Dec 23, 2015

Choose a reason for hiding this comment

calavera Dec 23, 2015

Choose a reason for hiding this comment

calavera Dec 23, 2015

Choose a reason for hiding this comment

dnephin commented Dec 23, 2015

calavera commented Dec 23, 2015

calavera commented Dec 24, 2015

TomasTomecek commented Dec 27, 2015

thaJeztah commented Dec 27, 2015

calavera commented Dec 28, 2015

calavera commented Dec 28, 2015

calavera commented Jan 4, 2016

thaJeztah commented Jan 4, 2016

moxiegirl commented Jan 4, 2016

thaJeztah commented Jan 4, 2016

calavera commented Jan 4, 2016

TomasTomecek commented Jan 5, 2016

thaJeztah commented Jan 5, 2016

calavera commented Dec 23, 2015 •

edited by thaJeztah