Stats Collection and Visualization

[dthain]

Make several improvements in stats collection in the kernel:

• Add a system call to query the current buffer cache stats. (They are already collected)
• Add a similar stats collection to the kobject layer, so we can observe objects at a finer granularity.
• Move the stats collection that is currently in ata up into the struct device_driver object instead, so that the device module automatically counts ops on each device driver in summary.

With that, the user should be able to do this:

syscall_device_stats(name,&dstats)
syscall_object_stats(fd,&ostats)
syscall_process_stats(pid,&pstats)
syscall_bcache_stats(&bstats)

Now, here is the fun part: build a little graphics visualization of one of these, say the buffer cache stats. Query the stats once per second and keep all the values going back, say, one minute. Plot each one as a single vertical line and move everything over one per second.

Then, start up the window manage with something interesting in the top window, and the visulization in the bottom window, and watch the I/O behavior over time.

[jmazanec15]

For getting stats for particular objects by file descriptor, because of #248 implementing shallow copying of kobjects on inheritance, would we expect that the object stats would only change within a single process?

[jmazanec15]

For the interface for object_get_stats, per our discussion Wednesday, would it look like this:

syscall_object_stats(int fd, struct object * stats, int level)

Level would specify how deep into the abstraction one would want to go. For instance, 1 may represent the kobject stats, 2 could represent the consoles stats, etc. Then, we could set up some flags users could use to indicate the level. For example, CONSOLE_STATS_LEVEL.

One question I have is should struct object * stats actually be a pointer to any stats object so that we could have more fine grained control over what stats a user gets. So, the interface would look like:

syscall_object_stats(int fd, void * stats, int level)

[dthain]

This is a tricky question of API design, and I see a couple of ways forward:

1 - Common Structure. If the various kinds of stats needed are very close to each other, then it would be easy to design a single structure the describes all the necessary elements (r/w ops, r/w bytes, etc) and have every layer use the same structure. Perhaps some fields only get used by certain layers and are left zero otherwise. Then, as you point out, the kernel interface can just be used to select the proper depth.

2 - Opaque Pointer. On the other hand, if the various stats structures are really quite different, then you could accommodate this in a single system call by having a single opaque pointer (void *), have the caller select the depth, and then be responsible for casting it to the appropriate structure type. In one way this is nice, because you end up having a single stats system call, but then it becomes harder to have type safety. Either the user or the kernel could make a mistake in casting.

3 - Multiple Stats Calls. A third way would be to accommodate very different structure types by having multiple system calls, each one requesting a specific structure type. That way, it's clear as day what is being asked for, and a minimum of casting types around. Of course, to add a new stats type requires adding a new stats call...

Does that help?
What do you think?

[jmazanec15]

I think I am leaning towards the opaque approach for a couple reasons. One thing I like about it is that it represents how objects are accessed through abstractions. Regardless of what type of object we are querying, we still need to pass an object descriptor to this system call and then that kobject will be grabbed in the ktable. I think having multiple stats calls might not represent the object abstraction as clearly.

I like having different stats structures because I think it gives us more freedom to choose which stats we are interested in as opposed to having more of a rigid structure, or unused fields. For instance, in graphics it might be interesting to see how many pixels have been plotted and that would not go with any other objects.

The problem with the Opaque pointer approach is, like you said, type safety. I think the kernel could use the is_valid_pointer to confirm that the user passed in a struct of the correct size of the object they are querying. And then, the rest will require the user to know what type of object they are passing in, which with the system call syscall_object_type(int fd), I think would be reasonable.