Implement MonitoredChannel

[jchmura-sc]

Purpose
The purpose of this PR is to add telemetry to our ShmChannel.

We want to know how many SampleMessage's are in each of our channels at any given time, since an empty channel implies a blocking recv, which implies a dataloader bottleneck. Conversely, if our channels stay comfortably above some low watermark then any changes upstream of dataloader.__next__ is unlikely to improve throughput.

PR Changes

1. New class SizedShmChannel(ShmChannel): adds qsize() method to ShmChannel by attaching to the channels memory region and inspecting the C++ struct layout. It calculates queue size as the delta between the circular buffer's write and read sequence numbers.
2. New class MonitoredShmChannel(SizedShmChannel): a subclass that integrates with the GiGL metrics service, pushing the qsize as a gauge on every recv() call.

Usage (colocated)

1. Call gigl.src.common.utils.metrics_service_provider.initialize_metrics()
2. Set GIGL_ENABLE_PERF_MONITORING env var to "True" or "1"

Context (Colocated-Mode)
Each of our N replicas launch N_local training workers. Each training worker initializes one or more data loaders (e.g. DistABLPLoader, DistNeighborLoader). Each dataloader is associated with a ShmChannel. For a given dataloader, multiple sampling workers (e.g. DistNeighborSampler, DistPPRNeighborSampler) push data into the queue.

The channel is roughly used like:

channel = ShmChannel()

# In dataloader process (single consumer)
def __next__(self):
    msg: SampleMessage = channel.recv() # Blocks if empty
    ... # Convert SampleMessage to PyG data

# In each`SamplingRuntime` worker process(multiple producers)
async def _send_adapter(self, ...):
    ... # Sample subgraphs    
    msg: SampleMessage = self._collate_fn(...) # Hydrate features over RPC
    channel.send(msg) # Blocks if channel is at capacity

Challenge
There is no direct way to gauge the queue size since GLT's ShmChannel does not expose the size to python.

Updated Changelog.md? NO

Ready for code review?: YES

[jchmura-sc]

/unit_test

[github-actions]

GiGL Automation

@ 21:34:08UTC : 🔄 C++ Unit Test started.

@ 21:35:52UTC : ✅ Workflow completed successfully.

[github-actions]

GiGL Automation

@ 21:34:10UTC : 🔄 Scala Unit Test started.

@ 21:43:27UTC : ✅ Workflow completed successfully.

[github-actions]

GiGL Automation

@ 21:34:10UTC : 🔄 Python Unit Test started.

[kmontemayor2-sc]

Thanks Jacob!

[mkolodner-sc]

Thanks a lot :)