Memory leaks for large objects

Ok, 

I think large objects are leaking memory because the large object functions
in the backend use their own GlobalMemoryContext (called Filesystem), which
(according to a quick grep) is never freed.

Supposing this is true and I ensure that the large object subsystem always
uses the current memory context for it's memory allocations.

What might go wrong? (Or why did the designers decide to use a
GlobalMemoryContext for large objects?).

I simple don't understand why one would create a special memory context
for large objects without some special reason. 
Or should I just try it and see is anything breaks?

Thanks for any comments.
Maurice