"He did not speak for so long a time that I began to find the silence awkward"

William James in Principles of Psychology in a chapter on the perception of time writes:

A day full of excitement, with no pause, is said to pass 'ere we know it.' On the contrary, a day full of waiting, of unsatisfied desire for change, will seem a small eternity. Tædium, ennui, Langweile, boredom, are words for which, probably, every language known to man has its equivalent. It comes about whenever, from the relative emptiness of content of a tract of time, we grow attentive to the passage of the time itself. Expecting, and being ready for, a new impression to succeed; when it fails to come, we get an empty time instead of it; and such experiences, ceaselessly renewed, make us most formidably aware of the extent of the mere time itself.[37] [...] The odiousness of the whole experience comes from its insipidity; for stimulation is the indispensable requisite for pleasure in an experience, and the feeling of bare time is the least stimulating experience we can have.[38] The sensation of tædium is a protest, says Volkmann, against the entire present.

The awkwardness that Maugham feels can be construed as a social dimension of this tædium.