Does a session becomes invalid as soon as the user closes all browser windows ?

Closing or terminating windows will not terminate the session.

A session can be ended either programatically by calling invalidate() method (or) timed out simply. When the user closes all the browser windows the session started by the particular request will not come to an end immediately. The server will wait until the session times out. But one major point to note here is that once when the browser window with which the request was made is closed the same session cannot be continued further with IE browser but possible with FireFox.

You can find more about this

Here

From Spec

In the HTTP protocol, there is no explicit termination signal when a client is no longer active. This means that the only mechanism that can be used to indicate whena client is no longer active is a timeout period.