You are unlikely to get true silence in an audio signal: there will almost always be some low-level noise. You could, though, set a threshold below which you deemed it to be silence.
I believe you should rethink what you are trying to accomplish. Speech becomes quite difficult to understand if you remove all the pauses between words and syllables. Silence is very important for the plosive consonants such as k and p .
