综合练习：词频统计(代码片段)

综合练习

词频统计预处理

下载一首英文的歌词或文章

将所有,.？！’:等分隔符全部替换为空格

str = ‘‘‘Passion is sweet
Love makes weak
You said you cherised freedom so
You refused to let it go
Follow your faith 
Love and hate
never failed to seize the day
Don‘t give yourself away
Oh when the night falls
And your all alone
In your deepest sleep 
What are you dreeeming of
My skin‘s still burning from your touch
Oh I just can‘t get enough 
I said I wouldn‘t ask for much
But your eyes are dangerous
So the tought keeps spinning in my head
Can we drop this masquerade
I can‘t predict where it ends
If you‘re the rock I‘ll crush against
Trapped in a crowd
Music‘s loud
I said I loved my freedom too
Now im not so sure i do
All eyes on you
Wings so true
Better quit while your ahead
Now im not so sure i am
Oh when the night falls
And your all alone
In your deepest sleep
What are you dreaming of
My skin‘s still burning from your touch
Oh I just can‘t get enough
I said I wouldn‘t ask for much
But your eyes are dangerous
So the thought keeps spinning in my head
Can we drop this masquerade 
I can‘t predict where it ends
If you‘re the rock I‘ll crush against
My soul, my heart
If your near or if your far
My life, my love
You can have it all
Oh when the night falls
And your all alone
In your deepest sleep
What are you dreaming of
My skin‘s still burning from your touch
Oh I just can‘t get enough
I said I wouldn‘t ask for much
But your eyes are dangerous 
So the thought keeps spinning in my head
Can we drop this masquerade
I can‘t predict where it ends
If you‘re the rock I‘ll crush against
If you‘re the rock i‘ll crush against‘‘‘
sep = ‘‘‘.?,‘"!‘‘‘
for i in sep:
    str = str.replace(i, ‘ ‘)
print(str)

结果如下：

Love and hate
never failed to seize the day
Don t give yourself away
Oh when the night falls
And your all alone
In your deepest sleep
What are you dreeeming of
My skin s still burning from your touch
Oh I just can t get enough
I said I wouldn t ask for much
But your eyes are dangerous
So the tought keeps spinning in my head
Can we drop this masquerade
I can t predict where it ends
If you re the rock I ll crush against
Trapped in a crowd
Music s loud
I said I loved my freedom too
Now im not so sure i do
All eyes on you
Wings so true
Better quit while your ahead
Now im not so sure i am
Oh when the night falls
And your all alone
In your deepest sleep
What are you dreaming of
My skin s still burning from your touch
Oh I just can t get enough
I said I wouldn t ask for much
But your eyes are dangerous
So the thought keeps spinning in my head
Can we drop this masquerade
I can t predict where it ends
If you re the rock I ll crush against
My soul my heart
If your near or if your far
My life my love
You can have it all
Oh when the night falls
And your all alone
In your deepest sleep
What are you dreaming of
My skin s still burning from your touch
Oh I just can t get enough
I said I wouldn t ask for much
But your eyes are dangerous
So the thought keeps spinning in my head
Can we drop this masquerade
I can t predict where it ends
If you re the rock I ll crush against
If you re the rock i ll crush against

将所有大写转换为小写

print(str.lower())

结果如下：

passion is sweet
love makes weak
you said you cherised freedom so
you refused to let it go
follow your faith
love and hate
never failed to seize the day
don t give yourself away
oh when the night falls
and your all alone
in your deepest sleep
what are you dreeeming of
my skin s still burning from your touch
oh i just can t get enough
i said i wouldn t ask for much
but your eyes are dangerous
so the tought keeps spinning in my head
can we drop this masquerade
i can t predict where it ends
if you re the rock i ll crush against
trapped in a crowd
music s loud
i said i loved my freedom too
now im not so sure i do
all eyes on you
wings so true
better quit while your ahead
now im not so sure i am
oh when the night falls
and your all alone
in your deepest sleep
what are you dreaming of
my skin s still burning from your touch
oh i just can t get enough
i said i wouldn t ask for much
but your eyes are dangerous
so the thought keeps spinning in my head
can we drop this masquerade
i can t predict where it ends
if you re the rock i ll crush against
my soul my heart
if your near or if your far
my life my love
you can have it all
oh when the night falls
and your all alone
in your deepest sleep
what are you dreaming of
my skin s still burning from your touch
oh i just can t get enough
i said i wouldn t ask for much
but your eyes are dangerous
so the thought keeps spinning in my head
can we drop this masquerade
i can t predict where it ends
if you re the rock i ll crush against
if you re the rock i ll crush against

生成单词列表

wordList = str.lower().split()
print(wordList)

结果如下：

[‘passion‘, ‘is‘, ‘sweet‘, ‘love‘, ‘makes‘, ‘weak‘, ‘you‘, ‘said‘, ‘you‘, ‘cherised‘, ‘freedom‘, ‘so‘, ‘you‘, ‘refused‘, ‘to‘, ‘let‘, ‘it‘, ‘go‘, ‘follow‘, ‘your‘, ‘faith‘, ‘love‘, ‘and‘, ‘hate‘, ‘never‘, ‘failed‘, ‘to‘, ‘seize‘, ‘the‘, ‘day‘, ‘don‘, ‘t‘, ‘give‘, ‘yourself‘, ‘away‘, ‘oh‘, ‘when‘, ‘the‘, ‘night‘, ‘falls‘, ‘and‘, ‘your‘, ‘all‘, ‘alone‘, ‘in‘, ‘your‘, ‘deepest‘, ‘sleep‘, ‘what‘, ‘are‘, ‘you‘, ‘dreeeming‘, ‘of‘, ‘my‘, ‘skin‘, ‘s‘, ‘still‘, ‘burning‘, ‘from‘, ‘your‘, ‘touch‘, ‘oh‘, ‘i‘, ‘just‘, ‘can‘, ‘t‘, ‘get‘, ‘enough‘, ‘i‘, ‘said‘, ‘i‘, ‘wouldn‘, ‘t‘, ‘ask‘, ‘for‘, ‘much‘, ‘but‘, ‘your‘, ‘eyes‘, ‘are‘, ‘dangerous‘, ‘so‘, ‘the‘, ‘tought‘, ‘keeps‘, ‘spinning‘, ‘in‘, ‘my‘, ‘head‘, ‘can‘, ‘we‘, ‘drop‘, ‘this‘, ‘masquerade‘, ‘i‘, ‘can‘, ‘t‘, ‘predict‘, ‘where‘, ‘it‘, ‘ends‘, ‘if‘, ‘you‘, ‘re‘, ‘the‘, ‘rock‘, ‘i‘, ‘ll‘, ‘crush‘, ‘against‘, ‘trapped‘, ‘in‘, ‘a‘, ‘crowd‘, ‘music‘, ‘s‘, ‘loud‘, ‘i‘, ‘said‘, ‘i‘, ‘loved‘, ‘my‘, ‘freedom‘, ‘too‘, ‘now‘, ‘im‘, ‘not‘, ‘so‘, ‘sure‘, ‘i‘, ‘do‘, ‘all‘, ‘eyes‘, ‘on‘, ‘you‘, ‘wings‘, ‘so‘, ‘true‘, ‘better‘, ‘quit‘, ‘while‘, ‘your‘, ‘ahead‘, ‘now‘, ‘im‘, ‘not‘, ‘so‘, ‘sure‘, ‘i‘, ‘am‘, ‘oh‘, ‘when‘, ‘the‘, ‘night‘, ‘falls‘, ‘and‘, ‘your‘, ‘all‘, ‘alone‘, ‘in‘, ‘your‘, ‘deepest‘, ‘sleep‘, ‘what‘, ‘are‘, ‘you‘, ‘dreaming‘, ‘of‘, ‘my‘, ‘skin‘, ‘s‘, ‘still‘, ‘burning‘, ‘from‘, ‘your‘, ‘touch‘, ‘oh‘, ‘i‘, ‘just‘, ‘can‘, ‘t‘, ‘get‘, ‘enough‘, ‘i‘, ‘said‘, ‘i‘, ‘wouldn‘, ‘t‘, ‘ask‘, ‘for‘, ‘much‘, ‘but‘, ‘your‘, ‘eyes‘, ‘are‘, ‘dangerous‘, ‘so‘, ‘the‘, ‘thought‘, ‘keeps‘, ‘spinning‘, ‘in‘, ‘my‘, ‘head‘, ‘can‘, ‘we‘, ‘drop‘, ‘this‘, ‘masquerade‘, ‘i‘, ‘can‘, ‘t‘, ‘predict‘, ‘where‘, ‘it‘, ‘ends‘, ‘if‘, ‘you‘, ‘re‘, ‘the‘, ‘rock‘, ‘i‘, ‘ll‘, ‘crush‘, ‘against‘, ‘my‘, ‘soul‘, ‘my‘, ‘heart‘, ‘if‘, ‘your‘, ‘near‘, ‘or‘, ‘if‘, ‘your‘, ‘far‘, ‘my‘, ‘life‘, ‘my‘, ‘love‘, ‘you‘, ‘can‘, ‘have‘, ‘it‘, ‘all‘, ‘oh‘, ‘when‘, ‘the‘, ‘night‘, ‘falls‘, ‘and‘, ‘your‘, ‘all‘, ‘alone‘, ‘in‘, ‘your‘, ‘deepest‘, ‘sleep‘, ‘what‘, ‘are‘, ‘you‘, ‘dreaming‘, ‘of‘, ‘my‘, ‘skin‘, ‘s‘, ‘still‘, ‘burning‘, ‘from‘, ‘your‘, ‘touch‘, ‘oh‘, ‘i‘, ‘just‘, ‘can‘, ‘t‘, ‘get‘, ‘enough‘, ‘i‘, ‘said‘, ‘i‘, ‘wouldn‘, ‘t‘, ‘ask‘, ‘for‘, ‘much‘, ‘but‘, ‘your‘, ‘eyes‘, ‘are‘, ‘dangerous‘, ‘so‘, ‘the‘, ‘thought‘, ‘keeps‘, ‘spinning‘, ‘in‘, ‘my‘, ‘head‘, ‘can‘, ‘we‘, ‘drop‘, ‘this‘, ‘masquerade‘, ‘i‘, ‘can‘, ‘t‘, ‘predict‘, ‘where‘, ‘it‘, ‘ends‘, ‘if‘, ‘you‘, ‘re‘, ‘the‘, ‘rock‘, ‘i‘, ‘ll‘, ‘crush‘, ‘against‘, ‘if‘, ‘you‘, ‘re‘, ‘the‘, ‘rock‘, ‘i‘, ‘ll‘, ‘crush‘, ‘against‘]

生成词频统计

wordDict = 
wordSet = set(wordList)
for i in wordSet:
    wordDict[i] = wordList.count(i);
print(wordDict)

结果如下：

‘thought‘: 2, ‘loved‘: 1, ‘wings‘: 1, ‘or‘: 1, ‘said‘: 5, ‘if‘: 6, ‘quit‘: 1, ‘spinning‘: 3, ‘night‘: 3, ‘ll‘: 4, ‘refused‘: 1, ‘get‘: 3, ‘am‘: 1, ‘cherised‘: 1, ‘your‘: 16, ‘oh‘: 6, ‘we‘: 3, ‘let‘: 1, ‘are‘: 6, ‘give‘: 1, ‘ahead‘: 1, ‘falls‘: 3, ‘when‘: 3, ‘burning‘: 3, ‘but‘: 3, ‘trapped‘: 1, ‘while‘: 1, ‘ask‘: 3, ‘alone‘: 3, ‘and‘: 4, ‘seize‘: 1, ‘is‘: 1, ‘against‘: 4, ‘keeps‘: 3, ‘makes‘: 1, ‘loud‘: 1, ‘t‘: 10, ‘of‘: 3, ‘head‘: 3, ‘dreaming‘: 2, ‘dangerous‘: 3, ‘enough‘: 3, ‘on‘: 1, ‘for‘: 3, ‘a‘: 1, ‘so‘: 7, ‘heart‘: 1, ‘much‘: 3, ‘ends‘: 3, ‘where‘: 3, ‘now‘: 2, ‘weak‘: 1, ‘rock‘: 4, ‘life‘: 1, ‘just‘: 3, ‘s‘: 4, ‘crowd‘: 1, ‘music‘: 1, ‘true‘: 1, ‘far‘: 1, ‘in‘: 7, ‘you‘: 12, ‘away‘: 1, ‘do‘: 1, ‘to‘: 2, ‘failed‘: 1, ‘this‘: 3, ‘better‘: 1, ‘it‘: 5, ‘sweet‘: 1, ‘im‘: 2, ‘from‘: 3, ‘all‘: 5, ‘eyes‘: 4, ‘can‘: 10, ‘dreeeming‘: 1, ‘the‘: 11, ‘sleep‘: 3, ‘go‘: 1, ‘faith‘: 1, ‘touch‘: 3, ‘hate‘: 1, ‘predict‘: 3, ‘i‘: 20, ‘day‘: 1, ‘tought‘: 1, ‘re‘: 4, ‘still‘: 3, ‘what‘: 3, ‘masquerade‘: 3, ‘drop‘: 3, ‘deepest‘: 3, ‘freedom‘: 2, ‘passion‘: 1, ‘too‘: 1, ‘don‘: 1, ‘yourself‘: 1, ‘not‘: 2, ‘have‘: 1, ‘never‘: 1, ‘crush‘: 4, ‘near‘: 1, ‘love‘: 3, ‘wouldn‘: 3, ‘sure‘: 2, ‘my‘: 11, ‘follow‘: 1, ‘skin‘: 3, ‘soul‘: 1

排序

dictList = list(wordDict.items())
dictList.sort(key=lambda x: x[1], reverse=True)
for i in dictList:
    print(i)

结果如下:

(‘i‘, 20)
(‘your‘, 16)
(‘you‘, 12)
(‘my‘, 11)
(‘the‘, 11)
(‘t‘, 10)
(‘can‘, 10)
(‘so‘, 7)
(‘in‘, 7)
(‘if‘, 6)
(‘oh‘, 6)
(‘are‘, 6)
(‘all‘, 5)
(‘said‘, 5)
(‘it‘, 5)
(‘crush‘, 4)
(‘against‘, 4)
(‘s‘, 4)
(‘re‘, 4)
(‘and‘, 4)
(‘ll‘, 4)
(‘eyes‘, 4)
(‘rock‘, 4)
(‘for‘, 3)
(‘alone‘, 3)
(‘ask‘, 3)
(‘night‘, 3)
(‘but‘, 3)
(‘spinning‘, 3)
(‘this‘, 3)
(‘predict‘, 3)
(‘from‘, 3)
(‘wouldn‘, 3)
(‘we‘, 3)
(‘touch‘, 3)
(‘when‘, 3)
(‘enough‘, 3)
(‘skin‘, 3)
(‘falls‘, 3)
(‘deepest‘, 3)
(‘what‘, 3)
(‘much‘, 3)
(‘sleep‘, 3)
(‘masquerade‘, 3)
(‘head‘, 3)
(‘just‘, 3)
(‘ends‘, 3)
(‘still‘, 3)
(‘where‘, 3)
(‘of‘, 3)
(‘drop‘, 3)
(‘get‘, 3)
(‘love‘, 3)
(‘keeps‘, 3)
(‘burning‘, 3)
(‘dangerous‘, 3)
(‘to‘, 2)
(‘dreaming‘, 2)
(‘not‘, 2)
(‘sure‘, 2)
(‘thought‘, 2)
(‘im‘, 2)
(‘now‘, 2)
(‘freedom‘, 2)
(‘let‘, 1)
(‘cherised‘, 1)
(‘have‘, 1)
(‘dreeeming‘, 1)
(‘give‘, 1)
(‘trapped‘, 1)
(‘music‘, 1)
(‘far‘, 1)
(‘follow‘, 1)
(‘day‘, 1)
(‘is‘, 1)
(‘crowd‘, 1)
(‘loud‘, 1)
(‘failed‘, 1)
(‘better‘, 1)
(‘passion‘, 1)
(‘sweet‘, 1)
(‘soul‘, 1)
(‘or‘, 1)
(‘never‘, 1)
(‘seize‘, 1)
(‘near‘, 1)
(‘hate‘, 1)
(‘a‘, 1)
(‘heart‘, 1)
(‘do‘, 1)
(‘yourself‘, 1)
(‘ahead‘, 1)
(‘am‘, 1)
(‘loved‘, 1)
(‘tought‘, 1)
(‘weak‘, 1)
(‘on‘, 1)
(‘quit‘, 1)
(‘while‘, 1)
(‘wings‘, 1)
(‘away‘, 1)
(‘go‘, 1)
(‘life‘, 1)
(‘too‘, 1)
(‘faith‘, 1)
(‘makes‘, 1)
(‘refused‘, 1)
(‘don‘, 1)
(‘true‘, 1)

排除语法型词汇，代词、冠词、连词

exculde = ‘the‘, ‘i‘, ‘you‘, ‘is‘, ‘and‘, ‘my‘, ‘or‘
for i in exculde:
    wordDict.pop(i)
print(wordDict)

结果如下：

‘s‘: 4, ‘on‘: 1, ‘don‘: 1, ‘too‘: 1, ‘better‘: 1, ‘day‘: 1, ‘wouldn‘: 3, ‘deepest‘: 3, ‘refused‘: 1, ‘t‘: 10, ‘away‘: 1, ‘spinning‘: 3, ‘ends‘: 3, ‘where‘: 3, ‘follow‘: 1, ‘drop‘: 3, ‘loud‘: 1, ‘freedom‘: 2, ‘near‘: 1, ‘while‘: 1, ‘do‘: 1, ‘it‘: 5, ‘sleep‘: 3, ‘failed‘: 1, ‘said‘: 5, ‘but‘: 3, ‘true‘: 1, ‘far‘: 1, ‘this‘: 3, ‘can‘: 10, ‘for‘: 3, ‘burning‘: 3, ‘from‘: 3, ‘love‘: 3, ‘all‘: 5, ‘to‘: 2, ‘loved‘: 1, ‘music‘: 1, ‘soul‘: 1, ‘so‘: 7, ‘skin‘: 3, ‘crush‘: 4, ‘touch‘: 3, ‘cherised‘: 1, ‘in‘: 7, ‘quit‘: 1, ‘enough‘: 3, ‘oh‘: 6, ‘am‘: 1, ‘weak‘: 1, ‘we‘: 3, ‘heart‘: 1, ‘eyes‘: 4, ‘not‘: 2, ‘yourself‘: 1, ‘now‘: 2, ‘seize‘: 1, ‘when‘: 3, ‘never‘: 1, ‘ask‘: 3, ‘head‘: 3, ‘a‘: 1, ‘get‘: 3, ‘if‘: 6, ‘night‘: 3, ‘faith‘: 1, ‘rock‘: 4, ‘predict‘: 3, ‘ll‘: 4, ‘hate‘: 1, ‘masquerade‘: 3, ‘passion‘: 1, ‘of‘: 3, ‘have‘: 1, ‘go‘: 1, ‘trapped‘: 1, ‘re‘: 4, ‘what‘: 3, ‘dreaming‘: 2, ‘dreeeming‘: 1, ‘alone‘: 3, ‘dangerous‘: 3, ‘sweet‘: 1, ‘against‘: 4, ‘tought‘: 1, ‘are‘: 6, ‘thought‘: 2, ‘your‘: 16, ‘falls‘: 3, ‘sure‘: 2, ‘life‘: 1, ‘makes‘: 1, ‘ahead‘: 1, ‘still‘: 3, ‘give‘: 1, ‘wings‘: 1, ‘let‘: 1, ‘keeps‘: 3, ‘much‘: 3, ‘crowd‘: 1, ‘just‘: 3, ‘im‘: 2

输出词频最大TOP20

for i in range(20):
    print(dictList[i])

结果如下：

(‘i‘, 20)
(‘your‘, 16)
(‘you‘, 12)
(‘the‘, 11)
(‘my‘, 11)
(‘t‘, 10)
(‘can‘, 10)
(‘in‘, 7)
(‘so‘, 7)
(‘are‘, 6)
(‘oh‘, 6)
(‘if‘, 6)
(‘it‘, 5)
(‘said‘, 5)
(‘all‘, 5)
(‘s‘, 4)
(‘crush‘, 4)
(‘re‘, 4)
(‘against‘, 4)
(‘ll‘, 4)

将分析对象存为utf-8编码的文件，通过文件读取的方式获得词频分析内容。

fo = open(‘music.txt‘, ‘r‘)
file = fo.read()
fo.close()
print(file)