複数の（特殊）文字を置き換える-最も効率的な方法は？

私が持っているテキストでは、次の特殊文字を1つのスペースに置き換えたいと思います。

symbols = ["`", "~", "!", "@", "#", "$", "%", "^", "&", "*", "(", ")", "_", "-", "+", "=", "{", "[", "]", "}", "|", "\\", ":", ";", "\"", "<", ",", ">", ".", "?", "/"]

これを行うための（コード実行時間の観点から）最も効率的な方法は何ですか？

たとえば、私はこれが欲しいです：

(Hello World)] *!

これになるために：

Hello World

候補となる方法は次のようです。

1. リスト内包
2. .replace()
3. .translate()
4. 正規表現

効率的なソリューションstr.maketransとして、これに使用できます。変換テーブルが定義されると、文字列内の文字をマッピングするだけでよいことに注意してください。これを行う方法は次のとおりです。

symbols = ["`", "~", "!", "@", "#", "$", "%", "^", "&", "*", "(", ")", "_", "-", "+",
           "=", "{", "[", "]", "}", "|", "\\", ":", ";", "\"", "<", ",", ">", ".", "?", "/"]

dict.fromkeys各エントリの値として単一のスペースを設定してシンボルから辞書を作成することから始め、辞書から翻訳テーブルを作成します。

d = dict.fromkeys(''.join(symbols), ' ')
# {'`': ' ', ',': ' ', '~': ' ', '!': ' ', '@': ' '...
t = str.maketrans(d)

次に、stringtranslateメソッドを呼び出して、上記の辞書の文字を空のスペースにマップします。

s = '[email protected]!a^[email protected]'
s.translate(t)
# ' this is a test '

いくつかのテストを開始した後、私はそれstr.translate()が最良の変種であると言うことができます。

入力データ：

symbols = {"`", "~", "!", "@", "#", "$", "%", "^", "&", "*", "(", ")", "_", "-", "+", "=", "{", "[", "]", "}", "|", "\\", ":", ";", "\"", "<", ",", ">", ".", "?", "/"} translate_table = {126: None, 93: None, 91: None, 125: None, 92: None, 42: None, 45: None, 94: None, 62: None, 47: None, 35: None, 59: None, 44: None, 58: None, 60: None, 124: None, 61: None, 36: None, 95: None, 43: None, 96: None, 123: None, 64: None, 33: None, 38: None, 63: None, 46: None, 34: None, 41: None, 37: None, 40: None} regular_expression = "[`[email protected]#$%^&*()_\-+={[\]}|\\:;\"<,>.?/]"
small_document = "Some**[email protected]]]\"dom t##xt"
normal_document = "TbsX^Kt$FZ%haZe+sLxu:Al\"xNAL\\Kix[mHp_gn]PrG`DqGd~GdNc;BoEq.SYD?Rp>ukq,UfO<XdTc=RUH}oifc&oP!CB*[email protected]{Qf-Li)gmXL/IQH#mne(Khaj|" big_document = "QOfY+dymyoGBAxTAoIeM+jEWlaECUZEUXuMvprJOqFtQR*OiHtTFZkUNbYipSTTDPOVkIdGTcjWrQmbmthKBHBSEOZ)lQAIJOrVgmGGFdtqbuFfj<Dls<JWtKczAFMPYMemiJBJHdPeeul\\x>lGIBvUsxBokagvVovrrdxdKMtAKx>MEexYv>DGqPUXYaBQKwiSIUobrPQYjilhHMQunE;RiqOZPTnyOEgRrpxcuobvvmGkFpTqgMxYYhrmRRnauiqgvCmZ\"UauceaXsgAMSakxewzPrlIrYkVCVZaEGh][email protected]@hegZbfhsHaJOxOxzQhZAVjVDgokosATfhKMT!WYyPWKcKAHKCzQGGJOCglYGZbftsuyntXZUKNqgGlsLJqgN,pUcOoA/tStXFXgpoSErgvw/OUMPWjJwt=bhMAIDayOZXJm=ifYYUuAvSIZjwnBfktNvEvZmvQso%HiNZEVqoDR%nQBtCkhjSfVfDuRSRsvp-sCunjDDUYSEVLICQdisxhEfqkUTkiPlLiUNNwrvO#WTDmweZyMeIbgNXkIsvaJeHYXV(HvRcGNZM(PPRIAyyLWivGiqMVBtwObqLfEEISyyjGNEdUU:ys`dXcVawkIEAjFXky`RUXNTm`LDM}mwTOcmsSo}haJXPnkwOhKLYwve}SWifzKq}grw}fMSQXXWguUQtlWpPZQymR^wBKEyolFlZnzEEmehSNenOqDOHWRit[Npm?R?DIPXAmQYYBbmJofxUzzWBsVCoPI?VmpXhoMxCfXyHEHowXzIJvExThiffLhBTtma_jk_NrbkPCGGypXvOuBqBxDYfC{bwIHoaqnJSKytxwWXBNnKG~PKuQklGblEwH~rJoGpKZmm~tTEFnPLdmzfrqJibMYIykzL$RZLPmsZjB$AAbZwFnByOydEOIfFvTaEQaSjbpeBZuUGY&ZfPQgLihmPYrhZxSwMzLrNF.WjFiDCLyXksdkLeMHVCfrdgCAotElQ|"
no_match_document = "XOtasggWqhtSLJpHEGoCmMRepFBlRfAGKTLPcEtKonFVsPgvWgAbvJVeMWILPgLapwAmTgXWVbxOJtUFmMygzIqYPqyAxzwElTFyYcGdtnNa"

コード：

def func1(doc):
    for c in symbols:
        doc = doc.replace(c, "")
    return doc


def func2(doc):
    return doc.translate(translate_table)


def func3(doc):
    return re.sub(regular_expression, "", doc)


def func4(doc):
    return "".join(c for c in doc if c not in symbols)

試験結果：

func1(small_document):      0.701037002
func1(normal_document):     1.1260866900000002
func1(big_document):        3.4234831459999997
func1(no_match_document):   0.7740780450000004

func2(small_document):      0.14135037500000003
func2(normal_document):     0.5368806810000004
func2(big_document):        0.8128472860000002
func2(no_match_document):   0.394245089

func3(small_document):      0.3157141610000007
func3(normal_document):     0.927359323000001
func3(big_document):        1.9310377590000005
func3(no_match_document):   0.18656399199999996

func4(small_document):      0.3034549070000008
func4(normal_document):     1.3695875739999988
func4(big_document):        10.115730064
func4(no_match_document):   1.2086623230000022

UPD。

私が提供した入力データは、純粋なメソッドテスト用に特別に「準備」されています。

生成するtranslate_tableために、次のdict理解を使用しました：

translate_table = {ord(s): None for s in symbols}

ここでは正規表現の検証のためのウェブサイトへのリンク（それが役に立つかもしれないが）です。

自分でテストを再計算したい場合は、次のコードを使用してください。

    if __name__ == '__main__':
    import timeit
    print("func1(small_document)", timeit.timeit("func1(small_document)", setup="from __main__ import func1, small_document", number=100000))
    print("func1(normal_document): ", timeit.timeit("func1(normal_document)", setup="from __main__ import func1, normal_document", number=100000))
    print("func1(big_document): ", timeit.timeit("func1(big_document)", setup="from __main__ import func1, big_document", number=100000))
    print("func1(no_match_document): ", timeit.timeit("func1(no_match_document)", setup="from __main__ import func1, no_match_document", number=100000))

    print("func2(small_document): ", timeit.timeit("func2(small_document)", setup="from __main__ import func2, small_document", number=100000))
    print("func2(normal_document): ", timeit.timeit("func2(normal_document)", setup="from __main__ import func2, normal_document", number=100000))
    print("func2(big_document): ", timeit.timeit("func2(big_document)", setup="from __main__ import func2, big_document", number=100000))
    print("func2(no_match_document): ", timeit.timeit("func2(no_match_document)", setup="from __main__ import func2, no_match_document", number=100000))

    print("func3(small_document): ", timeit.timeit("func3(small_document)", setup="from __main__ import func3, small_document", number=100000))
    print("func3(normal_document): ", timeit.timeit("func3(normal_document)", setup="from __main__ import func3, normal_document", number=100000))
    print("func3(big_document): ", timeit.timeit("func3(big_document)", setup="from __main__ import func3, big_document", number=100000))
    print("func3(no_match_document): ", timeit.timeit("func3(no_match_document)", setup="from __main__ import func3, no_match_document", number=100000))

    print("func4(small_document): ", timeit.timeit("func4(small_document)", setup="from __main__ import func4, small_document", number=100000))
    print("func4(normal_document): ", timeit.timeit("func4(normal_document)", setup="from __main__ import func4, normal_document", number=100000))
    print("func4(big_document): ", timeit.timeit("func4(big_document)", setup="from __main__ import func4, big_document", number=100000))
    print("func4(no_match_document): ", timeit.timeit("func4(no_match_document)", setup="from __main__ import func4, no_match_document", number=100000))

s = '''
def translate_():
    symbols = '`,~,!,@,#,$,%,^,&,*,(,),_,-,+,=,{,[,],},|,\,:,;,",<,,,>,.,?,/' s = '[email protected]!a^test @' t = str.maketrans(dict.fromkeys(symbols, ' ')) s.translate(t) return s def replace_(): symbols = '`,~,!,@,#,$,%,^,&,*,(,),_,-,+,=,{,[,],},|,\,:,;,",<,,,>,.,?,/'
    s = '[email protected]!a^test @'
    for symbol in symbols:
        s = s.replace(symbol, ' ')
    return s
'''

print(timeit.timeit('replace_()', setup=s, number=100000))
print(timeit.timeit('translate_()', setup=s, number=100000))

印刷します：

0.7663131961598992

0.4139239452779293

したがって、に置き換えるのtranslateは、数replace秒を使用するよりもほぼ2倍高速です。

私のコードは記号をスペースに置き換え、それらのスペースを削除しません。

短い文字列の場合.join()は高速ですが、大きな文字列の.translate()場合は、置き換えるものが多い場合は高速です。驚くべきことに、.replace()交換がほとんどない場合でも、非常に高速です。

text: '(Hello World)] *!'
using_replace                     0.046
using_join                        0.016
using_translate                   0.031

text: '[email protected]!a^[email protected]'
using_replace                     0.046
using_join                        0.017
using_translate                   0.029

text: '~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&t[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]~/()&[email protected]!&=)(/as/dw&%#a^[email protected]'
using_replace                     0.195
using_join                        2.327
using_translate                   0.061

text: 'a long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replacea long text without chars to replace'
using_replace                     0.051
using_join                        2.100
using_translate                   0.064

いくつかの戦略の比較：

def using_replace(text, symbols_to_replace, replacement=' '):
    for char in symbols_to_replace:
        text = text.replace(char, replacement)

    return text

def using_join(text, symbols_to_replace, replacement=' '):
    return ''.join(
        replacement if char in symbols_to_replace else char
        for char in text)

def using_translate(text, symbols_to_replace, replacement=' '):
    translation_dict = str.maketrans(
        dict.fromkeys(symbols_to_replace, replacement))

    return text.translate(translation_dict)

timeitさまざまなテキストのこのコードで：

    # a 'set' for faster lookup
    symbols = {
        '`', '~', '!', '@', '#', '$', '%', '^', '&', '*',
        '(', ')', '_', '-', '+', '=', '{', '[', ']', '}',
        '|', '/', ':', ';', '"', '<', ',', '>', '.', '?',
        '\\',
    }

    text_list = [
        '(Hello World)] *!',
        '[email protected]!a^[email protected]',
        '~/()&[email protected]!&=)(/as/dw&%#a^[email protected]' * 1000,
        'a long text without chars to replace' * 1000,
    ]
    for s in text_list:
        assert (
                using_replace(s, symbols)
                == using_join(s, symbols)
                == using_translate(s, symbols))

    for s in text_list:
        print()
        print('text:', repr(s))
        for func in [using_replace, using_join, using_translate]:
            t = timeit.timeit(
                'func(s, symbols)',
                'from __main__ import func, s, symbols',
                number=10000)
            print('{:30s} {:8.3f}'.format(func.__name__, t))

str.translate（）は確かに最速のメソッドです。文字を除外するための変換テーブルを作成する簡潔な方法は次のとおりです。

symbols = ["`", "~", "!", "@", "#", "$", "%", "^", "&", "*", "(", ")", "_", "-", "+", "=", "{", "[", "]", "}", "|", "\\", ":", ";", "\"", "<", ",", ">", ".", "?", "/"]
removeSymbols = str.maketrans("","","".join(symbols))

cleanText = "[Hello World] *!".translate(removeSymbols)
print(cleanText) # "Hello World "

maketrans（）関数は3つのパラメーターを取ることができます。最初のパラメーターは置換する文字を含む文字列、2番目のパラメーターはそれらの置換、3番目のパラメーターは削除する必要のある文字のリストです。すべての文字を率直に削除するには、削除する記号を含む文字列を3番目のパラメーターに指定する必要があります。

次に、変換テーブルremoveSymbolsは、シンボルリスト内の文字を完全に削除します。

スペースに置き換えるには、次のように変換テーブルを作成します。

removeSymbols = str.maketrans("".join(symbols)," "*len(symbols))