TechOverflow Logo
TechOverflow

Como analisar todos os arquivos baseline do PubMed em paralelo usando Python

English Deutsch Français 简体中文 Español Українська Čeština Русский Português

Em nosso post anterior Como analisar dados baseline do PubMed usando Python investigamos como usar a biblioteca pubmed_parser para analisar dados medline do PubMed usando Python.

Neste acompanhamento forneceremos um exemplo de como usar glob para selecionar todos os arquivos baseline do PubMed em um diretório e usar concurrent.futures com tqdm para fornecer um paralelismo de processo conveniente mas fácil de usar usando ProcessPoolExecutor e uma interface de barra de progresso para a linha de comando.

Primeiro, instale os requisitos usando

install_pubmed_parser.sh
pip install git+git://github.com/titipata/pubmed_parser.git six numpy tqdm

Agora baixe este script, certifique-se de que alguns arquivos como pubmed20n0002.xml.gz ou pubmed20n0004.xml.gz estão no mesmo diretório e execute-o:

parse_pubmed_parallel.py
#!/usr/bin/env python3
import pubmed_parser as pp
import glob
import os
from collections import Counter
import concurrent.futures
from tqdm import tqdm

# Source: https://techoverflow.net/2017/05/18/how-to-use-concurrent-futures-map-with-a-tqdm-progress-bar/
def tqdm_parallel_map(executor, fn, *iterables, **kwargs):
    """
    Equivalent to executor.map(fn, *iterables),
    but displays a tqdm-based progress bar.

    Does not support timeout or chunksize as executor.submit is used internally

    **kwargs is passed to tqdm.
    """
    futures_list = []
    for iterable in iterables:
        futures_list += [executor.submit(fn, i) for i in iterable]
    for f in tqdm(concurrent.futures.as_completed(futures_list), total=len(futures_list), **kwargs):
        yield f.result()

def parse_and_process_file(filename):
    """
    This function contains our parsing code. Usually, you would only modify this function.
    """
    # Don't parse authors and references for this example, since we don't need it
    dat = pp.parse_medline_xml(filename, author_list=False, reference_list=False)

    # For this example, we'll build a set of count of all MeSH IDs in this file
    ctr = Counter()
    for entry in dat:
        terms = [term.partition(":")[0].strip() for term in entry["mesh_terms"].split(";")]
        for term in terms:
            ctr[term] += 1
    return filename, ctr


if __name__ == "__main__":
    # Find all pubmed files in the current directory
    all_filenames = glob.glob("pubmed*.xml.gz")
    # For some workloads you might want to use a ThreadPoolExecutor,
    # but a ProcessPoolExecutor is a good default
    executor = concurrent.futures.ProcessPoolExecutor(os.cpu_count())
    # Iterate results as they come in (the order is not the same as in the input!)
    for filename, ctr in tqdm_parallel_map(executor, parse_and_process_file, all_filenames):
        # NOTE: If you print() here, this might interfere with the progress bar,
        # but we accept that here since it's just an example
        print(filename, ctr)

Agora você pode começar a modificar o exemplo, mais notavelmente a função parse_and_process_file() para fazer qualquer processamento que você pretende fazer.

Check out similar posts by category: Bioinformatics, Python