TechOverflow Logo
TechOverflow

Como parse all PubMed baseline files in parallel usando Python

English Deutsch Français 简体中文 Українська Čeština Português Русский Español

In nuestro previous post Como parse PubMed baseline data usando Python investigamos como usar el pubmed_parser library para parse PubMed medline data usando Python.

In este follow-up proporcionaremos un example de como usar glob para select all PubMed baseline files in un directory y usar concurrent.futures con tqdm para provide un convenient yet easy-to-use process parallelism usando ProcessPoolExecutor y un progress bar UI para el command line.

First, install los requirements usando

install_pubmed_parser.sh
pip install git+git://github.com/titipata/pubmed_parser.git six numpy tqdm

Now download este script, ensure some files como pubmed20n0002.xml.gz o pubmed20n0004.xml.gz estan in el same directory y run lo:

parse_pubmed_parallel.py
#!/usr/bin/env python3
import pubmed_parser as pp
import glob
import os
from collections import Counter
import concurrent.futures
from tqdm import tqdm

# Source: https://techoverflow.net/2017/05/18/how-to-use-concurrent-futures-map-with-a-tqdm-progress-bar/
def tqdm_parallel_map(executor, fn, *iterables, **kwargs):
    """
    Equivalent a executor.map(fn, *iterables),
    pero displays un tqdm-based progress bar.

    Does not support timeout o chunksize as executor.submit es used internally

    **kwargs es passed a tqdm.
    """
    futures_list = []
    for iterable in iterables:
        futures_list += [executor.submit(fn, i) for i in iterable]
    for f in tqdm(concurrent.futures.as_completed(futures_list), total=len(futures_list), **kwargs):
        yield f.result()

def parse_and_process_file(filename):
    """
    This function contiene nuestro parsing code. Usually, solo modifyarias esta function.
    """
    # Don't parse authors y references para este example, since no lo necesitamos
    dat = pp.parse_medline_xml(filename, author_list=False, reference_list=False)

    # Para este example, buildaremos un set de count de all MeSH IDs in este file
    ctr = Counter()
    for entry in dat:
        terms = [term.partition(":")[0].strip() for term in entry["mesh_terms"].split(";")]
        for term in terms:
            ctr[term] += 1
    return filename, ctr


if __name__ == "__main__":
    # Find all pubmed files in el current directory
    all_filenames = glob.glob("pubmed*.xml.gz")
    # Para some workloads podrias want usar un ThreadPoolExecutor,
    # pero un ProcessPoolExecutor es un good default
    executor = concurrent.futures.ProcessPoolExecutor(os.cpu_count())
    # Iterate results as vienen in (el order no es el same as in el input!)
    for filename, ctr in tqdm_parallel_map(executor, parse_and_process_file, all_filenames):
        # NOTE: If print() here, esto might interfere con el progress bar,
        # pero acceptamos eso here since es just un example
        print(filename, ctr)

Now puedes start modifying el example, most notably el parse_and_process_file() function para do whatever processing intendes do.

Echa un vistazo a artículos similares por categoría: Bioinformatics, Python