Links tagged “data-processing”
31 links, newest first.
github.com
A Rust library that tells scanned PDFs from text-based ones and extracts their text, so a pipeline can route each file to the right handler. Node, Python and WASM bindings.
pbpython.com
Explains how pandas' transform() paired with groupby() returns a result aligned to the original rows, unlike aggregation, using percentage-of-total as the example.
oreilly.com
Sets out the vocabulary of unbounded data processing — event time versus processing time, windowing, correctness — and argues streaming can subsume batch.
oreilly.com
Second part, working through windowing, watermarks, triggers and accumulation modes on concrete examples of out-of-order stream processing.
aws.amazon.com
Announces general availability of AWS Glue, a managed serverless ETL service that infers source schemas, generates transformation scripts and runs the jobs.
docs.scipy.org
NumPy reference page on universal functions, the vectorised element-wise operations over ndarrays, and their broadcasting, type-casting and output rules.
nikgrozev.com
Illustrated walkthrough of the pandas reshaping operations — pivot, pivot_table, stack and unstack — and what each one does to a dataframe.
bonobo-project.org
Lightweight ETL toolkit for Python 3.5+ where each transformation is a plain callable and pipelines are built by connecting those callables in a directed graph.
nifi.apache.org
Home page of Apache NiFi, an open-source system for automating the flow of data between systems, designed and monitored from a browser-based interface.
github.com
A curated list of notable ETL (extract, transform, load) frameworks, libraries and software.
pbpython.com
A tutorial on improving the formatting of the Excel files pandas writes, using the XlsxWriter library to style the exported DataFrames.
dask.pydata.org
Documentation home for Dask, a Python library for parallel and distributed computing, offering pandas-like DataFrames, NumPy-like arrays, bags and futures.
distributed.readthedocs.io
Documentation for Dask.distributed, the library that executes Dask task graphs across a cluster of machines.
en.wikipedia.org
Wikipedia article on record linkage — matching records that refer to the same entity across separate data sources, and the deterministic and probabilistic methods used to do it.
www-nlp.stanford.edu
Stanford's GloVe project — an unsupervised algorithm deriving word vectors from co-occurrence statistics, with pre-trained vectors to download.
streamhacker.com
Walkthrough of training gensim's word2vec on NLTK corpora and querying the resulting embeddings for similarly-used words.
developers.lyst.com
How Lyst applied word2vec and GloVe to its fashion catalogue to surface semantic relations between garments, designers and fashion vocabulary.
research.google.com
Google's VLDB 2010 paper on Dremel, a scalable interactive ad-hoc query system for read-only nested data that scans trillion-row tables in seconds.
dask.readthedocs.io
Documentation for Dask, a Python library for parallel and distributed computing that scales data processing from a laptop to a cluster.
pandas.pydata.org
Documentation for pandas, the Python library providing DataFrames and the tools that go with them for practical data analysis.
