datapipe

dataPipe - is a fluent data Api to manipulate, transform and process data in JavaScript

import { dataPipe } from 'datapipe-js'

const result = dataPipe()
        .parseCsv(sampleCsv)
        .tap(data => console.log('Original Parsed Data => ', data))
        .groupBy(r => r.Country)
        .select(g => ({
            orderDate: first(g).OrderDate,
            country: first(g).Country,
            sales: Math.round(dataPipe(g).sum(i => i.Sales), 2),
            averageSales: Math.round(avg(g, i => i.Sales), 2),
            count: g.length
        })
        )
        .where(r => r.sales > 5000)
        .sort("sales DESC")
        .toArray();

Alternatively you can call each function separately. The following code returns the same results

import { dataPipe } from 'datapipe-js'
import { first, avg, sum, select, groupBy, sort } from 'datapipe-js/array'
import { parseCsv } from 'datapipe-js/utils'


let data = parseCsv(sampleCsv);
const groups = groupBy(data, r => r.Country)

data = select(groups, g => ({
            orderDate: first(g).OrderDate,
            country: first(g).Country,
            sales: Math.round(dataPipe(g).sum(i => i.Sales), 2),
            averageSales: Math.round(avg(g, i => i.Sales), 2),
            count: g.length
        })
        )
        .where(r => r.sales > 5000)

data = sort(data, "sales DESC")

DataPipe Initialization#

dataPipe#

dataPipe(array: any[] = []): DataPipe

Creates new data pipe object initialized with values. Returns continuous DataPipe object

fromCsv#

Parses CSV string and load items into DataPipe.

Uses parseCsv. Returns continuous DataPipe object

fromCsv(content: string, options?: ParsingOptions): DataPipe

fromTable#

Loads dataPipe with Table information.

Uses fromTable. Returns continuous DataPipe object

rowsOrTable a datarows with 2 dimentional arrays or entire table. If you provide rows, then you have to specify fieldNames
fieldNames fieldNames what correspond to the rows
fieldDataTypes fieldNames what correspond to the rows

  fromTable(
    rowsOrTable: PrimitiveType[][] | TableDto,
    fieldNames?: string[],
    fieldDataTypes?: DataTypeName[]
   ): DataPipe

DataPipe Transformation functions#

groupBy#

Groups array items based on elementSelector function.

Uses groupBy function and returns continuous DataPipe object

groupBy(groupByFields: string | string[] | Selector): DataPipe

pivot#

Returns a reshaped array based on unique column values.

Uses pivot function and returns continuous DataPipe object

pivot(rowFields: string | string[],columnField: string, dataField: string,
    aggFunction?: (array: any[]) => any | null, columnValues?: string[]): DataPipe

rowFields row fields (or index fields). It can be one or more field names
columnField a field which values will be used to create columns
dataField a dataField which will be aggrated with aggregate function and groupped by rows and columns
aggFunction an aggregation function. Default value is sum. data field will be aggregated by this function
columnValues an optional initial column values. Use it to define a set of columns/values you would expect

transpose#

Returns a transposed array, where rows become columns.

Uses transpose function and returns continuous DataPipe object

transpose(): DataPipe 

select#

Creates new array based on selector.

elementSelector A Selector function that is invoked per iteration.

Uses select function and returns continuous DataPipe object

select(selector: string | string[] | Selector): DataPipe

map#

Alias for select and has same usage

where#

Filters array based on predicate function.

Uses where function and returns continuous DataPipe object

predicate A predicate function to filter items.

where(predicate: Predicate): DataPipe

filter#

Alias for where and has same usage

flattern#

Returns a flaterned array.

Uses flattern function and returns continuous DataPipe object

array The array to flatten recursively.

flatten(): DataPipe 

sort#

A simple sort array function with a convenient interface.

Uses sort function and returns continuous DataPipe object

fields field names with a sort order.

sort(...fields: string[]): DataPipe

example

 sort('name ASC', 'age DESC');

leftJoin#

Returns a joined array with all elements from the left array (leftArray), and the matched elements from the right array (rightArray). The result is NULL from the right side, if there is no match.

Uses leftJoin function and returns continuous DataPipe object

rightArray array for right side in a join
leftKey A key from left side array. What can be as a fieldName, multiple fields or key Selector
rightKey A key from right side array. what can be as a fieldName, multiple fields or key Selector
resultSelector A callback function that returns result value */

leftJoin(
    rightArray: any[],
    leftKeySelector:  string | string[] | Selector<any, string>,
    rightKeySelector: string | string[] | Selector<any, string>,
    resultSelector: (leftItem: any, rightItem: any) => any
): DataPipe

innerJoin#

Joins two arrays together by selecting elements that have matching values in both arrays. If there are elements in any array that do not have matches in other array, these elements will not be shown!

Uses innerJoin function and returns continuous DataPipe object

leftArray array for left side in a join
rightArray array for right side in a join
leftKey A key from left side array. What can be as a fieldName, multiple fields or key Selector
rightKey A key from right side array. what can be as a fieldName, multiple fields or key Selector
resultSelector A callback function that returns result value */

innerJoin(
    rightArray: any[],
    leftKey: string | string[] | Selector<any, string>,
    rightKey: string | string[] | Selector<any, string>,
    resultSelector: (leftItem: any, rightItem: any) => any
): DataPipe

fullJoin#

Returns all elements from the left array (leftArray), and all elements from the right array (rightArray). The result is NULL from the right/left side, if there is no match.

Uses fullJoin function and returns continuous DataPipe object

rightArray array for right side in a join
leftKey A key from left side array. What can be as a fieldName, multiple fields or key Selector
rightKey A key from right side array. what can be as a fieldName, multiple fields or key Selector
resultSelector A callback function that returns result value */

fullJoin(
    rightArray: any[],
    leftKey: string | string[] | Selector<any, string>,
    rightKey: string | string[] | Selector<any, string>,
    resultSelector: (leftItem: any, rightItem: any) => any
): DataPipe 

merge#

Merges elements from two arrays. It appends source element or overrides to target array based on matching keys provided.

Uses merge function and returns continuous DataPipe object

sourceArray source array
targetKey tartget key field, arry of fields or field serlector
sourceKey source key field, arry of fields or field serlector

merge(
    sourceArray: any[],
    targetKey: string | string[] | Selector<any, string>,
    sourceKey: string | string[] | Selector<any, string>
): DataPipe

Aggregation and stats functions#

sum#

Sum of items in array.

Uses sum function. This method stops DataPipe's continuity and returns result value

elementSelector Function invoked per iteration.

sum(field?: Selector | string): number | null

avg#

Average of array items.

Uses avg function. This method stops DataPipe's continuity and returns result value

elementSelector Function invoked per iteration.

avg(field?: Selector | string): number | null

average#

Alias for avg and has same usage

min#

Computes the minimum value of array.

elementSelector Function invoked per iteration.

Uses min function. This method stops DataPipe's continuity and returns result value

min(field?: Selector | string): number | Date | null

max#

Computes the maximum value of array.

elementSelector Function invoked per iteration.

Uses max function. This method stops DataPipe's continuity and returns result value

max(field?: Selector | string): number | Date | null

count#

Count of elements in array with optional predicate.

predicate Predicate function invoked per iteration.

Uses count function. This method stops DataPipe's continuity and returns result value

count(predicate?: Predicate): number | null 

first#

Gets first item in array that satisfies optional predicate.

predicate Predicate function invoked per iteration.

Uses first function. This method stops DataPipe's continuity and returns result value

first(predicate?: Predicate): T | null 

last#

Gets last item in array that satisfies optional predicate.

predicate Predicate function invoked per iteration.

Uses last function. This method stops DataPipe's continuity and returns result value

last(predicate?: Predicate): T | null 

countBy#

Gets counts map of values returned by elementSelector.

elementSelector Function invoked per iteration.

Uses countBy function. This method stops DataPipe's continuity and returns result value

countBy(elementSelector: Selector): Record<string, number>

mean#

Get mean of an array.

field Property name or Selector function invoked per iteration.

Uses mean function. This method stops DataPipe's continuity and returns result value

mean(field?: Selector | string): number | null 

quantile#

Get quantile of a sorted array.

field Property name or Selector function invoked per iteration.
p quantile.

Uses quantile function. This method stops DataPipe's continuity and returns result value

quantile(p: number, field?: Selector | string): number | null

variance#

Get sample variance of an array.

field Property name or Selector function invoked per iteration.

Uses variance function. This method stops DataPipe's continuity and returns result value

variance(field?: Selector | string): number | null

stdev#

Get the sample standard deviation of an array.

field Property name or Selector function invoked per iteration.

Uses stdev function. This method stops DataPipe's continuity and returns result value

stdev(field?: Selector | string): number | null

median#

Get median of an array.

field Property name or Selector function invoked per iteration.

Uses median function. This method stops DataPipe's continuity and returns result value

median(array: any[], field?: Selector | string): number | null 

DataPipe output#

toArray#

Returns a result as an array and stops DataPipe continuity

toArray(): any[]

toCsv#

Returns a result as CSV and stops DataPipe continuity.

It uses toCsv

 toCsv(delimiter = ','): string

toObject#

Returns a result as JS Object and stops DataPipe continuity.

keyField a key selector represented as a string (field name) or array of stringa (fieldNames) or custom selectors

It uses toObject

toObject(keyField: string | string[] | Selector<any, string>): Record<string, any>

toTable#

Returns a result as TableDto and stops DataPipe continuity.

It uses toTable

toTable(): TableDto

DataType other methods#

tap#

This method allows you to examine a a state of the data during pipe execution.

tap(dataFunc: (d: any[]) => void): DataPipe 