STRAWMAN: pipe method

[benlesh]

Background

There is a consistent problem with users using patch operators (rxjs/add/operator/map for example) and decoratiing the prototype, then having a consumer of their code depend on that map operator being there, without ensure that it is. This, in particular affects projects like Angular and Falcor (and I'm sure others).

After talking with @IgorMinar about ways to solve this, a solution was proposed that's not much different than the op PR found here #2034. It does, however, look cleaner with a few exceptions.

Proposal

• adds a compose utility function to Rx. This is used to compose a rest args of functions into a single function like so:

const double = (n: number) => n + n;
const toString = (n: number) => '' + n;
const toFixed = (size: number) =>  (n: number) => n.toFixed(size);
const concat = (...append: string[]) => (s: string) => [s, ...append].join('');

const composed = Rx.compose(double, toString, toFixed(2), concat('!!!'));

composed(2); // "4.00!!!"
composed(4); // "8.00!!!"

• adds a pipe method to Observable. This pipe method will take a rest args of "pipeable operators", which are higher-order functions that return (source: Observable<T>) => Observable<R>. This, internally, uses the compose function above.
• adds a Rx.Pipe.map and rxjs/pipe/map function that is a higher order function that returns a "pipeable operator". (used in the example above)

Using Pipe:

import { Observable } from 'rxjs/Observable';
import { map } from 'rxjs/pipe/map';

Observable.of(1, 2, 3)
  .pipe(
    map((x: number) => x + x),
    map((x: number) => x * x),
    map((x: number) => x + '!!!')
  )
  .subscribe(x => console.log(x));

// "4!!!"
// "16!!!"
// "36!!!"

vs the op/operate proposal:

import { Observable } from 'rxjs/Observable';
import { map } from 'rxjs/operator/map';

Observable.of(1, 2, 3)
  .op(map, (x: number) => x + x)
  .op(map, (x: number) => x * x)
  .op(map, (x: number) => x + '!!!')
  .subscribe(x => console.log(x));

// "4!!!"
// "16!!!"
// "36!!!"

Pros

1. Will allow fluent composition of Rx operators, including some functional programming goodness in combining them and dealing with them.
2. Prevents prototype mutation and the problems there in
3. It doesn't look that much different than using operators on the prototype (as opposed to the op/operate solution, which relies on comma separation.
4. Less verbose than the op/operate solution.

Cons

1. It's a layer of indirection over top of what we already have
2. We lose some type safety, because there isn't a great way to carry the type from one argument function that's passed topipe to the next.
3. Yet another way to use operators.
4. We'll need to wrap all of the existing operators as pipable.

Other Things

This PR adds the ability to compare two test observables via expectObservable(obs1).toBe(obs2). This was done to make it easier to test pipe operators as they are added.

Using Rx.compose in unison with Observable.prototype.pipe is a fairly powerful combination, enabling a user to compose a chain of operations and reuse it much more dynamically than they might be able to otherwise.

[benlesh]

Another con worth noting in comparison to the op/operator proposal is it loses type inference between operations. Which is a bit annoying if you're a TypeScript user.

[coveralls]

Coverage increased (+0.007%) to 97.696% when pulling 647e4e7 on benlesh:moar-functional into 01e1343 on ReactiveX:master.

[chrisnoringovo]

I like it pipe, no op(), would have been nice to call it more what it does like applyAll, pipe is probably an established name but it doesn't hurt to think wether ppl will have to look at the manual or intuitively understand just based on the name. Apply is a dangerous name in javascript land though :)

[jayphelps]

Two more random pitches:

Operator patching, but they must use a symbol to access the property.

If this is used, the operator is added to the prototype under a symbol, so anyone else using it doesn't need to know if it's been imported before or not, but they cannot accidentally depend on the operator existing without importing it themselves!

I'm initially pretty drawn to this solution until (if...) we get some sort of pipe/bind operator in JS. Though indeed this is not yet typesafe in TS

import { map } from 'rxjs/symbols/operator/map';

Observable.of(1, 2, 3)
  [map](x: number) => x + x)
  [map](x: number) => x * x)
  [map](x: number) => x + '!!!')
  .subscribe(x => console.log(x));

Weird chaining thing

I don't know what to call this, but I believe someone else suggested it in earlier discussions.

The main con to this is that the average JS user will probably be like wtf how does this work, even though it's really fairly simple chaining.

We could name it .lisp()

Observable.prototype.sorcery = function () {
  const chain = (operator, ...args) => operator.apply(this, args).sorcery();
  chain.subscribe = (...args) => this.subscribe(...args);
  return chain;
};

Observable.of(1, 2, 3)
  .sorcery()
  (map, (x: number) => x + x)
  (map, (x: number) => x * x)
  (map, (x: number) => x + '!!!')
  .subscribe(x => console.log(x));

[benlesh]

The symbol one is interesting, but in IE they'd end up with the same problem (depending on the Symbol polyfill). I like it because it was one of the first things I thought of. "Hey I want to alter the prototype without trampling other people's business"

[jayphelps]

@benlesh hmm I don't follow about IE. Even if the polyfill just used a unique string, they would have to know that unique string and use it directly without importing, which people wouldn't do?

Observable.of(1, 2, 3)
  ['i-am-a-fake-symbol-for-map'](x: number) => x + x)
  ['i-am-a-fake-symbol-for-map'](x: number) => x * x)
  ['i-am-a-fake-symbol-for-map'](x: number) => x + '!!!')
  .subscribe(x => console.log(x));

[xgrommx]

@benlesh, @jachenry Seriously? this is Short Fusion (and second law of Functor).

Observable.of(1,2,3).map(compose(f1, f2, f3)).subscribe(x => console.log(x))

For flatMap exists Kleisli composition. (Also let, publish will be helpful)

[benlesh]

@jayphelps some folks replace Symbol with "@@whatever" without some unique identifier on them. I've seen some really bad Symbol polyfills.

[jayphelps]

@benlesh Yeah, I can imagine. ☹️ but I don't feel people will rely on using operators without symbols because of their bad polyfill. The symbol Rx uses could be something like Symbol('@@map') or whatever, and if people used a polyfill that simply used that key as-is would have to know that, and use it. Which would be really really weird and IMO we can't always protect users from doing stupid things.

import 'bad-symbol-polyfill';

// I don't think people will ever do this, at least without knowing they're
// being hacky and that's on them
Observable.of(1, 2, 3)
  ['@@map'](x: number) => x + x)
  ['@@map'](x: number) => x * x)
  ['@@map'](x: number) => x + '!!!')
  .subscribe(x => console.log(x));

I think the primary concern should be to prevent stream.map from being available, so people need to either use the symbol (as desired) or be super hacky in a way that will only work in browsers that don't support Symbols--so their hacky shit wouldn't work in Chrome/Firefox/etc.

[staltz]

Really having a hard time understanding why we can't just use let for these purposes. Please can someone explain it to me like I'm five?

There is a consistent problem with users using patch operators (rxjs/add/operator/map for example) and decoratiing the prototype, then having a consumer of their code depend on that map operator being there, without ensure that it is. This, in particular affects projects like Angular and Falcor (and I'm sure others).

Seems like let(f) would solve this as long as f is a thisless function.

[staltz]

For reference, this is what we do in xstream (xstream compose is rxjs let):

• Usage https://github.com/staltz/xstream/blob/8f2503a4cc9ce97eadb2ba69f12a3df98b746c23/tests/extra/debounce.ts#L17
• Implementation https://github.com/staltz/xstream/blob/8f2503a4cc9ce97eadb2ba69f12a3df98b746c23/src/extra/debounce.ts#L93-L97

If the closure in the implementation is an issue for perf, we can also return a function or an Operator class with a call method, because then let could always f.call, no matter if f is a function or an operator class instance.

PS: and TypeScript stays happy.

[alancnet]

@staltz I agree. .let is the correct operator to use here. Beyond that, @benlesh proposed a compose function that would take normal scalar functions and bind them together. The problem here is that's too much responsibility for an operator. By all means, compose your functions together before you pass them into let if you want to.. Or use .let(fn1).let(fn2).let(fn3) which (in my opinion) is more readable, testable, and maintainable.

[alancnet]

Also, I must say, regarding his implementation of map:

export function map<T, R>(
  project: (value: T, index: number) => R,
  thisArg?: any
): (source: Observable<T>) => Observable<R> {
  return (source: Observable<T>) => mapProto.call(source, project);
} 

This seems to imply a too-high-order function. Does this function not ignore source and return a new function that takes a new source?

pipe is already a higher-order function. Why pass more higher-order functions into it?

[david-driscoll]

@alancnet

Some of the typings seem a bit verbose...

export function map<T, R>(
  project: (value: T, index: number) => R,
  thisArg?: any
) {
  return (source: Observable<T>) => mapProto.call(source, project);
} 
// Which resolves to something that behaves like...
map(x => !!x)(source);

// so then pipe will see a set of types like...
let pipes: ((source: Observable<T>) => Observable<T>)[]

[david-driscoll]

Overall I'm not a huge fan of the proposal simply because it makes typings manual in all situations (I tried again and couldn't make it work sadly 😞)

However if this is the way we want to go, here is a playground that allows us to flow the types through a little better than what exists in this PR.

Basically it boils down an extended interface for pipe.

interface IPipeOperator<T> {
	<R1>(op1: IPipeMethod<T, R1>): Observable<R1>;
	<R1, R2>(op1: IPipeMethod<T, R1>, op2: IPipeMethod<R1, R2>): Observable<R2>;
	<R1, R2, R3>(op1: IPipeMethod<T, R1>, op2: IPipeMethod<R1, R2>, op3: IPipeMethod<R2, R3>): Observable<R3>;
	<R>(...fns: ((x: Observable<T>) => Observable<any>)[]): Observable<R>;
}

I would recommend we expand this one out to about 9 overloads, to allow for a reasonably large amount of piped operators.

Which resolves down to, something like the following. Basically the types must all be on each operator (which sucks) but then if the types are wrong as they flow through, then you will get compiler errors at the very least.

let o: Observable<string>;
let o2: Observable<number> = o.pipe(
    map<string, string>(x => x),
    map<string, number>(x => x.length * 2) // map<number, number> would be a compiler error
);

Here are some related issues about the matter, it looks like something that may be solved, but I doubt it will be solved anytime soon.

microsoft/TypeScript#9366
microsoft/TypeScript#10247
microsoft/TypeScript#9949 (comment)

[david-driscoll]

Looking further at it, I agree with @staltz that this is essentially let that takes a set of rest args, so we could in theory just extend let (or alias let to pipe) could we not?❓

something like...

pipe(this: Observable<T>, ...fns: ILetMethod[]): Observable<T> {
    const composed = compose.apply(this, fns);
    return composed(this);
}

[jayphelps]

We discussed let before in #2034 (comment)

AFAICT three main reasons were brought up:

• More verbose (OK..not THAT much more..but it's still more)
• Not included by default, but we could include it by default obviously 😝
• Lack of type safety

source
  .let(s => filter.call(s, x => x > 1))
  .let(s => map.call(s, x => x + 10))

// vs.

source
  .pipe(
    filter(x => x > 1),
    map(x => x + 10)
  )

// vs.

source
  .op(filter, x => x > 1)
  .op(map, x => x + 10)

// etc...

I think we need some clarity on the goals of these various proposals. e.g. are we trying to appease library authors themselves who don't want to bleed implementation details aka leak operators or are we trying to create a new paradigm that we'll pitch as the preferred alternative of operator patching? Initially the discussion in #2034 seemed to be the former, but I've seen conversations between Ben and Igor suggesting to me the later is also now a possible goal.

If this is just for library authors, I think type safety is certainly preferred but is less critical than if this is intended for general usage by app devs. I mostly bring this up cause we otherwise might continue being deadlocked on this topic.

[david-driscoll]

Also interestingly, op uses the same higher-order functions for map and so on, that pipe/let would use. This would basically make op a slightly modified version of let.

Roughly...

function op<R>(func: (...args:any[]) => ((source: Observable<T>) => Observable<R>), ...args: any[]): Observable<R> {
    return func(...args)(this);
}

@jayphelps

If let uses the same higher order functions for pipe then they resolve to the same thing roughly.

source
  .let(filter(x => x > 1))
  .let(map(x => x + 10))

or we refactor let to support rest params then it behaves exactly the same.

source
  .let(
     filter(x => x > 1),
     map(x => x + 10)
   );

[mattpodwysocki]

@benlesh why aren't we just using a form of transducers if we're going to a point of operator fusion instead of just some shortcutting?

[jayphelps]

@david-driscoll hmmm from what I can see it isn't the same. Ben's proposed pipe requires new code wrapping all the operators since you're calling the "operator" like a factory map(x => x) whereas op and let use the existing operator .op(map, x => x) .let(self => map.call(self, x => x). Although they are all called map the one .pipe would use is new code that does not yet exist in RxJS--you aren't calling the existing map operator directly, but rather a factory or whatever because you're calling it without any source context.

Edit: it seems you're referring to a new, proposed change/extension to let or something similar, whereas I was referring to let as it exists today and there not yet being any "operator factory" functions map(x => x). The discussion is getting confusing real quick...

[benlesh]

Really having a hard time understanding why we can't just use let for these purposes. Please can someone explain it to me like I'm five?

@staltz is right. compose is just let with rest params. (and arguably a MUCH better name)

[benlesh]

why aren't we just using a form of transducers if we're going to a point of operator fusion instead of just some shortcutting?

Those are fine for synchronous actions... I played with this via a "Scannable" once upon a time. The goal here is to give people easy access to using all of the operators without mutating prototype. This is an ask from the Angular team, as well as others I've talked to that work on libraries that are to be consumed. Falcor-router had issues with this as well. (The issue being people depending on operators they didn't add themselves, and then getting broken when a dependency stops adding the operators)

[jayphelps]

To add to @benlesh description of why something is needed, I feel this is also a major problem for the general user as well when they are testing. If you use operator patching in your tests, your app may be accidentally depending on that operator but not importing itself. All tests pass but your app fails on its own.

[jayphelps]

@alancnet neither provides type safety. let would need to be included by default (without operator patching).

[alancnet]

@jayphelps is that a simple fix?

[jayphelps]

@alancnet for let AFAICT no, you need to use call, which isn't yet type safe but there has been active experimentation on it.

For pipe, I'm not sure. benlesh would have to clarify as I haven't had time to dig in that deeply. He did mention this:

We lose some type safety, because there isn't a great way to carry the type from one argument function that's passed to pipe to the next.

[jayphelps]

@alancnet also, I don't think the ego quip is productive. I don't believe Ben is having ego issues. The problem is complex and has been in discussion for almost a year in many forms, including discussing the overlap with let. Let's just continue hashing out the issues. 👍

Edit: rereading my response, it may come off as a little harsh. Sorry about that. I mostly want to keep the discussion away from personal stuff--I know I would feel upset if someone publicly implied my ego was getting in the way of a productive conversation. He may or may not have. Of course, if you do feel someone is having ego issues (we all do from time to time) you might bring it up to them in private. I would bet most of the time it's just a misunderstanding from lack of context combined with the cold nature of comments in text. Or at least they won't feel publicly attacked and things escalate.

[staltz]

We could just do what @david-driscoll proposed

If let uses the same higher order functions for pipe then they resolve to the same thing roughly.

source
  .let(filter(x => x > 1))
  .let(map(x => x + 10))

Yes it would require some new code for these operator factories, but all these 3 problems would be solved:

@jayphelps

• More verbose
• Not included by default
• Lack of type safety

Or is the concern here that an operator factory is slightly slower? Wouldn't be nice for rxjs users to be confused around let vs op, if they're almost the same thing.

@jayphelps
I think we need some clarity on the goals of these various proposals. e.g. are we trying to appease library authors themselves who don't want to bleed implementation details aka leak operators or are we trying to create a new paradigm that we'll pitch as the preferred alternative of operator patching? Initially the discussion in #2034 seemed to be the former, but I've seen conversations between Ben and Igor suggesting to me the later is also now a possible goal.

Just a playful thought: RxJS could come with zero operators, each operator would live in its own package, then we would have presets. Babel style. And let (or o, or function bind syntax 😢 or ES pipe proposal |> 😢) used for every operator. Certainly would reduce the amount of "operator proposal" issues in this repo. But I'm not actually serious about this one.

[jayphelps]

@staltz If changing let to be the same thing as what the proposed pipe is, Ben had said it wasn't type safe.

We lose some type safety, because there isn't a great way to carry the type from one argument function that's passed to pipe to the next.

I just took a deeper look and can confirm, it doesn't seem possible to make it type safe. Someone who knows more TypeScript tricks might know of some way though.

---

I don't have any concerns about initial performance of applying operators as long as the perf during data flow isn't affected.

[staltz]

No, let would stay untouched, we'd just need higher-order operator function thingies.

[david-driscoll]

We can use the following interface to help enforce "type flow" (in that if the specified types are incorrect, it will be a compiler error). The higher order methods however will have to be strictly typed at creation.

type ILetMethod<T, R> = (x: Observable<T>) => Observable<R>;
interface ILetOperator<T> {
	<R1>(op1: IPipeMethod<T, R1>): Observable<R1>;
	<R1, R2>(op1: IPipeMethod<T, R1>, op2: IPipeMethod<R1, R2>): Observable<R2>;
	<R1, R2, R3>(op1: IPipeMethod<T, R1>, op2: IPipeMethod<R1, R2>, op3: IPipeMethod<R2, R3>): Observable<R3>;
	<R>(...fns: ((x: Observable<T>) => Observable<any>)[]): Observable<R>;
}

Giving an example of...

let o: Observable<string>;
let o2: Observable<number> = o.let(
    map<string, string>(x => x),
    map<string, number>(x => x.length * 2) ,
    map<string, number>(x => x * 10) // compiler error
);

versus

let o: Observable<string>;
let o2: Observable<number> = o.let(
    map(x => x), // T: {}, R: {}
    map(x => x.length * 2), // T: {}, R: {}
    map(x => x.length * 10) // T: {}, R: {}  // no compiler error
);

[david-driscoll]

Derp I left out what IPipeMethod is, updated.

[david-driscoll]

and replace pipe with let (I'm slow today sorry! 🐌 )

[benlesh]

I don't believe Ben is having ego issues.

Correct... I was more like "D'oh! Why didn't I notice that?". I really don't care if I have to throw out work.

[benlesh]

just need higher-order operator function thingies.

I agree with this.

However, I don't see the harm in allowing let to accept a rest of arguments, if only for ergonomics to prevent obs$.let(compose(....)) everywhere.

[david-driscoll]

So I did a little bit of thinking in the car ride home, and now have the idea of a set of operators that will work with let, or using Russian doll syntax.

let o: Observable<string>;

// using let
let o2: Observable<number> = o.let(
    oo => map(oo, x => x.length * 2),
    oo => map(oo, x => x.length * 10)
);

// using Russian doll
let o2: Observable<number> = map(map(o, x => x.length * 2), x => x.length * 10));

Basically the operators are just the prototype based operator, but instead of using the this context, the argument is passed as the first argument. I like this a little better, as they interact with let much easier (you just have to wrap it in an arrow function). It's not as convenient as the more functional pipe operators but it allows for more type safety.

I have both scripted out in #2529 as a test the different ideas, with unit tests for map on both sides.

[benlesh]

FWIW: I'm heavily in favor of pushing this forward. I'd like to either change pipe to compose or augment let to allow more than one argument (although I've never liked the name let, honestly). This will pave the way for upcoming changes and performance tweaks to the library.

[coveralls]

Coverage increased (+0.007%) to 97.652% when pulling 60269f2 on benlesh:moar-functional into ebf6393 on ReactiveX:master.

[david-driscoll]

What are your thoughts on the approach in #2529?

Basically we have these signatures:

1. function map<T, R>(this: Observable<T>, project: (value: T) => R): Observable<R>; (Today)
2. function map<T, R>(source: Observable<T>, project: (value: T) => R): Observable<R>; (WIP: (Proposal) fp and pipe operators #2529)
3. function map<T, R>(project: (value: T) => R): (source: Observable<T>) => Observable<R>; (STRAWMAN: pipe method #2489)

With the state of :: today rather like the options given by making the observable the first argument of the method. This allows for strongly typed transformation to both the prototype methods (1). with augmentation. And additionally for transformations to a higher level method that can be shared / reused.

FWIW compared to lodash (2) maps fairly well with those methods in addition on the roadmap for version 5 where partials will be replaced in favor of arrow functions.

[benlesh]

This is now a thing in 5.5 beta.

[jayphelps]

your face is now a thing in 5.5 beta

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.