Add sigmoid(), sigmoid_approx(), sigmoid_affine() and sigmoid_affine_approx() functions

[NickUfer]

Closes Redot-Engine/redot-proposals#12

[NickUfer]

Ran a benchmark. Results on AMD Ryzen 9 5950X:

sigmoid              5.1 ns
sigmoid_approx       1.4 ns
sigmoid_fast         1.8 ns

In theory, the fast one should be faster than the approximation, but that is not the case. Also the output range is different with -1 to 1 for the fast one. We should think about if we want to keep it.

Test Code

#include <cstdio>
#include <ctime>
#include <cstdlib>
#include <cmath>
#include <random>
#include <chrono>

constexpr int SIZE = 100;
constexpr int CYCLES = 10000000;

double benchmark(const char *name, double (*fun)(double)) {
  double xs[SIZE];

  std::mt19937_64 generator(std::chrono::steady_clock::now().time_since_epoch().count());
  std::uniform_real_distribution distribution(-100.0, 100.0);

  for (double & x : xs) {
      x = distribution(generator);
  }

  clock_t start = clock();
  for (int repeat = 0; repeat < CYCLES; repeat++) {
      for (const double x : xs) {
          (*fun)(x);
      }
  }
  const clock_t stop = clock();
  const double t_ns = (stop - start) * 1.0e9 / CLOCKS_PER_SEC / CYCLES / SIZE;
  printf("%-17s %6.1f ns\n", name, t_ns);
  return t_ns;
}

double sigmoid(double p_x) {
  return 1.0 / (1.0 + ::exp(-p_x));
}

double sigmoid_approx(double p_x) { return 0.5 + p_x / (4.0 + fabs(p_x)); }

double sigmoid_fast(double p_x) { return p_x / (1.0 + fabs(p_x)); }

int main(int argc, char **argv) {
  benchmark("sigmoid", sigmoid);
  benchmark("sigmoid_approx", sigmoid_approx);
  benchmark("sigmoid_fast", sigmoid_fast);
}

[JohnnyThunder2]

Can confirm, this Sigmoid code adds a few Sigmoid Functions...

[8bitprodigy]

Implements Redot-Engine/redot-proposals#12

YES!

[8bitprodigy]

Also the output range is different with -1 to 1 for the fast one.

Shouldn't all output be [1..-1]?

[NickUfer]

Also the output range is different with -1 to 1 for the fast one.

Shouldn't all output be [1..-1]?

Well it depends how we want it. The output of the real / normal sigmoid is between 0 and 1. Overall I would vote for removing the sigmoid_fast() impl as its not the fastest. This would also remove the anomaly.

[8bitprodigy]

Well it depends how we want it. The output of the real / normal sigmoid is between 0 and 1. Overall I would vote for removing the sigmoid_fast() impl as its not the fastest. This would also remove the anomaly.

Ah, right. Well, as long as they're all consistent. Transforming them could be done by the user, though maybe versions that take a y-transform could be added. Would also help to have an argument for the amplitude so those bits are done in C++, rather than GDScript, as those would be common things people would want to do with it.

[NickUfer]

If you mean the y_translate as in your original example, then yeah, can do add that tomorrow. Anything else? Btw I couldn't find anything about your amplitude... seams like this is no valid sigmoid.

[8bitprodigy]

If you mean the y_translate as in your original example, then yeah, can do add that tomorrow. Anything else? Btw I couldn't find anything about your amplitude... seams like this is no valid sigmoid.

Amplitude as in the Y-scale between bounds, so instead of being [0..1], you could pass it a 2 to make it [0..2] or something.

[NickUfer]

Removed sigmoid_fast() and added sigmoid_affine() and sigmoid_affine_approx() with these benchmark results:

sigmoid               5.0 ns
sigmoid_approx        1.4 ns
sigmoid_affine        5.0 ns
sigmoid_affine_approx 1.4 ns

Test Code

#include <cstdio>
#include <ctime>
#include <cstdlib>
#include <cmath>
#include <random>
#include <chrono>

constexpr int SIZE = 100;
constexpr int CYCLES = 10000000;

double benchmark(const char *name, double (*fun)(double)) {
  double xs[SIZE];

  std::mt19937_64 generator(std::chrono::steady_clock::now().time_since_epoch().count());
  std::uniform_real_distribution distribution(-100.0, 100.0);

  for (double & x : xs) {
      x = distribution(generator);
  }

  clock_t start = clock();
  for (int repeat = 0; repeat < CYCLES; repeat++) {
      for (const double x : xs) {
          (*fun)(x);
      }
  }
  const clock_t stop = clock();
  const double t_ns = (stop - start) * 1.0e9 / CLOCKS_PER_SEC / CYCLES / SIZE;
  printf("%-17s %6.1f ns\n", name, t_ns);
  return t_ns;
}

double benchmark_affine(const char *name, double (*fun)(double, double, double)) {
  double xs[SIZE];

  std::mt19937_64 generator(std::chrono::steady_clock::now().time_since_epoch().count());
  std::uniform_real_distribution distribution(-100.0, 100.0);

  for (double & x : xs) {
      x = distribution(generator);
  }

  clock_t start = clock();
  for (int repeat = 0; repeat < CYCLES; repeat++) {
      for (const double x : xs) {
          (*fun)(x, x, x);
      }
  }
  const clock_t stop = clock();
  const double t_ns = (stop - start) * 1.0e9 / CLOCKS_PER_SEC / CYCLES / SIZE;
  printf("%-17s %6.1f ns\n", name, t_ns);
  return t_ns;
}

double sigmoid(double p_x) {
  return 1.0 / (1.0 + ::exp(-p_x));
}

double sigmoid_approx(double p_x) { return 0.5 + p_x / (4.0 + fabs(p_x)); }

double affine_sigmoid(double p_x, double p_amplitude, double p_y_translation) {
  return p_amplitude / (1.0 + ::exp(-p_x)) + p_y_translation;
}

double affine_sigmoid_approx(double p_x, double p_amplitude, double p_y_translation) {
  return p_amplitude * (0.5 + p_x / (4.0 + fabs(p_x))) + p_y_translation;
}

int main(int argc, char **argv) {
  benchmark("sigmoid", sigmoid);
  benchmark("sigmoid_approx", sigmoid_approx);
  benchmark_affine("affine_sigmoid", affine_sigmoid);
  benchmark_affine("affine_sigmoid_approx", affine_sigmoid_approx);
}

[8bitprodigy]

Nice work! That said, I thought you'd just overload the functions, since it's C++.

Either way works, though.

[NickUfer]

If you mean overloads in GDScript, that does only seem work if the overload is for a "Variant" type. But other than that, I reduced the c++ math funcs to only keep the affine version and call them with the original sigmoid values

[8bitprodigy]

If you mean overloads in GDScript, that does only seem work if the overload is for a "Variant" type. But other than that, I reduced the c++ math funcs to only keep the affine version and call them with the original sigmoid values

No, I'm talking about the C++ itself, hence why I said "since it's C++". Also why get rid of the versions that take only one argument? You can have both kinds, just overloading the function names, so one regular sigmoid with its affine version also being called sigmoid.

[NickUfer]

Okay now I get what you mean. Well, the way it is right now has the same effect, just less methods and still inlined. Maybe the compiler can even optimize away the extra multiplication by 1.0

[8bitprodigy]

Well, here's hoping it makes it in to Redot's GDScript!

[Spartan322]

This needs to be squashed, would also like to push this upstream to Godot first, but that depends on whether @8bitprodigy and @NickUfer would be okay with that.

[NickUfer]

This needs to be squashed, would also like to push this upstream to Godot first, but that depends on whether @8bitprodigy and @NickUfer would be okay with that.

Doesn't GitHub support squash on merge? Or is that a GitLab thing?

And wasn't one problem with upstream Godot that they do not merge anything the community wants and even if, that it takes forever to merge? 🤔

[Spartan322]

The way Github does squashing ruins the git history, it ruins any reference to the PR, we rely on Godot's PR Workflow still for consistency and because it is actually useful.

And even in that case, we try to stay up to date with Godot's upstream and if you haven't even tried a proposal with Godot then you can't actually know that, we prefer to look at stalled, abandoned, or rejected PRs that the community wants, those that weren't sent to Godot we'd prefer they'd be tried on Godot first to reduce the chance of merge conflicts and to give good will from ourselves despite what they did, with the exception for anyone who doesn't want to see their PR or proposal submitted upstream at all.

[NickUfer]

The way Github does squashing ruins the git history, it ruins any reference to the PR, we rely on Godot's PR Workflow still for consistency and because it is actually useful.

And even in that case, we try to stay up to date with Godot's upstream and if you haven't even tried a proposal with Godot then you can't actually know that, we prefer to look at stalled, abandoned, or rejected PRs that the community wants, those that weren't sent to Godot we'd prefer they'd be tried on Godot first to reduce the chance of merge conflicts and to give good will from ourselves despite what they did, with the exception for anyone who doesn't want to see their PR or proposal submitted upstream at all.

Well okay, then I guess this one gets forgotten. I personally do not care if this is in original Godot or not, but this already breaks some trust with what this project actually wants to achieve, to merge things the community wants, and this was a community proposal here.

[SkogiB]

The way Github does squashing ruins the git history, it ruins any reference to the PR, we rely on Godot's PR Workflow still for consistency and because it is actually useful.
And even in that case, we try to stay up to date with Godot's upstream and if you haven't even tried a proposal with Godot then you can't actually know that, we prefer to look at stalled, abandoned, or rejected PRs that the community wants, those that weren't sent to Godot we'd prefer they'd be tried on Godot first to reduce the chance of merge conflicts and to give good will from ourselves despite what they did, with the exception for anyone who doesn't want to see their PR or proposal submitted upstream at all.

Well okay, then I guess this one gets forgotten. I personally do not care if this is in original Godot or not, but this already breaks some trust with what this project actually wants to achieve, to merge things the community wants, and this was a community proposal here.

We're not going to let them sit for months before we decide to merge them, we've discussed that on the core team. If we send stuff up to Godot, it is to minimize merge conflicts on our end mostly. Let's say they have a recent PR for a fix, but drag their feet. We cherry pick it and merge it, and then 3 months from now they decide to finally merge it too. Now we risk a merge conflict that we didn't need to have.

That's basically the reason for this. It also shows good faith on our part, that we aren't being overly tribalistic.

When things are handled by sending a PR to Godot, it won't be 3 months of waiting. If they don't show interest within perhaps a week (the exact time table has not been settled on), we will then pull the trigger on a merge into Redot

[SkogiB]

Also want to apologize for the delays on this PR, we've just been busy. I do want this one merged (and cherrypicked to 4.3, the 4.3 pipeline will be much faster due to not caring if Godot wants it at all)

You don't HAVE to submit a proposal + PR to Godot, it just does the legwork for us on that front. We can cherry pick things up to them if we feel the need. If proposals were rejected by Godot, that expedites that process on our front, we know they don't want it and thus we don't foresee merge conflicts.

We ran the workflow and it failed the static check, I included a screenshot on that one. Pre-Commit is highly recommended to help catch these faster as well. If you get that fixed we'll look at it and see where we're at.

Are there any existing PRs/proposals for a sigmoid function in Godot? I feel like I saw that. Seeing they rejected it like a year ago or something would expedite our decision big time. Also, cherry pick it to 4.3 and make a PR and we can do stuff even faster for the stable builds.

[decryptedchaos]

We need to iron our a public facing statement for this, Since we are using the patch version (e.g 4.3.1) where Godot does not, it might be a good idea to tag it out for a patch release.

That will give us a timeline to track, and when Godot abandons it we'll already have a plan in place to merge things in our backlog.

My personal opinion:

I do not think Godot is going to suddenly grow a sense of urgency because we exist, we can PR them all we want, and it will most likely be met with much the same response..

As for the merge conflict issue, i think we might be creating our own problem with that, by sending them up to Godot we are increasing the chance of a merge conflict, because since we know that Godot is slow to act. it means we will have to make a decision before they do in nearly all cases.. So that means had we just merged it in the first place we would have had to do less work.

[SkogiB]

Agree with everything decrypted said, going to discuss with the core guys on reassessing the position.

[Spartan322]

This PR still needs its documentation to be fixed. @NickUfer

[8bitprodigy]

This needs to be squashed, would also like to push this upstream to Godot first, but that depends on whether @8bitprodigy and @NickUfer would be okay with that.

I'd be okay with it, though I only created the issue this is responding to. Nick did all the work.

[8bitprodigy]

I do want this one merged (and cherrypicked to 4.3, the 4.3 pipeline will be much faster due to not caring if Godot wants it at all)

I am excited to hear this!

[SkogiB]

Thanks for squashing, we just need this fixed and then squashed again and we should be good to go once it passes the workflow checks.