Difficulty achieving deterministic results

[ptrdtznr]

Hi Liviu and FET community,

I am trying to achieve a deterministic generation process. My goal is to be able to recreate the exact same timetable after making very minor adjustments (or even with no changes at all) to see how the algorithm reacts to small data variations.

However, even though I have set a fixed random seed in the settings, every time I hit "Generate", I get a slightly different result (different number of iterations or different placement of activities).

My setup:
FET Version: 7.8.0
Random Seed: Set to "Fixed" (e.g., 12345).
Data: I am testing this without changing any data between runs.

Am I missing something? I suspect it might be related to the number of threads used for generation. Does the multi-threading affect the determinism of the algorithm even if the seed is fixed?

Should I set the "Number of threads" to 1 to get 100% reproducible results? And are there any other "hidden" factors (like the order of activities or constraints) that could break the deterministic chain?

Thanks in advance for your help

[Liviu Lalescu]

Hello, ptrdtznr,

Fixed means s10, s11, s12, s20, s21, and s22 have respectively the same values as s'10, ..., s'22.

To replicate a timetable, the seed must be fixed, the FET algorithm must be the same (so, the same FET version, or a FET version with an identical algorithm), and the data file must be identical. Even a slight variation can lead to totally different results.

The single generation starts with the inputted seed in the settings; the multiple uses the inputted seed from the settings to initialize each thread differently - the seeds for each thread are written in the fet-results. You need to either have a single thread to have the same seed as single generation, or generate multiple on the same number of threads respectively, and the seeds will be the same.

No other hidden things to change behavior, but only the above.

[ptrdtznr]

Thanks for the detailed answer. Any chance to set the seeds s10, s11, s12, s20, s21, s22 for all time to the same value? I don't want to change it after every run, to re-create the same results (in case I haven't made a single change in the planing)

[Liviu Lalescu]

You are welcome!

It would be possible to make an official FET setting for this, but I really don't want, because it is very bad for the casual users, who might misunderstand its use.

In FET-CL, you can input the seed s10, ..., s22 as a command-line parameter. But not in FET.

It is of course possible to modify the FET sources and recompile. I can help with advice, if you need.

[ptrdtznr]

I mean, if you have this options disabled and only allow it, once the user has read carefully the implications, I think this is a reasonable solution...

I've seen that you've uploaded the tar/compiled files on GitHub. In case the whole source-code would have been there, I could implement it myself...

[Liviu Lalescu]

I will add your suggestion in the TODO. But I strongly resist/object it...

You can download fet-7.8.0.tar.xz from the FET homepage; on GitHub it is the same. All the source code is there.

[ptrdtznr]

I know, I've seen it. But I'd like to open a pull request, once I've made the feature/change

[Liviu Lalescu]

I never used Git. I don't know what a pull request is  (I have a vague idea.)

And, unfortunately, I prefer to code my way, without Git.

There were a few users who sent me code, and I incorporated it, after very carefully checking it. Also, Volker, when he codes something, sends me the modified files and I KDiff3 them.

[Volker Dirr]

I also think this is a bad option for most users.
Also: Even if you need it now, as soon as you do the new timetable 6 or 12 month later, you probably forgot that you set that fixed feed.
So if you ever add the feature, then in my opinion it must be also added:
a) A warning if you try to generate multiple timetables at once. It should disallow it and warn that it is only useful if you disable the settings again.
b) A warning with EACH single generation. Because users will forget after some time that they fixed the seed.

[ptrdtznr]

The good news is: You are already doing exactly what a "Pull Request" is! When Volker sends you modified files, he is essentially making a "request" for you to "pull" his changes into your code. You using KDiff3 to verify them is the "Review" phase.

You Are the Gatekeeper
The most important thing to know is that even with Git, you keep 100% of the control.
Nothing happens automatically: A "Pull Request" is not someone forcing code into your project:
* You are the boss: It's just a formal way for someone to say, "Hey, I changed these lines; do you want them?"
* The "Accept" Button: Just like you do now with KDiff3, you see the differences, and only you have the power to click "Accept" (Merge) to let that code into your main version.

[ptrdtznr]

I also think this is a bad option for most users.
Also: Even if you need it now, as soon as you do the new timetable 6 or 12 month later, you probably forgot that you set that fixed feed.
So if you ever add the feature, then in my opinion it must be also added:
a) A warning if you try to generate multiple timetables at once. It should disallow it and warn that it is only useful if you disable the settings again.
b) A warning with EACH single generation. Because users will forget after some time that they fixed the seed.

Totally agree. For my simple use case, it's enough to use it in the manual generation (not in the multiple). I'll will have a look in 3 weeks as I'm going on a trip now.


I'm not trying to convince you to accept my feature. I'm just a big fand of open source and collaborative development. And it also could boost your visibility, etc pp.

[Liviu Lalescu]

I'll answer a bit later, as I am also preparing for a visit. 

Thank you for your kind thoughts, and have a great trip!

[ptrdtznr]

I'll answer a bit later, as I am also preparing for a visit. 

Thank you for your kind thoughts, and have a great trip!

Thank you. Enjoy visit/trip...I'll be back 😂

[Liviu Lalescu]

Oh, I see. The thing is that it is difficult for me to learn Git, with its commands. I can tell you that we have a contributor, Rodolfo Ribeiro Gomes, who created a FET repository and made some changes, which I saw and used/adapted to the official version. This was a long time ago ( https://bitbucket.org/rodolforg/fet/ , listed on the FET Homepage, Tools/Links).

[ptrdtznr]

Hello Liviu,

I am back so I today some time to dig into FET It took a little longer than expected to get my CLion environment dialed in and dive into the codebase, but I spent this afternoon extending the code and I've successfully implemented the requested feature!

I've added a new GroupBox within the Settings menu to house these options. Here is a quick breakdown of how it's currently functioning:

- Scope: This is strictly for the "Generate" (single generation) function. It will not apply to "Multiple" generations to avoid any conflicts with batch processing.
- Persistent Warnings: I've added a warning that triggers with every single generation. Since it's easy to forget that a seed has been fixed, I wanted to ensure users don't accidentally get stuck with identical results without realizing why.
- Auto-Reset Logic: For safety and to keep the default behavior clean, the feature is automatically disabled every time you restart FET.

Probably I've broke every design rule you've ever made

Cheers,
Peter

PS: I've send you an email, since I am not able to attach any ZIP file here